Tidy corpus linguistics data with tidyr

The tidyr package is part of the tidyverse. As its name indicates, it is meant to help you create tidy data or tidy messy data according to the tidy data principles: each variable forms a column; each observation forms a row; each type of observational unit forms a table. This post illustrates how to tidy a data set in R using two tidyr functions: pivot_longer() and separate().

First, let us load the package.


Pivot data from wide to long

Tab. 1 was featured in a previous post. It is messy because the column headers are values (actual age ranges) that should be grouped under a single variable name (“age”).

Table 1. A messy data frame with columns headers as values

To tidy the data table, we need to pivot it, i.e. increase the number of rows and decrease the number of columns. This is done with pivot_longer().

Load the messy data:

df.messy.1 <- read.table("https://bit.ly/366nbkn", header=T, sep="\t", check.names = F)

Apply pivot_longer():

df.longer.1 <- df.messy.1 %>%
   !(variable), # all the columns except 'variable' are concerned
   names_to = "age", # new column
   values_to = "frequency", # where the counts will appear
   values_drop_na = TRUE # do not include NA values (providing NA values appear)
df.longer.1 # inspect

The age ranges are now grouped under a single variable: age.

Separate a character column into multiple columns

Tab. 2 was also featured as messy in a previous post because of its multiple variables stored in the same column. Indeed, each column apart from general_extender conflates two variables: city and socioeconomic status (WC = ‘working class’; MC = ‘middle class’)

and that44994410
and all that414241
and stuff36645562
and things32035012
and everything2116221830
or something7220301723
Table 2. A messy data frame with multiple variables stored in the same column

Tidying the Tab. 2 involves two steps. We need to:

  • pivot the data frame (i.e. increase the number of rows and decrease the number of columns)
  • split each column into two distinct variables: city and socioeconomic status.

Load the data:

df.messy.2 <- read.table("https://bit.ly/34KGjER", header=TRUE, sep="\t")


Pivot the messy data frame with pivot_longer():

df.longer.2 <- df.messy.2 %>%
   names_to = "city_socioeconomicstatus",
   values_to = "frequency",
   values_drop_na = TRUE
df.longer.2 # inspect

Split columns

Split each column into two distinct variables (city and socioeconomic status) with separate():

df.separate <- separate(df.longer.2,
   into=c("city", "socioeconomic_status"))
df.separate # inspect

The data frame is now tidy.

More functionalities

There are of course more functionalities than the two I have illustrated above. Fore more details, Please refer to the tidyr documentation.

Cite this article as: Guillaume Desagulier, "Tidy corpus linguistics data with tidyr," in Around the word, 17/11/2020, https://corpling.hypotheses.org/3441.

Guillaume Desagulier

UMR 7114 MoDyCo — Université Paris 8, CNRS, Université Paris Nanterre, Institut Universitaire de France.

Vous aimerez aussi...

Laisser un commentaire

Votre adresse e-mail ne sera pas publiée. Les champs obligatoires sont indiqués avec *

Ce site utilise Akismet pour réduire les indésirables. En savoir plus sur comment les données de vos commentaires sont utilisées.