Tidy corpus linguistics data with tidyr
tidyr package is part of the tidyverse. As its name indicates, it is meant to help you create tidy data or tidy messy data according to the tidy data principles: each variable forms a column; each observation forms a row; each type of observational unit forms a table. This post illustrates how to tidy a data set in R using two
First, let us load the package.
Pivot data from wide to long
Tab. 1 was featured in a previous post. It is messy because the column headers are values (actual age ranges) that should be grouped under a single variable name (“age”).
To tidy the data table, we need to pivot it, i.e. increase the number of rows and decrease the number of columns. This is done with
Load the messy data:
df.messy.1 <- read.table("https://bit.ly/366nbkn", header=T, sep="\t", check.names = F)
df.longer.1 <- df.messy.1 %>% pivot_longer( !(variable), # all the columns except 'variable' are concerned names_to = "age", # new column values_to = "frequency", # where the counts will appear values_drop_na = TRUE # do not include NA values (providing NA values appear) ) df.longer.1 # inspect
The age ranges are now grouped under a single variable:
Separate a character column into multiple columns
Tab. 2 was also featured as messy in a previous post because of its multiple variables stored in the same column. Indeed, each column apart from
general_extender conflates two variables: city and socioeconomic status (WC = ‘working class’; MC = ‘middle class’)
|and all that||4||14||2||4||1|
Tidying the Tab. 2 involves two steps. We need to:
- pivot the data frame (i.e. increase the number of rows and decrease the number of columns)
- split each column into two distinct variables: city and socioeconomic status.
Load the data:
df.messy.2 <- read.table("https://bit.ly/34KGjER", header=TRUE, sep="\t")
Pivot the messy data frame with
df.longer.2 <- df.messy.2 %>% pivot_longer( !(general_extender), names_to = "city_socioeconomicstatus", values_to = "frequency", values_drop_na = TRUE ) df.longer.2 # inspect
Split each column into two distinct variables (city and socioeconomic status) with
df.separate <- separate(df.longer.2, city_socioeconomicstatus, sep="_", into=c("city", "socioeconomic_status")) df.separate # inspect
The data frame is now tidy.
There are of course more functionalities than the two I have illustrated above. Fore more details, Please refer to the