Tidy corpus linguistics data with tidyr
The tidyr package is part of the tidyverse. As its name indicates, it is meant to help you create tidy data or tidy messy data according to the tidy data principles: each variable forms a column; each observation forms a row; each type of observational unit forms a table. This post illustrates how to tidy a data set in R using two tidyr functions: pivot_longer() and separate().
First, let us load the package.
library(tidyr)
Pivot data from wide to long
Tab. 1 was featured in a previous post. It is messy because the column headers are values (actual age ranges) that should be grouped under a single variable name (“age”).
| variable | 0_10 | 11_18 | 19_29 | 30_39 | 40_49 | 50_59 | 60_69 | 70_79 | 80_89 | 90_99 |
|---|---|---|---|---|---|---|---|---|---|---|
| hello | 77 | 139 | 377 | 186 | 261 | 90 | 80 | 59 | 20 | 9 |
| hi | 14 | 52 | 305 | 46 | 57 | 19 | 36 | 11 | 3 | 0 |
To tidy the data table, we need to pivot it, i.e. increase the number of rows and decrease the number of columns. This is done with pivot_longer().
Load the messy data:
df.messy.1 <- read.table("https://bit.ly/366nbkn", header=T, sep="\t", check.names = F)
Apply pivot_longer():
df.longer.1 <- df.messy.1 %>% pivot_longer( !(variable), # all the columns except 'variable' are concerned names_to = "age", # new column values_to = "frequency", # where the counts will appear values_drop_na = TRUE # do not include NA values (providing NA values appear) ) df.longer.1 # inspect

The age ranges are now grouped under a single variable: age.
Separate a character column into multiple columns
Tab. 2 was also featured as messy in a previous post because of its multiple variables stored in the same column. Indeed, each column apart from general_extender conflates two variables: city and socioeconomic status (WC = ‘working class’; MC = ‘middle class’)
| general_extender | Reading_MC | Reading_WC | Milton.Keynes_MC | Milton.Keynes_WC | Hull_MC |
|---|---|---|---|---|---|
| and that | 4 | 49 | 9 | 44 | 10 |
| and all that | 4 | 14 | 2 | 4 | 1 |
| and stuff | 36 | 6 | 45 | 5 | 62 |
| and things | 32 | 0 | 35 | 0 | 12 |
| and everything | 21 | 16 | 22 | 18 | 30 |
| or something | 72 | 20 | 30 | 17 | 23 |
Tidying the Tab. 2 involves two steps. We need to:
- pivot the data frame (i.e. increase the number of rows and decrease the number of columns)
- split each column into two distinct variables: city and socioeconomic status.
Load the data:
df.messy.2 <- read.table("https://bit.ly/34KGjER", header=TRUE, sep="\t")
pivot
Pivot the messy data frame with pivot_longer():
df.longer.2 <- df.messy.2 %>% pivot_longer( !(general_extender), names_to = "city_socioeconomicstatus", values_to = "frequency", values_drop_na = TRUE ) df.longer.2 # inspect

Split columns
Split each column into two distinct variables (city and socioeconomic status) with separate():
df.separate <- separate(df.longer.2,
city_socioeconomicstatus,
sep="_",
into=c("city", "socioeconomic_status"))
df.separate # inspect

The data frame is now tidy.
More functionalities
There are of course more functionalities than the two I have illustrated above. Fore more details, Please refer to the tidyr documentation.
OpenEdition suggests that you cite this post as follows:
Guillaume Desagulier (November 17, 2020). Tidy corpus linguistics data with tidyr. Around the word. Retrieved May 21, 2026 from https://doi.org/10.58079/n4v3


orcid.org/0000-0003-4895-0788