Should corpus linguists embrace the tidyverse?

A new coding style is gradually establishing itself as a new standard in the world of R under the impulse of Hadley Wickham: the tidy style. It is the cornerstone of the tidyverse package.

Since the publication of my book on corpus linguistics with R, many colleagues have asked me if I was planning to rewrite the whole code in tidy style (some of the code in the book makes use of some tidyverse packages). If you want to know my answer, please read on.

The vast majority of new R users seem to be jumping at the tidy style becayse the tidyverse is the way to go, full stop. To me, who started programming with R when the tidyverse did not even exist yet, the question is well worth asking. I am still wondering if this extra-layer of R programming outperforms base-R functions. It does, to some extent.

The tidy philosophy

The tidyverse package is a suite of related packages for data manipulation (dplyr), visualization (ggplot2), new ways of handling strings (stringr), factors (forcats), and data frames (tidyr), among other functionalities. The full list of tidyverse packages is available here.

Why a new style?

You may be wondering why a new programming style is needed when R’s base style and functions work like a charm. Hadley Wickham and colleagues started from the assumption that data cleaning and preparation is a time-consuming activity, especially when one deals with a workflow that involves switching between programs and packages (Wickham 2014).

Packages should therefore share a set structuring principles that guarantee interoperability and make importing data, manipulating data, and visualizing data easier.

The tidy workflow hinges on:

  • tidy data
  • tidy tools

Tidy data

Tidy data have the following properties:

Each variable forms a column.

Each observation forms a row.

Each type of observational unit forms a table.

Wickham (2014:4)

Messy data, on the other hand, is often afflicted with the following problems:

  • column headers are values, not variable names
  • multiple variables are stored in one column
  • variables are stored in both rows and columns
  • multiple types of observational units are stored in the same table
  • a single observational unit is stored in multiple tables

Below, I illustrate two of the most common issues, namely (a) columns headers as values, and (b) multiple variables stored in one column.

Column headers as values

Tab. 1 was compiled using BNC.2014.query(), an interactive R script for a sociolinguistic exploration of the spoken component of the BNC-2014. It summarizes the use of hello and hi across age groups.

variable0_1011_1819_2930_3940_4950_5960_6970_7980_8990_99
hello77139377186261908059209
hi1452305465719361130
Table 1. A messy data frame with columns headers as values

In the tidyverse, Tab. 1 is considered messy because the column headers are values (actual age ranges) that should be grouped under a single variable name (“age”), as in Tab. 2.

variableagefrequency
hello0_1077
hello11_18139
hello19_29377
hello30_39186
hello40_49261
hello50_5990
hello60_6980
hello70_7959
hello80_8920
hello90_999
hi0_1014
hi11_1852
hi19_29305
hi30_3946
hi40_4957
hi50_5919
hi60_6936
hi70_7911
hi80_893
hi90_990
Table 2. A tidy data frame where values are grouped under a single variable
Multiple variables stored in one column

Tab. 3 is adapted from Cheshire (2007). In the tidyverse, it is considered messy because each column conflates two variables: city and socioeconomic status (WC = ‘working class’; MC = ‘middle class’).

general_extenderReading_MCReading_WCMilton.Keynes_MCMilton.Keynes_WCHull_MC
and that44994410
and all that414241
and stuff36645562
and things32035012
and everything2116221830
or something7220301723
Table 3. A messy data frame with multiple variables stored in the same column

The tidy equivalent is Tab. 4.

general_extendercitysocioeconomic statusfrequency
and thatReadingMC4
and thatReadingWC49
and thatMilton.KeynesMC9
and thatMilton.KeynesWC44
and thatHullMC10
and all thatReadingMC4
and all thatReadingWC14
and all thatMilton.KeynesMC2
and all thatMilton.KeynesWC4
and all thatHullMC1
and stuffReadingMC36
and stuffReadingWC6
and stuffMilton.KeynesMC45
and stuffMilton.KeynesWC5
and stuffHullMC62
and thingsReadingMC32
and thingsReadingWC0
and thingsMilton.KeynesMC35
and thingsMilton.KeynesWC0
and thingsHullMC12
and everythingReadingMC21
and everythingReadingWC16
and everythingMilton.KeynesMC22
and everythingMilton.KeynesWC18
and everythingHullMC30
or somethingReadingMC72
or somethingReadingWC20
or somethingMilton.KeynesMC30
or somethingMilton.KeynesWC17
or somethingHullMC23
Table 4. A tidy data frame with one variable per column

Tidy tools

All the tidy tools comply with four basic principles listed in the tidy tools manifesto:

  1. Reuse existing data structures.
  2. Compose simple functions with the pipe.
  3. Embrace functional programming.
  4. Design for humans.
Reuse existing data structures

The packages written following the tidyverse philopsophy work with the same kind of data structures. In other words, if a given data structure is tidy, it should be compatible with any tidyverse package.

Compose simple functions with the pipe

Complex operations can be broken down into a series of simple, compatible, and non-overlapping steps. Single functions are composed with the pipe: %>%, which works across all packages. Examples of how this works are given below.

Embrace functional programming

Unlike Python or C#, which are object-oriented languages, R is a functional programming language. Object-oriented programming (abbreviated OOP) and functional programming (FP) share two basic components:

  • the data
  • what you do with the data

They differ with respect to how these two components combine.

OOP has been around for longer than FP. In OOP, data and its manipulation are merged in objects. In other words, whenever you manipulate the data, you create an object that combines instructions and data. One benefit is that nothing is hidden behind a function, and whatever change is visible in the code. Another benefit is that the code is read like a simple set of instructions, just like a computer would read it. The downside is that the structure of the data is changed based on the programmer’s subjective conception of how that data should be processed, and parallel programming is therefore more difficult.

FP maintains a strict separation between data and functions. Both remain stable during the programming process. Using clean and transparent functions leads to stable results. What OOP supporters do not like about FP is that several layers of functions hide the data manipulation process. Another downside is that you need to learn how each function works, and the implied workload is demultiplied by the number of functions needed. This last point is addressed in the tidyverse.

Design for humans

A package and its functions should be easy to understand and use by humans. This means that computer efficiency is not a secondary concern because the bottleneck in most data analysis is thinking time, not computing time, according to Hadley Wickham.

Emphasis is put on functions whose names are explicit. This means that a function’s name is a clear prompt to what it does. Families of functions (i.e. functions that do similar operations) should share an identical prefix.

tidyverse packages (and conflicts)

The tools included in the tidyverse package comply with the tidy tools manifesto outlined above. When you load tidyverse, R tells you what tidy packages are installed.

library(tidyverse)

R also tells you what conflicts there are. Conflicts are in fact tidyverse functions whose names are identical to functions provided by other packages that may be installed on your version of R. For example, R tells me that the arrange() function from dplyr conflicts with a function with the same name from the plyr package.

To minimize conflicts, you can either:

  • load the tidyverse components individually, avoiding the component that is responsible for the conflicts, or
  • remove the packages that are not part of tidyverse and that cause the conflicts.

For example, if need tibble and ggplot2 but not dplyr, I will choose the first option and load each sub-package individually:

library(readr)
library(ggplot2)

On the other hand, if I am not using plyr and I need dplyr instead, I will adopt the second option, i.e. I will unload the conflicting package with detach():

detach("package:plyr", unload=TRUE)

You can print a list of conflicts at any time by entering:

tidyverse_conflicts()

You can also access the full list of packages in the tidyverse with tidyverse_packages()

Pros and cons

the pros

In my opinion, the pros are the following:

  • tidyverse is consistent: its constellation of packages adopt the same grammar.
  • tidyverse is heavily documented.
  • yes, messy data is a pain in the a**.
  • if it is not already the case, tidyverse supporters are going to outnumber base-R programmers.
  • knowing the tidy style will get you a job in the data-manipulation industry.
  • The tidyverse has cool merchandise.

the cons

In my opinion, the cons are the following:

  • lack of stability: package versions change fast, much faster than base-R packages and functions, which means that you will have to udpate your code regularly.
  • occasional useless complexity: some base-R one-liners take three lines of tidyverse code.
  • lack of compatibility: the tidyverse is to R programming what Apple is to electronic equipment. It is trendy, cool, efficient, but awkward if you do not possess ALL the side equipment. Some popular and efficient packages outside the tidyverse will simply not work if your data is tidy in the tidyverse sense. For example, I have used FactoMineR for ten years, it works like a charm, and I refuse to consider that its input data format is messy.
  • Not knowing the tidy style might keep you out of a job in the data industry even if you are good at manipulating data in base-R.
  • The spirit behind the tidyverse is somewhat dogmatic, and not everyone likes it. Let me spin further the comparison with Apple. On the one hand, Apple gives a great user experience pride of place with cool features and extensive applications. Similarly, the tidyverse is a much welcome addition to the R world (especially ggplot2, which brings visualization to the next level). Apple make sure the customer feels like it is worth paying the higher price. With respect to the tidyverse, the price to pay is the new coding format (and the introduction of the pipe operator %>%). On the other hand, Apple’s marketing creates waves of raving fans who stand in line for hours by making them believe that if you buy their products you are cool, if you do not, you are a dinosaur. Likewise, the tidyverse community make it sound like data manipulation is cool with tidy packages and conservative with base-R. Admittedly, the tidyverse is full of great features, but I refuse to believe that you are a programming dinosaur if you combine some elements of the tidyverse with some elements of base-R. When I code, I have the same critical mind as when I buy electronic products (I own a Mac, an IPad, but my phone is a Samsung).

Notwithstanding the above, the tidyverse is swamping the world of R programming. Indeed, tidyverse supporters train more and more R beginners everyday. As a result, the balance is clearly shifting towards the tidyverse.

bottom line

Even if you do not like the tidy style, and do not use it on a regular basis, you had better know it because if you share some code with the outside world, people are very likely to ask you if you have a tidyverse equivalent because they will not understand good-old base-R style.

What I will do

I will keep on integrating more tidyverse components into my code (including future editions of my book) as long as these components bring more efficiency with respect to base-R.

Future posts will feature what I believe are great tidyverse improvements upon base-R in corpus linguistics. They will also mention data-manipulation contexts where base-R is still ahead in terms of simplicity and efficiency.

References

Cheshire, Jenny. 2007. “Discourse Variation, Grammaticalisation and Stuff Like That.” Journal of Sociolinguistics 11 (2): 155–93. https://doi.org/10.1111/j.1467-9841.2007.00317.x.

Wickham, Hadley. 2014. “Tidy Data.” Journal of Statistical Software, Articles 59 (10): 1–23. https://doi.org/10.18637/jss.v059.i10.

Cite this article as: Guillaume Desagulier, "Should corpus linguists embrace the tidyverse?," in Around the word, 16/11/2020, https://corpling.hypotheses.org/3428.

Guillaume Desagulier

UMR 7114 MoDyCo — Université Paris 8, CNRS, Université Paris Nanterre, Institut Universitaire de France.

Vous aimerez aussi...

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *

Ce site utilise Akismet pour réduire les indésirables. En savoir plus sur comment les données de vos commentaires sont utilisées.

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search