Should corpus linguists embrace the tidyverse?
A new coding style is gradually establishing itself as a new standard in the world of R under the impulse of Hadley Wickham: the tidy style. It is the cornerstone of the
Since the publication of my book on corpus linguistics with R, many colleagues have asked me if I was planning to rewrite the whole code in tidy style (some of the code in the book makes use of some
tidyverse packages). If you want to know my answer, please read on.
The vast majority of new R users seem to be jumping at the tidy style becayse the tidyverse is the way to go, full stop. To me, who started programming with R when the tidyverse did not even exist yet, the question is well worth asking. I am still wondering if this extra-layer of R programming outperforms base-R functions. It does, to some extent.
The tidy philosophy
tidyverse package is a suite of related packages for data manipulation (
dplyr), visualization (
ggplot2), new ways of handling strings (
stringr), factors (
forcats), and data frames (
tidyr), among other functionalities. The full list of tidyverse packages is available here.
Why a new style?
You may be wondering why a new programming style is needed when R’s base style and functions work like a charm. Hadley Wickham and colleagues started from the assumption that data cleaning and preparation is a time-consuming activity, especially when one deals with a workflow that involves switching between programs and packages (Wickham 2014).
Packages should therefore share a set of structuring principles that guarantee interoperability and make importing data, manipulating data, and visualizing data easier.
The tidy workflow hinges on:
- tidy data
- tidy tools
Tidy data have the following properties:
Each variable forms a column.
Each observation forms a row.
Each type of observational unit forms a table.Wickham (2014:4)
Messy data, on the other hand, is often afflicted with the following problems:
- column headers are values, not variable names
- multiple variables are stored in one column
- variables are stored in both rows and columns
- multiple types of observational units are stored in the same table
- a single observational unit is stored in multiple tables
Below, I illustrate two of the most common issues, namely (a) columns headers as values, and (b) multiple variables stored in one column.
Column headers as values
Tab. 1 was compiled using
BNC.2014.query(), an interactive R script for a sociolinguistic exploration of the spoken component of the BNC-2014. It summarizes the use of hello and hi across age groups.
In the tidyverse, Tab. 1 is considered messy because the column headers are values (actual age ranges) that should be grouped under a single variable name (“age”), as in Tab. 2.
Multiple variables stored in one column
Tab. 3 is adapted from Cheshire (2007). In the tidyverse, it is considered messy because each column conflates two variables: city and socioeconomic status (WC = ‘working class’; MC = ‘middle class’).
|and all that||4||14||2||4||1|
The tidy equivalent is Tab. 4.
|and all that||Reading||MC||4|
|and all that||Reading||WC||14|
|and all that||Milton.Keynes||MC||2|
|and all that||Milton.Keynes||WC||4|
|and all that||Hull||MC||1|
All the tidy tools comply with four basic principles listed in the tidy tools manifesto:
- Reuse existing data structures.
- Compose simple functions with the pipe.
- Embrace functional programming.
- Design for humans.
Reuse existing data structures
The packages written following the tidyverse philopsophy work with the same kind of data structures. In other words, if a given data structure is tidy, it should be compatible with any
Compose simple functions with the pipe
Complex operations can be broken down into a series of simple, compatible, and non-overlapping steps. Single functions are composed with the pipe:
%>%, which works across all packages. Examples of how this works are given below.
Embrace functional programming
Unlike Python or C#, which are object-oriented languages, R is a functional programming language. Object-oriented programming (abbreviated OOP) and functional programming (FP) share two basic components:
- the data
- what you do with the data
They differ with respect to how these two components combine.
OOP has been around for longer than FP. In OOP, data and its manipulation are merged in objects. In other words, whenever you manipulate the data, you create an object that combines instructions and data. One benefit is that nothing is hidden behind a function, and whatever change is visible in the code. Another benefit is that the code is read like a simple set of instructions, just like a computer would read it. The downside is that the structure of the data is changed based on the programmer’s subjective conception of how that data should be processed, and parallel programming is therefore more difficult.
FP maintains a strict separation between data and functions. Both remain stable during the programming process. Using clean and transparent functions leads to stable results. What OOP supporters do not like about FP is that several layers of functions hide the data manipulation process. Another downside is that you need to learn how each function works, and the implied workload is demultiplied by the number of functions needed. This last point is addressed in the tidyverse.
Design for humans
A package and its functions should be easy to understand and use by humans. This means that computer efficiency is not a secondary concern because the bottleneck in most data analysis is thinking time, not computing time, according to Hadley Wickham.
Emphasis is put on functions whose names are explicit. This means that a function’s name is a clear prompt to what it does. Families of functions (i.e. functions that do similar operations) should share an identical prefix.
tidyverse packages (and conflicts)
The tools included in the
tidyverse package comply with the tidy tools manifesto outlined above. When you load
tidyverse, R tells you what tidy packages are installed.
R also tells you what conflicts there are. Conflicts are in fact tidyverse functions whose names are identical to functions provided by other packages that may be installed on your version of R. For example, R tells me that the
arrange() function from
dplyr conflicts with a function with the same name from the
To minimize conflicts, you can either:
- load the
tidyversecomponents individually, avoiding the component that is responsible for the conflicts, or
- remove the packages that are not part of
tidyverseand that cause the conflicts.
For example, if need
ggplot2 but not
dplyr, I will choose the first option and load each sub-package individually:
On the other hand, if I am not using
plyr and I need
dplyr instead, I will adopt the second option, i.e. I will unload the conflicting package with
You can print a list of conflicts at any time by entering:
You can also access the full list of packages in the tidyverse with
Pros and cons
In my opinion, the pros are the following:
tidyverseis consistent: its constellation of packages adopt the same grammar.
tidyverseis heavily documented.
- yes, messy data is a pain in the a**.
- if it is not already the case, tidyverse supporters are going to outnumber base-R programmers.
- knowing the tidy style will get you a job in the data-manipulation industry.
- The tidyverse has cool merchandise.
In my opinion, the cons are the following:
- lack of stability: package versions change fast, much faster than base-R packages and functions, which means that you will have to udpate your code often.
- occasional useless complexity: some base-R one-liners take three lines of
- lack of compatibility: the tidyverse is to R programming what Apple is to electronic equipment. It is trendy, cool, efficient, but awkward if you do not possess ALL the side equipment. Some popular and efficient packages outside the tidyverse will simply not work if your data is tidy in the tidyverse sense. For example, I have used
FactoMineRfor ten years, it works like a charm, and I refuse to consider that its input data format is messy.
- Not knowing the tidy style might keep you out of a job in the data industry even if you are good at manipulating data in base-R.
- The spirit behind the tidyverse is somewhat dogmatic, and not everyone likes it. Let me spin further the comparison with Apple. On the one hand, Apple gives a great user experience pride of place with cool features and extensive applications. Similarly, the tidyverse is a much welcome addition to the R world (especially
ggplot2, which brings visualization to the next level). Apple make sure the customer feels like it is worth paying the higher price. With respect to the tidyverse, the price to pay is the new coding format (and the introduction of the pipe operator
%>%). On the other hand, Apple’s marketing creates waves of raving fans who stand in line for hours by making them believe that if you buy their products you are cool, if you do not, you are a dinosaur. Likewise, the tidyverse community make it sound like data manipulation is cool with tidy packages and conservative with base-R. Admittedly, the tidyverse is full of great features, but I refuse to believe that you are a programming dinosaur if you combine some elements of the tidyverse with some elements of base-R. When I code, I have the same critical mind as when I buy electronic products (I own a Mac, an IPad, but my phone is a Samsung).
Notwithstanding the above, the tidyverse is swamping the world of R programming. Indeed, tidyverse supporters train more and more R beginners everyday. As a result, the balance is clearly shifting towards the tidyverse.
Even if you do not like the tidy style, and do not use it on a regular basis, you had better know it because if you share some code with the outside world, people are very likely to ask you if you have a tidyverse equivalent because they will not understand good-old base-R style.
What I will do
I will keep on integrating more
tidyverse components into my code (including future editions of my book) as long as these components bring more efficiency with respect to base-R.
Future posts will feature what I believe are great tidyverse improvements upon base-R in corpus linguistics. They will also mention data-manipulation contexts where base-R is still ahead in terms of simplicity and efficiency.
Cheshire, Jenny. 2007. “Discourse Variation, Grammaticalisation and Stuff Like That.” Journal of Sociolinguistics 11 (2): 155–93. https://doi.org/10.1111/j.1467-9841.2007.00317.x.
Wickham, Hadley. 2014. “Tidy Data.” Journal of Statistical Software, Articles 59 (10): 1–23. https://doi.org/10.18637/jss.v059.i10.