Creating frequency lists with R
In a previous post, I provided two frequency lists without revealing the scripts to make them. This post explains how to create word frequency lists (lemmatized and unlemmatized) from the BNC 2014 (spoken), using R programming. In the field of corpus linguistics, frequency lists are an important tool in analyzing language usage. A frequency list provides a count of how often each word or lemma occurs in a given corpus, shedding light on the lexical coloration of your dataset. For more details, see Section 5.4 of Corpus Linguistics and Statistics with R.
Getting Started
Before diving into the code, make sure that you have the gsubfn library installed and loaded. This is done by running the following command in your R console:
install.packages("gsubfn")
The gsubfn library provides string manipulation functions. In the scripts below, it is used for pattern matching and replacement.
Next, download . The 11.5-million-word spoken component of the BNC2014 consists of transcripts of recorded conversations involving 672 speakers from different parts of the UK between 2012 and 2016. The corpus breaks down into 1,251 files, i.e. one per conversation. You need to download the BNC2014 corpus files from this page before proceeding with the code below.
As mentioned in my introductory post to the BNC2014, once you have downloaded the files and stored them on your hard drive, the folder architecture looks like this:

We are interested in the tagged folder because we want to retrieve the POS tags.
The lemmatized freqlist
We begin with the lemmatized frequency list. We want a three-column table: the first column contains the lemmas, the second column their respective POS tags, and the third column their respective frequency counts.
First, we clear the workspace and load gsubfn.
# Clear workspace
rm(list=ls(all=TRUE))
# Load necessary libraries
library(gsubfn)
Next, we specify the Path to where the BNC2014 Spoken are stored. The list.files function is then used to get a list of file names matching the pattern .xml in the specified directory.
corpus.files <- list.files(path="/bnc2014spoken/spoken/tagged", pattern="\.xml$", full.names=TRUE)
We create an empty character vector all.matches to collect all the matches found during processing.
all.matches <- character()
The code below enters a loop to iterate through each file in the list of corpus files.
for (i in 1:length(corpus.files)) {
The current corpus file is read into a character vector using the scan function.
corpus.file <- scan(corpus.files[i], what="char", sep="\n")
Regular expressions are used to extract information (lemmas and classes) from the corpus file. The strapplyc function is applied to extract matching patterns.
lemmas <- unlist(strapplyc(words, "lemma=\"(\w+)\"", backref=1))
classes <- unlist(strapplyc(words, "class=\"(\w+)\"", backref=1))
Lemmas and classes are combined and stored in the all.matches vector.
lemmas.classes <- paste(lemmas, classes, sep="_")
all.matches <- c(all.matches, lemmas.classes)
}
Note that the loop will take some time to run. The time varies depending on the speed of your processor and how much memory (RAM) your system has.
The table function is used to create a frequency table of the combined lemmas and classes.
all.matches.table <- table(all.matches)
The frequency table is sorted in decreasing order.
all.matches.sorted.table <- sort(all.matches.table, decreasing=TRUE)
The sorted frequency table is formatted into a tab-separated table.
tab.table <- paste(names(all.matches.sorted.table), all.matches.sorted.table, sep="\t")
tab.table.2 <- gsub("_", "\t", tab.table, perl=TRUE)
The final step involves saving the formatted frequency table to a text file on the desktop.
cat("LEMMA\tCLASS\tFREQUENCY", tab.table.2, file="/Users/yourname/Desktop/freqlist.bnc.2014.txt", sep="\n")
Note that you must replace /Users/yourname/Desktop/ with the actual path where you want to save the output file. The file freqlist.bnc.2014.txt can now be opened with a spreadsheet software.
Here is the same code in one single chunk:
# Clear workspace
rm(list=ls(all=TRUE))
# Load necessary libraries
library(gsubfn)
# Specify the path to the BNC 2014 spoken corpus files
corpus.files <- list.files(path="/bnc2014spoken/spoken/tagged", pattern="\\.xml$", full.names=TRUE)
# Prepare an empty vector to collect all matches
all.matches <- character()
# Enter the loop
for (i in 1:length(corpus.files)) {
# Load current corpus file
corpus.file <- scan(corpus.files[i], what="char", sep="\n")
# Collect relevant elements (lemmas and classes)
lemmas <- unlist(strapplyc(words, "lemma=\"(\\w+)\"", backref=1))
classes <- unlist(strapplyc(words, "class=\"(\\w+)\"", backref=1))
# Collect all matches
lemmas.classes <- paste(lemmas, classes, sep="_")
all.matches <- c(all.matches, lemmas.classes)
}
# Create a frequency table
all.matches.table <- table(all.matches)
# Sort the frequency table
all.matches.sorted.table <- sort(all.matches.table, decreasing=TRUE)
# Prepare the output table
tab.table <- paste(names(all.matches.sorted.table), all.matches.sorted.table, sep="\t")
tab.table.2 <- gsub("_", "\t", tab.table, perl=TRUE)
# Save the frequency list to a text file
cat("LEMMA\tCLASS\tFREQUENCY", tab.table.2, file="/Users/yourname/Desktop/freqlist.bnc.2014.txt", sep="\n")
Upon inspection with a spreadsheet software (I am using Excel), your frequency list should look like this:

The unlemmatized freqlist
Now, let’s modify the above script for creating an unlemmatized frequency list. What changes is the last part of the loop, namely:
# Collect relevant elements (lemmas and classes)
words <- unlist(strapplyc(corpus.file, "<w pos=\"\\w+\" lemma=\"\\w+\" class=\"\\w+\" usas=\"\\w+\">", backref=1))
classes <- unlist(strapplyc(words, "class=\"(\\w+)\"", backref=1))
# Collect all matches
words.classes <- paste(words, classes, sep="_")
all.matches <- c(all.matches, words.classes)
Instead of collecting lemmas, we collect words (words <- unlist(strapplyc(corpus.file, "<w pos=\"\\w+\" lemma=\"\\w+\" class=\"\\w+\" usas=\"\\w+\">", backref=1))). Here is the code as a single chunk:
# Clear workspace
rm(list=ls(all=TRUE))
# Load necessary libraries
library(gsubfn)
# Specify the path to the BNC 2014 spoken corpus files
corpus.files <- list.files(path="/bnc2014spoken/spoken/tagged", pattern="\\.xml$", full.names=TRUE)
# Prepare an empty vector to collect all matches
all.matches <- character()
# Enter the loop
for (i in 1:length(corpus.files)) {
# Load current corpus file
corpus.file <- scan(corpus.files[i], what="char", sep="\n")
# Collect relevant elements (lemmas and classes)
words <- unlist(strapplyc(corpus.file, "<w pos=\"\\w+\" lemma=\"\\w+\" class=\"\\w+\" usas=\"\\w+\">", backref=1))
classes <- unlist(strapplyc(words, "class=\"(\\w+)\"", backref=1))
# Collect all matches
words.classes <- paste(words, classes, sep="_")
all.matches <- c(all.matches, words.classes)
}
# Create a frequency table
all.matches.table <- table(all.matches)
# Sort the frequency table
all.matches.sorted.table <- sort(all.matches.table, decreasing=TRUE)
# Prepare the output table
tab.table <- paste(names(all.matches.sorted.table), all.matches.sorted.table, sep="\t")
tab.table.2 <- gsub("_", "\t", tab.table, perl=TRUE)
# Save the frequency list to a text file
cat("WORD\tCLASS\tFREQUENCY", tab.table.2, file="/Users/yourname/Desktop/freqlist.bnc.2014.txt", sep="\n")
This script saves the unlemmatized frequency list to a separate text file (freqlist.bnc.2014.unlem.txt).

The frequency lists’ files are available from me upon request.
Cover image credits: Glen Carrie.
The text only may be used under licence Creative Commons Attribution Non Commercial 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated.
OpenEdition suggests that you cite this post as follows:
Guillaume Desagulier (December 21, 2023). Creating frequency lists with R. Around the word. Retrieved May 20, 2026 from https://doi.org/10.58079/vekz


orcid.org/0000-0003-4895-0788