BNC.query(). An interactive R script for a sociolinguistic exploration of the spoken component of the BNC-XML

BNC.query() is an interactive R script that I wrote for a course in computational sociolinguistics last semester. It is designed to run queries over the BNC-XML (spoken component). It extracts a single word or a complex expression, along with speaker information (gender, age class, and social grade), tabulates the results, computes frequencies, and makes a barplot or an association plot (along with a χ2 test).

For this demo, I have used R version 3.5.1 (2018-07-02) — « Feather Spray » on RStudio (Version 1.1.463). I cannot guarantee that the script will work on previous versions. 

Obtaining the corpus

For the script to run properly, the BNC-XML files must be installed on your system. To download the BNC-XML for free, visit the Oxford Text Archive and download the zipped archive. Once you have downloaded the corpus, unzip it and place it preferably at the root of your hard drive. The original architecture of the BNC-XML directory should be modified so that all the text files are located in a single folder.

The path to the folder that contains the corpus files

At some point, the script will need the path to the directory that contains the tagged corpus files. If you have installed the corpus at the root of your hard drive, the path is likely to be the following:

  • C:\\BNC-XML\\texts (PC);
  • /BNC-XML/texts (Mac).

If you want to avoid retyping the path each time you run a query, save it in a text file somewhere to copy/paste it later on.

Launching the program

Download, install, and start R.

Before you run BNC.query(), I advise you to check your working directory. The working directory is a folder which you want R to read data from and store output into. This is where the script will save your data (4 files). To know what your current working directory is, enter:


To change the working directory, enter its path between quotes as an argument of setwd(). For example, if I want my working directory to be /Users/guillaumedesagulier, I enter:


Type the following lines, and press ENTER:


The first line clears R’s memory. The second line downloads the script. The third line launches the program.

Loading the required packages

The script needs four R packages:

  • gsubfn
  • ggplot2
  • plyr
  • stringi

If these packages are installed, the script proceeds to the next step. If they are not, the scripts downloads and installs them.

Formulating the query

First, the program prompts the user to input the corpus query.

Your can search for:

  • a simple lexeme (e.g. « hello »);
  • several simple lexemes (« hello » and « hi »);
  • a complex expression (e.g. « good morning »);
  • several complex expressions (e.g. « good morning », « good afternoon », « good evening »).

The query protocol is based on R-compatible regular expressions. For each of the above query, here is the corresponding expression:

word what to type in R
« hello » hello
« hello » and « hi » (hello|hi)
« good morning » good morning
« good morning », « good afternoon », « good evening » good (morning|afternoon|evening)

In the above code, | means « or ». For more information on character-string processing and regular expressions, see chapters 4 and 5 of Corpus Linguistics and Statistics with R.

Note that if you want to compare a one-word expression and a multiword expression (e.g. « hello » and « good morning »), you are going to have to run two separate queries.1

If you investigate a single lexeme or several simple lexemes, you may specify a POS tag. This option is not available for multiword expressions. Given that BNC-XML was tagged with CLAWS5, read the tagset and enter the correct tag (e.g. ITJ for an interjection, AJ0 for an adjective, etc.). If you do not want to bother looking for the tag, if you think the tag is irrelevant, or if the lexemes you investigate belong to different grammatical categories, enter \w+ (this is computer language for ‘one or more word characters’, i.e. any tag).

For the purpose of illustration, we explore non-standard plural agreement in the context of be (you/we/they was) in a comparative perspective. The search expression is the following:

(you|we|they) (was|were)

which translates as « you or we or they followed by a space followed by was or were« .

The script converts your search expression into XML…

Running the query over the whole corpus or just a sample

Next, the script prompts you to want to run the query over the whole spoken BNC-XML (908 files) or just a random sample (200 files). The latter is faster than the former and useful if you are not sure about the outcome of your query. Here, we enter 1 to run the query over the whole corpus.

The scripts prompts you to enter the path to the directory that contains the corpus files. If you work with a PC, don’t forget to double each backslash (see above). Because I work with a Mac, and because the corpus is installed at the root of my hard drive, I enter /BNC-XML/texts.

Select your operating system. This guarantees that the working files are saved with the proper specifications.

The script prints the paths to the corpus files. This is just to check that the path to the directory that contains them has been typed correctly. Press ENTER to launch the query.

The script loops over each corpus file. Sit back, relax, and wait. A progress bar indicates how much time you have to drink that coffee while the program is working. On my Mac, the script takes between 5 and 10 minutes to complete when all the corpus files are selected.2

Once the script has finished looping over the files, press ENTER to save the results in a text file named <interim.results.txt>.(( The file is also saved as an R data file named <interim.results.rds>. )) If you open the text file, you will see that there is one line per observation. Each word is described by the ID of the speaker who used it. Next, press ENTER to save the results in a text file named <>.3

After pressing ENTER, the first 20 lines of <> are displayed. Each observation consists of the speaker ID, the word or expression itself, the speaker’s gender, age group, and social grade. Read the BNC-XML documentation to know more about these variables.

You may now plot the results. Three options are available:

  • a barplot based on raw frequencies,
  • an association plot,
  • no plot (+ archive working files).

The association plot makes sense only when your study consists of at least two words or expressions.

For the working files to be archived means that they are time-stamped. This prevents the script from over-writing your working files when you run another query.

We choose the association plot. We can investigate three effects:

  • gender,
  • age,
  • social class.

We choose social class.

The script provides the table of counts with the variable(s) in the rows and the age groups in the columns. It also runs a χ2 test with simulated p-values (see Corpus Linguistics and Statistics with R, Sect. 8.9). If your table does not meet the test’s assumptions, an error message may be issued. This is likely to occur if you choose to work on a portion of the corpus only because the frequency counts might be too low.

The association plot below can be exported as an image or as pdf from R or RStudio. Each cell of the table of counts is represented by a tile whose area is proportional to the difference between observed and expected frequencies. The dotted line is the baseline. It represents independence between the variables. If the observed frequency of a cell is greater than its expected frequency, the tile appears above the baseline and is shaded black. If the observed frequency of a cell is smaller than its expected frequency, the tile appears below the baseline and is shaded red. One interesting finding here is that speakers from social group C2 (skilled working class) prefer the non-standard variant. (( For more details on how to interpret an association plot, see Corpus Linguistics and Statistics with R, pp. 184-185. ))

You now have the choice between running another query or exiting the program. If you choose to exit, the script adds a time stamp to your working files, for archiving purposes and later re-use. If my working files are time-stamped as follows:

  • <interim.results_08Jan2019141334.txt>, and 
  • <data.final_08Jan2019141334.txt>,

this means that they were created on January 8th, 2019 at 2.13pm (and 34 seconds!). Again, as long as you do not run the script from scratch again, your current working files are still intact.

Built-in plotting functions

If you want to make several plots, you can do so by using a built-in function. This will only work for the current session, i.e. as long as the working files have not been over-written. Here is the list of functions:

function call what it does
make.barplot() opens the barplot menu
barplot.gender() makes a barplot based on gender
barplot.age() makes a barplot based on age groups makes a barplot based on social grades
make.assocplot() opens the association plot menu
assplot.gender() makes an association plot based on gender
assplot.age() makes an association plot based on age groups
assplot.soc.class() makes an association plot based on social grades

The barplot below is obtained by entering It is made with ggplot2.

Video recap

The video below has been edited. In real life, the script takes slightly longer to run. The processing time depends on your hardware.

Cite this article as: Guillaume Desagulier, "BNC.query(). An interactive R script for a sociolinguistic exploration of the spoken component of the BNC-XML," in Around the word, 08/01/2019,
  1. If you want to plot the results, you can easily merge the output files later on. []
  2. MacBook Pro (13-inch, 2017); Processor: 3.3 GHz Intel Core i5; Memory: 16 GB 2133 MHz LPDDR3. []
  3. The file is also saved as an R data file named <>. []

Guillaume Desagulier

UMR 7114 MoDyCo — Université Paris 8, CNRS, Université Paris Nanterre, Institut Universitaire de France.

Vous aimerez aussi...

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *

Ce site utilise Akismet pour réduire les indésirables. En savoir plus sur comment les données de vos commentaires sont utilisées.