BNC.query(). An interactive R script for a sociolinguistic exploration of the spoken component of the BNC-XML
BNC.query() is an interactive R script that I wrote for a course in computational sociolinguistics last semester. It is designed to run queries over the BNC-XML (spoken component). It extracts a single word or a complex expression, along with speaker information (gender, age class, and social grade), tabulates the results, computes frequencies, and makes a barplot or an association plot (along with a χ2 test).
For this demo, I have used R version 3.5.1 (2018-07-02) — « Feather Spray » on RStudio (Version 1.1.463). I cannot guarantee that the script will work on previous versions.
Obtaining the corpus
For the script to run properly, the BNC-XML files must be installed on your system. To download the BNC-XML for free, visit the Oxford Text Archive and download the zipped archive. Once you have downloaded the corpus, unzip it and place it preferably at the root of your hard drive. The original architecture of the BNC-XML directory should be modified so that all the text files are located in a single folder.
The path to the folder that contains the corpus files
At some point, the script will need the path to the directory that contains the tagged corpus files. If you have installed the corpus at the root of your hard drive, the path is likely to be the following:
If you want to avoid retyping the path each time you run a query, save it in a text file somewhere to copy/paste it later on.
Launching the program
Download, install, and start R.
Before you run BNC.query(), I advise you to check your working directory. The working directory is a folder which you want R to read data from and store output into. This is where the script will save your data (4 files). To know what your current working directory is, enter:
To change the working directory, enter its path between quotes as an argument of
setwd(). For example, if I want my working directory to be
/Users/guillaumedesagulier, I enter:
Type the following lines, and press ENTER:
The first line clears R’s memory. The second line downloads the script. The third line launches the program.
Loading the required packages
The script needs four R packages:
If these packages are installed, the script proceeds to the next step. If they are not, the scripts downloads and installs them.
Formulating the query
First, the program prompts the user to input the corpus query.
Your can search for:
- a simple lexeme (e.g. « hello »);
- several simple lexemes (« hello » and « hi »);
- a complex expression (e.g. « good morning »);
- several complex expressions (e.g. « good morning », « good afternoon », « good evening »).
The query protocol is based on R-compatible regular expressions. For each of the above query, here is the corresponding expression:
|word||what to type in R|
|« hello »||
|« hello » and « hi »||
|« good morning »||
|« good morning », « good afternoon », « good evening »||
In the above code,
| means « or ». For more information on character-string processing and regular expressions, see chapters 4 and 5 of Corpus Linguistics and Statistics with R.
Note that if you want to compare a one-word expression and a multiword expression (e.g. « hello » and « good morning »), you are going to have to run two separate queries.1
If you investigate a single lexeme or several simple lexemes, you may specify a POS tag. This option is not available for multiword expressions. Given that BNC-XML was tagged with CLAWS5, read the tagset and enter the correct tag (e.g.
ITJ for an interjection,
AJ0 for an adjective, etc.). If you do not want to bother looking for the tag, if you think the tag is irrelevant, or if the lexemes you investigate belong to different grammatical categories, enter
\w+ (this is computer language for ‘one or more word characters’, i.e. any tag).
For the purpose of illustration, we explore non-standard plural agreement in the context of be (you/we/they was) in a comparative perspective. The search expression is the following:
which translates as « you or we or they followed by a space followed by was or were« .
The script converts your search expression into XML…
Running the query over the whole corpus or just a sample
Next, the script prompts you to want to run the query over the whole spoken BNC-XML (908 files) or just a random sample (200 files). The latter is faster than the former and useful if you are not sure about the outcome of your query. Here, we enter 1 to run the query over the whole corpus.
The scripts prompts you to enter the path to the directory that contains the corpus files. If you work with a PC, don’t forget to double each backslash (see above). Because I work with a Mac, and because the corpus is installed at the root of my hard drive, I enter
Select your operating system. This guarantees that the working files are saved with the proper specifications.
The script prints the paths to the corpus files. This is just to check that the path to the directory that contains them has been typed correctly. Press ENTER to launch the query.
The script loops over each corpus file. Sit back, relax, and wait. A progress bar indicates how much time you have to drink that coffee while the program is working. On my Mac, the script takes between 5 and 10 minutes to complete when all the corpus files are selected.2
Once the script has finished looping over the files, press ENTER to save the results in a text file named <interim.results.txt>.(( The file is also saved as an R data file named <interim.results.rds>. )) If you open the text file, you will see that there is one line per observation. Each word is described by the ID of the speaker who used it. Next, press ENTER to save the results in a text file named <data.final.txt>.3
After pressing ENTER, the first 20 lines of <data.final.txt> are displayed. Each observation consists of the speaker ID, the word or expression itself, the speaker’s gender, age group, and social grade. Read the BNC-XML documentation to know more about these variables.
You may now plot the results. Three options are available:
- a barplot based on raw frequencies,
- an association plot,
- no plot (+ archive working files).
The association plot makes sense only when your study consists of at least two words or expressions.
For the working files to be archived means that they are time-stamped. This prevents the script from over-writing your working files when you run another query.
We choose the association plot. We can investigate three effects:
- social class.
We choose social class.
The script provides the table of counts with the variable(s) in the rows and the age groups in the columns. It also runs a χ2 test with simulated p-values (see Corpus Linguistics and Statistics with R, Sect. 8.9). If your table does not meet the test’s assumptions, an error message may be issued. This is likely to occur if you choose to work on a portion of the corpus only because the frequency counts might be too low.
The association plot below can be exported as an image or as pdf from R or RStudio. Each cell of the table of counts is represented by a tile whose area is proportional to the difference between observed and expected frequencies. The dotted line is the baseline. It represents independence between the variables. If the observed frequency of a cell is greater than its expected frequency, the tile appears above the baseline and is shaded black. If the observed frequency of a cell is smaller than its expected frequency, the tile appears below the baseline and is shaded red. One interesting finding here is that speakers from social group C2 (skilled working class) prefer the non-standard variant. (( For more details on how to interpret an association plot, see Corpus Linguistics and Statistics with R, pp. 184-185. ))
You now have the choice between running another query or exiting the program. If you choose to exit, the script adds a time stamp to your working files, for archiving purposes and later re-use. If my working files are time-stamped as follows:
- <interim.results_08Jan2019141334.txt>, and
this means that they were created on January 8th, 2019 at 2.13pm (and 34 seconds!). Again, as long as you do not run the script from scratch again, your current working files are still intact.
Built-in plotting functions
If you want to make several plots, you can do so by using a built-in function. This will only work for the current session, i.e. as long as the working files have not been over-written. Here is the list of functions:
|function call||what it does|
||opens the barplot menu|
||makes a barplot based on gender|
||makes a barplot based on age groups|
||makes a barplot based on social grades|
||opens the association plot menu|
||makes an association plot based on gender|
||makes an association plot based on age groups|
||makes an association plot based on social grades|
The barplot below is obtained by entering
barplot.social.class(). It is made with
The video below has been edited. In real life, the script takes slightly longer to run. The processing time depends on your hardware.
- If you want to plot the results, you can easily merge the output files later on.
- MacBook Pro (13-inch, 2017); Processor: 3.3 GHz Intel Core i5; Memory: 16 GB 2133 MHz LPDDR3.
- The file is also saved as an R data file named <data.final.rds>.