BNC.2014.query(). An interactive R script for a sociolinguistic exploration of the spoken component of the BNC-2014
Last edit: June 7th, 2019
BNC.2014.query() is an interactive R script that I wrote for a course in computational sociolinguistics last semester. It is designed to run queries over the BNC-2014 (spoken component). It extracts a single word or a complex expression, along with speaker information (gender, age class, and social grade), tabulates the results, computes frequencies, and makes a barplot or an association plot (along with a χ2 test).
For this demo, I have used R version 3.5.1 (2018-07-02) — “Feather Spray” on RStudio (Version 1.1.463). I cannot guarantee that the script will work on previous versions.
Obtaining the corpus
For the script to run properly, the BNC-2014 files must be installed on your system. To download the BNC-2014 for free, visit this page and complete the signup form at the bottom of the page. Once you have downloaded the corpus, unzip it and place it preferably at the root of your hard drive. The architecture of the bnc2014spoken directory is displayed below.
The path to the folder that contains the corpus files
At some point, the script will need the path to the directory that contains the tagged corpus files. If you have installed the corpus at the root of your hard drive, the path is likely to be the following:
If you want to avoid retyping the path each time you run a query, save it in a text file somewhere to copy/paste it later on.
Launching the program
Download, install, and start R.
Before you run BNC.2014.query(), I advise you to check your working directory. The working directory is a folder which you want R to read data from and store output into. This is where the script will save your data (4 files). To know what your current working directory is, enter:
To change the working directory, enter its path between quotes as an argument of
setwd(). For example, if I want my working directory to be
/Users/guillaumedesagulier, I enter:
Type the following lines, and press ENTER:
The first line clears R’s memory. The second line downloads the script. The third line launches the program.
Loading the required packages
The script needs four R packages:
If these packages are installed, the script proceeds to the next step. If they are not, the scripts downloads and installs them.
Formulating the query
First, the program prompts the user to input the corpus query.
Your can search for:
- a simple lexeme (e.g. “hello”);
- several simple lexemes (“hello” and “hi”);
- a complex expression (e.g. “good morning”);
- several complex expressions (e.g. “good morning”, “good afternoon”, “good evening”).
The query protocol is based on R-compatible regular expressions. For each of the above query, here is the corresponding expression:
|word||what to type in R|
|“hello” and “hi”||
|“good morning”, “good afternoon”, “good evening”||
In the above code,
| means “or”. For more information on character-string processing and regular expressions, see chapters 4 and 5 of Corpus Linguistics and Statistics with R.
For the purpose of illustration, we compare the use of “hello” and “hi”. The search expression is
Note that if you want to compare a one-word expression and a multiword expression (e.g. “hello” and “good morning”), you are going to have to run two separate queries.1
If you investigate a single lexeme or several simple lexemes, you may specify a POS tag. This option is not available for multiword expressions. Given that BNC-2014 was tagged with CLAWS7, read the tagset and enter the correct tag. Both “hello” and “hi” are interjections. The CLAWS7 tag for interjections is
UH. If you do not want to bother looking for the tag, if you think the tag is irrelevant, or if the lexemes you investigate belong to different grammatical categories, enter
\w+ (this is computer language for ‘one or more word characters’, i.e. any tag).
The script converts your search expression into XML…
Entering the path to the directory that contains the corpus files
Next, the scripts prompts you to enter the path to the directory that contains the corpus files. If you work with a PC, don’t forget to double each backslash (see above).
Running the query over the whole corpus or just a sample
After entering the path to the corpus files, the script asks you if you want to run the query over the whole corpus (1251 files) or just a sample (100 files). The latter is much faster than the former and useful if you are not sure about the outcome of your query. Here, we enter 1 to run the query over the whole corpus.
The script prints the paths to a random selection of ten corpus files. This is just to check that the path to the directory that contains them has been typed correctly. Press ENTER to launch the query.
Now, the script loops over each corpus file. Sit back, relax, and wait. A progress bar indicates how much time you have to drink that coffee while the program is working.
On my Mac, the script takes between 5 and 10 minutes to complete when all the corpus files are selected.2
Once the script has finished looping over the files, press ENTER to save the results in a text file named <interim.results.BNC.2014.txt>.(( The file is also saved as an R data file named <interim.results.BNC.2014.rds>. )) Here is a snaspshot of the file. There is one line per observation. Each word is described by the ID of the speaker who used it.
Next, press ENTER to save the results in a text file named <data.final.BNC.2014.txt>.3
The first 20 lines of <data.final.BNC.2014.txt> are displayed. Each observation consists of the speaker ID, the word itself, the age group, the gender, the speaker’s city, the speaker’s dialect, and the social grade. Read the BNC-2014 documentation to know more about these variables.
You may now plot the results. Three options are available:
- a barplot based on raw frequencies,
- an association plot,
- no plot (+ archive working files).
The association plot makes sense only when your study consists of at least two words or expressions.
For the working files to be archived means that they are time-stamped. This prevents the script from over-writing your working files when you run another query.
We choose the association plot. We can investigate three effects:
- social class.
We choose age.
The script provides the table of counts with the variable(s) in the rows and the age groups in the columns. It also runs a χ2 test. If your table does not meet the test’s assumptions, an error message may be issued. This is likely to occur if you choose to work on a portion of the corpus only. (see Corpus Linguistics and Statistics with R, Sect. 8.9)
The association plot below can be exported as an image or as pdf from R or RStudio. Each cell of the table of counts is represented by a tile whose area is proportional to the difference between observed and expected frequencies. The dotted line is the baseline. It represents independence between the variables. If the observed frequency of a cell is greater than its expected frequency, the tile appears above the baseline and is shaded black. If the observed frequency of a cell is smaller than its expected frequency, the tile appears below the baseline and is shaded red. We see that speakers from age groups 19-29 and, to a much lesser extent 60-69, prefer hi over hello.4
You now have the choice between running another query or exiting the program. If you choose to exit, the script adds a time stamp to your working files, for archiving purposes and later re-use. If my working files are time-stamped as follows:
- <interim.results.BNC.2014_29Dec2018121603.txt>, and
this means that they were created on December 29th, 2018 at 12.16pm (and 3 seconds!). Again, as long as you do not run the script from scratch again, your current working files are still intact.
Built-in plotting functions
If you want to make several plots, you can do so by using a built-in function. This will only work for the current session, i.e. as long as the working files have not been over-written. Here is the list of functions:
|function call||what it does|
||opens the barplot menu|
||makes a barplot based on gender|
||makes a barplot based on age groups|
||makes a barplot based on social grades|
||opens the association plot menu|
||makes an association plot based on gender|
||makes an association plot based on age groups|
||makes an association plot based on social grades|
The barplot below is obtained by entering
barplot.age(). It is made with
The video below has been edited. In real life, the script takes slightly longer to run. The processing time depends on your hardware.
This program is still in beta. If you experience bugs or think of better or faster functionalities, please leave a comment.
Citing the program
If you use the program for your research, please cite it as follows:
Desagulier, Guillaume. 2019. BNC.2014.query(), v. 0.3. An R script for a sociolinguistic exploration of the spoken component of the BNC 2014.
This program is protected by the following Creative Commons licence: CC-BY-NC-ND 4.0.
Marsaz, Drôme, France – Jan 2019
- If you want to plot the results, you can easily merge the output files later on. [↩]
- MacBook Pro (13-inch, 2017); Processor: 3.3 GHz Intel Core i5; Memory: 16 GB 2133 MHz LPDDR3. [↩]
- The file is also saved as an R data file named <data.final.BNC.2014.rds>. [↩]
- For more details on how to interpret an association plot, see Corpus Linguistics and Statistics with R, pp. 184-185. [↩]
Could you please add the metadata category about the speaker’s educational background to the R code? (as we’ve talked via Twitter :)! ) Thank you! Look forward to it. Your code has been very useful for me to study the target lexical item with reference to the sociolinguistic info behind!
Here is a beta of v0.3. Please tell me if this works.
I need to use the quotation marks: “” instead of «» to run the code.
But apart from that, it works perfectly!!!! Thanks so much 🙂 You’ve been very helpful and this new version just helps me overcome the difficulties I have lately encountered in my research on Spoken BNC 2014
BTW, for my case, it needs to be the following:
I mean the double quotation marks. Seems like here we are having restrictions of font types so my comment couldn’t be able to show it hahaha
Thanks for such a useful resource! I followed the instruction and did my own search. My word of interest was “bloody”. It was all good and smooth until R asked me “Do you want to run a query over the whole spoken BNC 2014 or just a sample?”. Having chosen either option, I got the error feedback: “C stack usage 8591217 is too close to the limit” or “C stack usage 10524960 is too close to the limit”. I was wondering if was because I had not defined the POS tag of the word “bloody” ( I just put “\w+” there). So I decided to indicate its POS info this time. I worked! Then I chose to run a query over a random sample (100 files). In the final steps, I chose to barplot the social class of the use of “bloody”. R did return a bar chart for me. But I found the frequency info odd. Some of the social grades reached more than 5000 hits. The problem is that if you search “bloody” in CQP web (https://cqpweb.lancs.ac.uk/bnc2014spoken/), the query “bloody” only returned 1,459 matches in total. Did I make mistakes or misunderstand the instruction?
Looking forward to your reply!
You’re right, the frequency counts were messed up because of a loop that didn’t work as expected. I have fixed the script. It should work now, although it takes much longer to run (because the faulty loop is now working, and loops are slow). I plan to fix this in the near future. Thanks for your interest and your feedback!
And I’ve just fixed the speed issue. The script is now much faster. Hope this helps.