The spoken component of the British National Corpus 2014 is out!
British National Corpus 2014 is a project led by the Centre for Corpus Approaches to Social Science at Lancaster University to create a 100M word corpus of contemporary British English, the BNC-XML, which is now over 20 years old. On November 19th, 2018, the spoken component of the BNC 2014 was made available for download for offline analysis. Before then, it was available via Lancaster University’s CQPweb. It is now accessible online in full, free of charge.
The 11.5-million-word spoken component of the BNC2014 consists of transcripts of recorded conversations involving 672 speakers from different parts of the UK between 2012 and 2016. The corpus breaks down into 1,251 files, i.e. one per conversation.
The ‘old’ BNC
I remember how thrilled my syntax professor was, back in 1999, when he announced that my department had purchased a CD-Rom copy of the original British National Corpus (BNC 1.0 if I remember correctly). When we asked what we could do with it, he replied: “Well, I’m sure we can glean some interesting examples.” He was right, of course, but browsing the corpus in search of interesting examples, just like one would chase butterflies with a net, was not even the tip of the iceberg of things to do with such a textual treasure trove. It is not until years later, when I became a corpus linguist, armed with the proper techniques, that I realized the full range of queries, extractions, tabulations, and quantifications that came with the mining of these raw files. I remember purchasing the BNC-XML as a CD-ROM shortly after its release in 2007. It is now available for download in full, free of charge from the Oxford Text Archive, along with other versions (BNC sampler and baby edition).
The BNC-XML is large (4,049 corpus files, about 100M word tokens) and annotated: it consists of more than the actual words in the document. Each corpus file comes with a TEI header, which provides a rich and structured description of its contents (mode, topic, genre, subgenre, date of original production, author/speaker, encoding, revisions, etc.). This information makes the file machine-processable.
Each word is tagged in a XML fashion, following the CLAWS5 tagset. Here is sentence 126 from file A00.xml:
Home care Coordinator, Margaret Gillies, currently has a team of 20 volunteers from a variety of churches providing practical help to a number of clients already referred.
Now, here is what it looks like once annotated in XML:
<s n="126"><w c5="NN1" hw="home" pos="SUBST">Home </w><w c5="NN1-VVB" hw="care" pos="SUBST">care </w><w c5="NN1" hw="coordinator" pos="SUBST">Coordinator</w><c c5="PUN">, </c><w c5="NP0" hw="margaret" pos="SUBST">Margaret </w><w c5="NP0" hw="gillies" pos="SUBST">Gillies</w><c c5="PUN">, </c><w c5="AV0" hw="currently" pos="ADV">currently </w><w c5="VHZ" hw="have" pos="VERB">has </w><w c5="AT0" hw="a" pos="ART">a </w><w c5="NN1" hw="team" pos="SUBST">team </w><w c5="PRF" hw="of" pos="PREP">of </w><w c5="CRD" hw="20" pos="ADJ">20 </w><w c5="NN2" hw="volunteer" pos="SUBST">volunteers </w><w c5="PRP" hw="from" pos="PREP">from </w><w c5="AT0" hw="a" pos="ART">a </w><w c5="NN1" hw="variety" pos="SUBST">variety </w><w c5="PRF" hw="of" pos="PREP">of </w><w c5="NN2" hw="church" pos="SUBST">churches </w><w c5="VVG" hw="provide" pos="VERB">providing </w><w c5="AJ0" hw="practical" pos="ADJ">practical </w><w c5="NN1" hw="help" pos="SUBST">help </w><w c5="PRP" hw="to" pos="PREP">to </w><w c5="AT0" hw="a" pos="ART">a </w><w c5="NN1" hw="number" pos="SUBST">number </w><w c5="PRF" hw="of" pos="PREP">of </w><w c5="NN2" hw="client" pos="SUBST">clients </w><w c5="AV0" hw="already" pos="ADV">already </w><w c5="VVN" hw="refer" pos="VERB">referred</w><c c5="PUN">.</c></s>
Each word is delimited by a start tag (
<w...>) and a closing tag (
</w>). The start tag contains:
- a code letter (
- a POS tag based on the CLAWS5 tagset (e.g.
AV0for an adverb);
- the head word (
hw), or lemma;
- the value of the CLAWS5 wordclass (e.g.
ADVfor an adverb).
Put together, these annotations yield a file that looks like the following:
For the above reasons, the BNC-XML has been one of my favorite corpus for many years. See chapters 3, 4, and 5 of my book to see what you can do with XML markup and tagging.
However, the BNC-XML is getting old. The texts and transcripts that the BNC-XML is made of were produced between the 1960s and 1993. There is nothing inherently wrong with using an ageing corpus, of course, but English has evolved since then, and an update is never a bad thing.
So what’s new?
The new BNC is meant to be an updated version of its predecessor with a target objective of 100M words, once the written component is appended to the current project. The structure is therefore very similar, except this time the metadata are kept apart from the corpus files.
The corpus comes in two flavors: tagged and untagged. The structure of the tagged corpus files is slightly different from what we had in the original BNC. This time, we have one word per line. Below is a screenshot from file S2A5-tgd.xml.
<u ...> </u> tag delimits utterances. The word tags have changed too. The start tag contains:
BNC 2014 frequency lists
In future posts, I will present concrete illustrations of what can be done with this corpus. In the meantime, here are two BNC2014 frequency lists: one unlemmatized, the other lemmatized. The code that was used to make these freqlists is from my book ;-).
Love, R., Dembry, C., Hardie, A., Brezina, V., & McEnery, T. (2017). The Spoken BNC2014: Designing and building a spoken corpus of everyday conversations. International Journal of Corpus Linguistics 22:3 (2017), 319–344. doi 10.1075/ijcl.22.3.02lov