Tilting at windmills with text-mining techniques (part 1)
Two years ago, I gave a mini workshop on text-mining techniques at a one-day conference on philology and the digital humanities at Paris 8 University. The conference was organized by colleagues from the Department of Spanish Studies.
Besides linguistics, other disciplines in the digital humanities tap into corpora by means of text-mining techniques: history, literature, stylistics, etc. Literature experts might want to know more about the style of an author or the linguistic trends of a period. Linguists make use of textual databases to know more about a language or about the speakers of this language.
As a linguist, I offered to contribute to the conference by applying some of the techniques that I use in the study of meaning to a monument of Spanish literature: El Ingenioso Hidalgo Don Quijote de la Mancha, by Miguel de Cervantes Saavedra. The book has two parts, which appeared in 1605 and 1615 respectively.
To my great surprise, no digital version with XML was available on the Web, although I did find scanned prints. Although pleasant to the eye, these are not of much use to the corpus linguist.
I used the digital version from Project Gutenberg instead.
The raw-text version comes without:
- page numbering (although the original chaptering is preserved);
- XML markup;
- annotation (syntactic or semantic).
Armed with R, and a handful of packages (see below), we can gain some insights into the story and its characters.
Some of the methods that I propose below are explained in great detail by Matthew Jockers in his book Text Analysis with R for Students of Literature (Springer, 2014).
Load the text
Copy and paste the text from Project Gutenberg into a text file (preferably using Notepad++ for Windows or TextWrangler for Mac). Name the file don_quijote.txt and save it in a convenient location on your hard drive (for example the root drive).
Each text comes with a legal header and a legal footer. Delete the header, i.e. the text between « The Project Gutenberg EBook of Don Quijote, by Miguel de Cervantes Saavedra… » and « …Text
file corrections and new HTML file by Joaquin Cuenca Abela. » Delete the footer, i.e. the text between « End of Project Gutenberg’s Don Quijote, by Miguel de Cervantes Saavedra. » and « … how to
subscribe to our email newsletter to hear about new eBooks. » Save the changes in the file.
The code below clears R’s memory and launches an interactive window asking you to select the file. Choose don_quijote.txt. The text is loaded into a character vector named dq.vector. Inspect the first 50 elements of the vector with head(). There are as many elements as line breaks. I suppose the line breaks correspond to those found in the edition that the Project Gutenberg volunteer found in the Don Quixote edition that s/he used for the transcription.
# clear R's memory rm(list=ls(all=TRUE) # load the text dq.vector <- scan(file=file.choose(), what="character", sep="\n", fileEncoding="UTF-8") # inspect the text head(text.v, 50)
Another way of loading the text consists in entering the path to the text file. Supposing you stored don_quijote.txt in the folder named text_mining at the root of your hard drive, the path will be /text_mining/don_quijote.txt for Mac users and C:\\text_mining\\don_quijote.txt for windows users. The scan() function should therefore look like the following.
# Mac users dq.vector <- scan("/text_mining/don_quijote.txt", what="character", sep="\n", fileEncoding="UTF-8") # Windows users dq.vector <- scan("C:\\text_mining\\don_quijote.txt", what="character", sep="\n", fileEncoding="UTF-8")
Word tokens and word types
The cat is on the mat
contains six word tokens but five word types. This is because the sentence has six word occurrences, but only five different words. The word the occurs twice.
To count the number of word tokens and word types in the novel, we need to split the text into words. Because a computer does not know what a word is, we need to define it. How you define a word is not straightforward because it is language-specific. Typologist Martin Haspelmath goes as far as to claim that there are no good morphosyntactic criteria for defining the word in general.1
The question of what we should call a word is challenging even for morphosyntactically simple languages. For example, the English phrase
a state-of-the-art method
consists of 3 words if you decide that state-of-the-art counts as one word, and 6 words if you break the compound expression into its four constituents (state + of + the + art). If you require the service of a professional translator, who charges by the word, the question is worth asking.
Prior to splitting the novel vector into words, we remove line breaks so as to convert the multiple-element vector into a single-element vector. Technically, we paste the vector elements of dq.vector, which are all delimited by line brakes, and separate them with a space (collapse= » « )
dq.vector.one.line <- paste(dq.vector, collapse=" ")
dq.vector.one.line.lc <- tolower(dq.vector.one.line)
dq.words <- unlist(strsplit(dq.vector.one.line.lc, "\\W")) head(dq.words, 10) # index
non.empty <- which(dq.words!="") dq.words <- dq.words[non.empty]
# word tokens length(dq.words) # word types length(unique(dq.words))
Making a frequency list
You generate a frequency list when you want to know how often words occur in a corpus. A frequency list is a two-column table with all word types occurring in the corpus in one column and their type frequency in the other column. A frequency list is useful when you want to know what a text is mostly about. The principle of frequency lists is rooted is the distinction between type and token (see above).
dq.freqs.t <- table(dq.words) sorted.dq.freqs <- sort(dq.freqs.t, decreasing=TRUE) head(sorted.dq.freqs, 20) # inspect the first 20 elements
Plotting a frequency list
par(cex.axis=.7, cex.lab=.7) plot(sorted.dq.freqs[1:50], type="h", xlab="words", ylab="proportion", xaxt ="n") axis(1,1:50, labels=names(sorted.dq.freqs[1:50]), las=2)
plot(sorted.dq.freqs, type="l", xlab="words", ylab="proportion", xaxt="n")
The curve decreases from its peak rapidly and continuously. This is typical of the distribution of words in natural languages. If you plot the frequency list of any corpus of natural language, the curve will look invariably the same providing the corpus is large enough.
This distribution is Zipfian. It is named after Zipf’s law: a large number of rare words coexist with a small number of highly frequent words.3 I will dedicate a full post to this law, and the debates it has triggered.
To be continued!
- Haspelmath, M. (2011). The indeterminacy of word segmentation and the nature of morphology and syntax. Folia Linguistica, 45(1), 31-80.
- strsplit() outputs a list. To convert the list into a character vector, we use unlist().
- Zipf, George K. (1949). Human Behavior and the Principle of Least Effort. Cambridge: Addison-Wesley.