Tilting at windmills with text-mining techniques (part 1)

Two years ago, I gave a mini workshop on text-mining techniques at a one-day conference on philology and the digital humanities at Paris 8 University. The conference was organized by colleagues from the Department of Spanish Studies.

Besides linguistics, other disciplines in the digital humanities tap into corpora by means of text-mining techniques: history, literature, stylistics, etc. Literature experts might want to know more about the style of an author or the linguistic trends of a period. Linguists make use of textual databases to know more about a language or about the speakers of this language.

As a linguist, I offered to contribute to the conference by applying some of the techniques that I use in the study of meaning to a monument of Spanish literature: El Ingenioso Hidalgo Don Quijote de la Mancha, by Miguel de Cervantes Saavedra. The book has two parts, which appeared in 1605 and 1615 respectively.

To my great surprise, no digital version with XML was available on the Web, although I did find scanned prints. Although pleasant to the eye, these are not of much use to the corpus linguist.


The first page of Don Quixote (Biblioteca Virtual Miguel de Cervantes, http://www.cervantesvirtual.com/)

I used the digital version from Project Gutenberg instead.

The first sentences of Don Quixote in raw text (Project Gutenberg).

The raw-text version comes without:

  • page numbering (although the original chaptering is preserved);
  • XML markup;
  • tokenization;
  • lemmatization;
  • annotation (syntactic or semantic).

Armed with R, and a handful of packages (see below), we can gain some insights into the story and its characters.

Some of the methods that I propose below are explained in great detail by Matthew Jockers in his book Text Analysis with R for Students of Literature (Springer, 2014).

Load the text

Copy and paste the text from Project Gutenberg into a text file (preferably using Notepad++ for Windows or TextWrangler for Mac). Name the file don_quijote.txt and save it in a convenient location on your hard drive (for example the root drive).

Each text comes with a legal header and a legal footer. Delete the header, i.e. the text between “The Project Gutenberg EBook of Don Quijote, by Miguel de Cervantes Saavedra…” and “…Text
file corrections and new HTML file by Joaquin Cuenca Abela.” Delete the footer, i.e. the text between “End of Project Gutenberg’s Don Quijote, by Miguel de Cervantes Saavedra.” and “… how to
subscribe to our email newsletter to hear about new eBooks.” Save the changes in the file.

The code below clears R’s memory and launches an interactive window asking you to select the file. Choose don_quijote.txt. The text is loaded into a character vector named dq.vector. Inspect the first 50 elements of the vector with head(). There are as many elements as line breaks. I suppose the line breaks correspond to those found in the edition that the Project Gutenberg volunteer found in the Don Quixote edition that s/he used for the transcription.

# clear R's memory
# load the text
dq.vector <- scan(file=file.choose(), what="character", sep="\n", fileEncoding="UTF-8")
# inspect the text
head(text.v, 50)

Another way of loading the text consists in entering the path to the text file. Supposing you stored don_quijote.txt in the folder named text_mining at the root of your hard drive, the path will be /text_mining/don_quijote.txt for Mac users and C:\\text_mining\\don_quijote.txt for windows users. The scan() function should therefore look like the following.

# Mac users
dq.vector <- scan("/text_mining/don_quijote.txt", what="character", sep="\n", fileEncoding="UTF-8")
# Windows users
dq.vector <- scan("C:\\text_mining\\don_quijote.txt", what="character", sep="\n", fileEncoding="UTF-8")
Word tokens and word types

The sentence

The cat is on the mat

contains six word tokens but five word types. This is because the sentence has six word occurrences, but only five different words. The word the occurs twice.

To count the number of word tokens and word types in the novel, we need to split the text into words. Because a computer does not know what a word is, we need to define it. How you define a word is not straightforward because it is language-specific. Typologist Martin Haspelmath goes as far as to claim that there are no good morphosyntactic criteria for defining the word in general.1

The question of what we should call a word is challenging even for morphosyntactically simple languages. For example, the English phrase

a state-of-the-art method

consists of 3 words if you decide that state-of-the-art counts as one word, and 6 words if you break the compound expression into its four constituents (stateof + the + art). If you require the service of a professional translator, who charges by the word, the question is worth asking.

Prior to splitting the novel vector into words, we remove line breaks so as to convert the multiple-element vector into a single-element vector. Technically, we paste the vector elements of dq.vector, which are all delimited by line brakes, and separate them with a space (collapse=” “)

dq.vector.one.line <- paste(dq.vector, collapse=" ")
Coming back to the The cat is on the mat example, let us not forget that R will make a difference between The (with capital T) and the (all lowercase). To prevent R from seeing two distinct words when in fact they belong to the same type, we switch the whole novel to lowercase.
dq.vector.one.line.lc <- tolower(dq.vector.one.line)
Next, we use the strsplit() function to split the text into “words” anytime it finds a non-word character, which we translate as \\W.2
dq.words <- unlist(strsplit(dq.vector.one.line.lc, "\\W"))
head(dq.words, 10) # index
 Upon inspection, we see that there are empty strings (“”) due to the choice of \\W as a splitter. We remove them in two steps. First, we tell R to find the positions of the empty strings in the vector with which() and the logical operator != (which means ‘not’). Second, we tell R to keep the positions of the non-empty strings via subsetting.
non.empty <- which(dq.words!="")
dq.words <- dq.words[non.empty]
We can now easily count the number of word tokens and word types in Don Quixote.
# word tokens
# word types 
There are 381352 word tokens and 22942 word types (i.e. unique words) in the novel.
Making a frequency list

You generate a frequency list when you want to know how often words occur in a corpus. A frequency list is a two-column table with all word types occurring in the corpus in one column and their type frequency in the other column. A frequency list is useful when you want to know what a text is mostly about. The principle of frequency lists is rooted is the distinction between type and token (see above).

With table(), we generate the frequency list. We sort it in decreasing order with sort().
dq.freqs.t <- table(dq.words)
sorted.dq.freqs <- sort(dq.freqs.t, decreasing=TRUE)
head(sorted.dq.freqs, 20) # inspect the first 20 elements
Inevitably, the top of the list consists mostly of function words (determiners, prepositions, conjunctions, etc.):
You will observe this phenomenon regardless of the text you make a frequently list from.
Plotting a frequency list
You plot the 50 most frequent words as follows.
par(cex.axis=.7, cex.lab=.7)
plot(sorted.dq.freqs[1:50], type="h", xlab="words", ylab="proportion", xaxt ="n")
axis(1,1:50, labels=names(sorted.dq.freqs[1:50]), las=2)

The 50 most frequent words in Don Quixote

 Now, see what happens when you plot the whole frequency list.
plot(sorted.dq.freqs, type="l", xlab="words", ylab="proportion", xaxt="n")

Plotting the full frequency list in Don Quixote

The curve decreases from its peak rapidly and continuously. This is typical of the distribution of words in natural languages. If you plot the frequency list of any corpus of natural language, the curve will look invariably the same providing the corpus is large enough.

This distribution is Zipfian. It is named after Zipf’s law: a large number of rare words coexist with a small number of highly frequent words.3 I will dedicate a full post to this law, and the debates it has triggered.

To be continued!

Cite this article as: Guillaume Desagulier, "Tilting at windmills with text-mining techniques (part 1)," in Around the word, 22/11/2017, https://corpling.hypotheses.org/125.
  1. Haspelmath, M. (2011). The indeterminacy of word segmentation and the nature of morphology and syntaxFolia Linguistica45(1), 31-80. []
  2. strsplit() outputs a list. To convert the list into a character vector, we use unlist() []
  3. Zipf, George K. (1949). Human Behavior and the Principle of Least Effort. Cambridge: Addison-Wesley. []

Guillaume Desagulier

UMR 7114 MoDyCo — Université Paris 8, CNRS, Université Paris Nanterre, Institut Universitaire de France.

Vous aimerez aussi...

Laisser un commentaire

Votre adresse e-mail ne sera pas publiée. Les champs obligatoires sont indiqués avec *

Ce site utilise Akismet pour réduire les indésirables. En savoir plus sur comment les données de vos commentaires sont utilisées.