‘Literally’ the most interesting project ever!
What happens when you ask fifteen Master’s students to annotate the pragmatic uses of a single adverb across eight thousand historical sentences, then train a neural network on their collective judgments? This post chronicles a three-week-long annotation project on the adverb literally, conducted as part of a Pragmatics MA seminar at the University of Bordeaux Montaigne. The project, from initial inter-annotator agreement woes to the deployment of XLM-RoBERTa on unannotated data, was grounded in Kostadinova’s (2018) corpus-based work on American English speakers’ awareness of the different uses of literally. Along the way, we confronted the usual suspects of corpus annotation: the granularity problem, the reliability-validity trade-off, and the pedagogical challenge of making abstract linguistic categories operational.
The linguistic phenomenon: literally and semantic bleaching
The adverb literally has become a poster child for semantic change and pragmatic drift. Its basic, etymological meaning is “in a literal manner, or sense (…) in the very words, word for word” (OED), as in She translated the text literally. However, literally has undergone what linguists call semantic bleaching, i.e., a gradual loss of semantic content accompanied by pragmatic strengthening. Contemporary usage abounds with non-literal instances: I literally died laughing, The exam was literally impossible, They were literally drowning in paperwork.

These non-literal uses have attracted prescriptivist ire and descriptivist fascination in equal measure. From a pragmatic perspective, literally in such contexts functions as an intensifier or emphatic marker, which amplifies the force of hyperbolic statements rather than asserting their truth-conditional content. This functional shift raises interesting questions about the conventionalization of implicature and the boundaries between semantics and pragmatics.
Kostadinova (2018) shows that American English speakers are quite aware of the different uses of literally and can articulate why they employ it in various contexts. Through a combination of corpus analysis and questionnaire data, she proves that speakers consciously distinguish between what she calls the “basic” meaning (literal, word-for-word interpretation) and the “non-basic” meaning (intensifying, emphatic function). Importantly, speakers do not perceive the non-basic use as incorrect or degraded. Rather, they recognize it as serving distinct pragmatic functions in discourse. The fact that speakers are metalinguistically aware of usage specificities provided the theoretical foundation for our annotation project: if speakers themselves recognize these functional distinctions, annotators should be able to operationalize them, at least in principle.
The challenge for corpus annotation lies precisely in this gradient nature of meaning. How do we operationalize the distinction between “basic literal” and “non-literal” uses when confronted with thousands of attested examples? And what happens when Kostadinova (2018) introduces an intermediate category (“dual”) to capture instances where both literal and emphatic readings coexist?
The annotation scheme: from three categories to two
Kostadinova’s initial annotation scheme distinguished three categories:
- BASIC: The adverb retains its etymological sense of literalness (He literally translated every word)
- DUAL: Ambiguous cases where both literal and non-literal interpretations are plausible (The streets were literally empty)
- NONLITERAL: Clear pragmatic uses as intensifier or emphasis marker (I am literally starving)
This tripartite distinction seemed theoretically motivated. The DUAL category would capture the transitional nature of semantic change, instances where the bleaching process is incomplete or where context permits both readings. From a diachronic perspective, such instances might reveal the mechanisms of semantic drift.
However, theory and practice diverged sharply. When fifteen students independently annotated the same instances, inter-annotator agreement was disappointing. Table 1 presents Cohen’s kappa scores for all annotator pairs using the three-category scheme.
Table 1: Cohen’s kappa scores (three categories: BASIC, DUAL, NONLITERAL)
| Annotator A | Annotator B | N instances | Kappa | Agreement % |
|---|---|---|---|---|
| CASSANDRA | JAILYS | 50 | 0.138 | 38.0 |
| FLORA | HOURIA | 50 | -0.018 | 32.0 |
| HOURIA | KHOULOUD | 50 | 0.380 | 60.0 |
| NIAMH | STEPHANIE | 50 | 0.132 | 36.0 |
| LAURIE-ANNE | MATTHIEU | 50 | 0.267 | 54.0 |
| AMANDINE | CASSANDRA | 50 | 0.158 | 34.0 |
| FLORA | KAREN | 49 | 0.043 | 42.9 |
| AMANDINE | NATALIA | 50 | 0.291 | 52.0 |
| ALEX | LAURIE-ANNE | 50 | 0.417 | 64.0 |
| ALEX | CAMILLE | 50 | 0.230 | 50.0 |
| KHOULOUD | NIAMH | 50 | 0.287 | 52.0 |
| CAMILLE | KAREN | 50 | 0.214 | 54.0 |
| MARYAM | MATTHIEU | 50 | 0.230 | 56.0 |
| JAILYS | MARYAM | 50 | 0.304 | 60.0 |
| NATALIA | STEPHANIE | 50 | 0.110 | 40.0 |
Mean κ = 0.212, median κ = 0.230. By conventional interpretation (Landis & Koch 1977), these values indicate “fair” agreement at best, barely above the threshold for “slight” agreement (κ < 0.20). One pair even achieved negative kappa (Flora–Houria, κ = -0.018), which tells us that agreement was worse than chance (yikes!).
The culprit was clearly the DUAL category. Students consistently disagreed about which instances belonged there versus in NONLITERAL. This is unsurprising from a theoretical standpoint: the very notion of “dual interpretation” assumes conscious metalinguistic awareness that may not be present for language users in real communicative situations. Moreover, the distinction between “potentially literal if construed charitably” and “pragmatically strengthened” proved too subtle for reliable operationalization.
The solution was methodologically conservative but pragmatically sound: collapse DUAL into NONLITERAL, and thus create a binary distinction between basic literal uses and everything else. The theoretical justification is straightforward: if an instance of literally can plausibly receive an intensifying or emphatic interpretation, even if a literal reading is also available, then the pragmatic drift has already occurred. The adverb has already acquired its secondary function, regardless of whether that function is dominant or merely co-present.
Table 2 shows the dramatic improvement in inter-annotator agreement after merging categories.
Table 2: Cohen’s kappa scores (binary: BASIC vs. NONLITERAL)
| Annotator A | Annotator B | N instances | Kappa | Agreement % |
|---|---|---|---|---|
| CASSANDRA | JAILYS | 50 | 0.261 | 68.0 |
| FLORA | HOURIA | 50 | -0.050 | 46.0 |
| HOURIA | KHOULOUD | 50 | 0.437 | 72.0 |
| NIAMH | STEPHANIE | 50 | 0.262 | 64.0 |
| LAURIE-ANNE | MATTHIEU | 50 | 0.532 | 82.0 |
| AMANDINE | CASSANDRA | 50 | 0.263 | 66.0 |
| FLORA | KAREN | 50 | 0.245 | 69.4 |
| AMANDINE | NATALIA | 50 | 0.396 | 78.0 |
| ALEX | LAURIE-ANNE | 50 | 0.671 | 88.0 |
| ALEX | CAMILLE | 50 | 0.148 | 66.0 |
| KHOULOUD | NIAMH | 50 | 0.290 | 66.0 |
| CAMILLE | KAREN | 50 | 0.455 | 78.0 |
| MARYAM | MATTHIEU | 50 | 0.720 | 94.0 |
| JAILYS | MARYAM | 50 | 0.453 | 80.0 |
| NATALIA | STEPHANIE | 50 | 0.157 | 70.0 |
Mean κ = 0.349, median κ = 0.290. This represents a 64.6% improvement in mean kappa, in other words a transition from “fair” to “moderate” agreement. Several pairs now achieve “substantial” agreement (κ > 0.60), with Maryam–Matthieu reaching κ = 0.720. Even the lowest-performing pair improved markedly.
This outcome underscores that more granular is not always better. Theoretical sophistication must yield to practical reliability. If annotators cannot reliably apply a distinction, it cannot serve as ground truth for corpus analysis or machine learning.
The annotation infrastructure
The project required each student to annotate 100 instances, with every instance annotated by exactly two students. This double annotation enables calculation of Cohen’s kappa and provides a gold standard for instances where annotators agree.
With 15 students and 100 instances each, we obtain 1,500 total annotations. Since each instance requires two annotations, this yields 750 unique instances, i.e., roughly 9% of our full dataset of 8,600 sentences containing literally from the Corpus of Historical American English (COHA).
The assignment strategy was implemented in R using what I call a circular pairing scheme. Rather than creating static pairs of students, each student receives two blocks of 50 instances: their “own” block plus the next student’s block. Student 1 annotates blocks 1 and 2, Student 2 annotates blocks 2 and 3, and so forth, with Student 15 annotating blocks 15 and 1, closing the circle. This creates 15 overlapping pairs, each sharing exactly 50 instances.
The advantage of this scheme is pedagogical: it distributes the annotation workload evenly while creating natural discussion pairs. Students who share annotations can compare their judgments, discuss disagreements, and develop metalinguistic awareness of the categorization challenges. The disadvantage is analytical: instead of 7 stable pairs (each with 100 shared instances), we have 15 pairs (each with 50 instances), which complicates the statistical picture slightly. However, 50 instances per pair still provides reasonable power for kappa calculation.
Here is the essential R code for the assignment:
# Prepare Student Annotations
students <- c("NIAMH", "KHOULOUD", "HOURIA", "FLORA", "KAREN",
"CAMILLE", "ALEX", "LAURIE-ANNE", "MATTHIEU", "MARYAM",
"JAILYS", "CASSANDRA", "AMANDINE", "NATALIA", "STEPHANIE")
num_students <- length(students)
rows_needed <- (num_students * 100) / 2 # 750 rows
annotation_pool <- sampled_df[sample(nrow(sampled_df), rows_needed), ]
# Split into 15 blocks of 50
blocks <- split(annotation_pool, rep(1:num_students, each = 50))
# Assign blocks with circular pairing
for (i in 1:num_students) {
next_student <- ifelse(i == num_students, 1, i + 1)
student_data <- rbind(blocks[[i]], blocks[[next_student]])
student_data$annotation <- ""
file_name <- paste0(path_output, students[i], "_literally_annotation.xlsx")
write.xlsx(student_data, file_name, rowNames = FALSE)
}
After students completed their annotations, the files were aggregated and pivoted to create a master table with paired annotations:
# Read all annotation files
file_list <- list.files(path = path_finished, pattern = "*.xlsx", full.names = TRUE)
file_list <- file_list[!grepl("MASTER_ANNOTATIONS|KAPPA", file_list)] # Exclude outputs
all_annotations <- map_df(file_list, function(f) {
data <- read_excel(f)
student_name <- gsub("_literally_annotation.xlsx", "", basename(f))
data$annotator <- student_name
return(data)
})
# Pivot to paired format
grouped_annotations <- all_annotations %>%
group_by(filename, match_context, year) %>%
filter(n() == 2) %>% # Only instances with exactly 2 annotations
summarise(
Annotator_1 = first(annotator),
Label_1 = first(annotation),
Annotator_2 = last(annotator),
Label_2 = last(annotation),
.groups = 'drop'
)
This yields a data frame where each row represents one instance with its two annotations. Instances where annotators disagree can be flagged for adjudication or excluded from training data. Instances where they agree become the gold standard.
XLM-RoBERTa
With 750 doubly-annotated instances (reduced to 362 after removing disagreements), we trained a transformer model to predict labels for the remaining 7,905 unannotated instances. The choice of XLM-RoBERTa rather than, say, BERT or RoBERTa, reflects the multilingual nature of modern corpus linguistics, even though our dataset is English-only. XLM-RoBERTa’s cross-lingual pretraining provides robust representations that generalize well to various linguistic phenomena.
The training procedure was standard for sequence classification: tokenization with a maximum sequence length of 256 tokens, 15% test split with stratification, five training epochs with early stopping, and evaluation using F1 score and accuracy. The model was trained on a Google Colab GPU (Tesla T4). It completed training in approximately seven minutes.
Table 3: Summary of the training procedure
| Epoch | Training Loss | Validation Loss | Accuracy | F1 |
|---|---|---|---|---|
| 1 | No log | 0.700471 | 0.309091 | 0.145960 |
| 2 | 0.732892 | 0.639915 | 0.690909 | 0.564614 |
| 3 | 0.617149 | 0.509002 | 0.800000 | 0.774588 |
| 4 | 0.570008 | 0.494466 | 0.818182 | 0.790874 |
| 5 | 0.473535 | 0.497197 | 0.727273 | 0.736720 |
Model Performance
The final model achieved an F1 score of 0.79 and accuracy of 81.8% on the held-out test set. Table 3 presents the classification report.
Table 4: Classification report (test set, N=55)
| Class | Precision | Recall | F1-score | Support |
|---|---|---|---|---|
| BASIC | 1.000 | 0.412 | 0.583 | 17 |
| NONLITERAL | 0.792 | 1.000 | 0.884 | 38 |
| Accuracy | 0.818 | 55 | ||
| Macro avg | 0.896 | 0.706 | 0.734 | 55 |
| Weighted avg | 0.856 | 0.818 | 0.791 | 55 |
The confusion matrix (Figure 1) reveals the model’s conservative behavior: it never incorrectly labeled a NONLITERAL instance as BASIC (zero false positives for BASIC), but it frequently missed true BASIC instances and labeled them as NONLITERAL instead (7 out of 17, giving a recall of only 41.2%). Conversely, it correctly identified all NONLITERAL instances (recall of 100%).

This asymmetry reflects the class imbalance in the training data (approximately 70% NONLITERAL, 30% BASIC) and suggests the model has learned a conservative heuristic: when in doubt, predict NONLITERAL. From a linguistic perspective, this is actually a defensible strategy. Instances that are genuinely ambiguous between literal and non-literal readings are precisely those where pragmatic strengthening has taken hold. A cautious model that favors NONLITERAL for borderline cases may be capturing the right generalization.
Predictions on Unannotated Data
When applied to the 7,905 unannotated instances, the model predicted:
- NONLITERAL: 7,544 instances (95.4%)
- BASIC: 361 instances (4.6%)
The mean confidence score was 0.820, indicating the model was reasonably certain about its predictions. However, the extreme skew toward NONLITERAL is noteworthy. Recall that in the annotated training data, the distribution was approximately 70-30. The predicted distribution of 95-5 represents a substantial shift.
I believe there are two very likely reasons. First, sampling is biased. The 750 sentences manually annotated happened to oversample BASIC instances relative to the full corpus. If the true distribution in COHA across the period covered genuinely favors non-literal uses at a 95-5 ratio, then the model’s predictions are accurate. Second, the model is conservative: The model’s tendency to default to NONLITERAL when uncertain is amplifying the class imbalance. Instances that human annotators might code as BASIC with 60% confidence are being classified as NONLITERAL by the model.
The truth likely lies at the crossroads of these poles. Historical corpora like COHA show clear diachronic trends in the pragmaticization of adverbial intensifiers, and it would be unsurprising if earlier periods contain more literal uses while later periods are dominated by intensifier uses. A follow-up analysis stratifying predictions by century would illuminate this pattern.
Figure 2 shows the distribution of predictions on the unannotated data. The stark imbalance between NONLITERAL and BASIC is striking.

Examining a handful of individual predictions
The aggregate statistics tell only part of the story. A closer look at individual predictions reveals both the model’s strengths and its systematic uncertainties. Table 4 presents a representative sample of predictions from across the historical span of COHA.
Table 5: Sample predictions with confidence scores
| Source | Genre | Year | Context | Prediction | Confidence |
|---|---|---|---|---|---|
| wlp_mag_1990 | mag | 1990 | the desire for connectedness that is literally embodied in the technology of computer networking | NONLITERAL | 0.837 |
| wlp_acad_1940 | acad | 1940 | the latin word anima meant literally breath and was applied to the life process | BASIC | 0.560 |
| wlp_acad_1980 | acad | 1980 | thus (translating literally from the german) he signed a letter | BASIC | 0.595 |
| wlp_fic_1890 | fic | 1890 | do you really think they will be literally fulfilled on the earth? | NONLITERAL | 0.846 |
| wlp_mag_1840 | mag | 1840 | was, in fact, taken, we believe, a little more literally than it was meant | NONLITERAL | 0.716 |
These instances reveal several patterns that are worth examining critically. The 1990 magazine instance (literally embodied) receives high confidence (0.837) for NONLITERAL, and this seems defensible. While bodies and embodiment are concrete, the phrase “desire for connectedness embodied in technology” is clearly metaphorical: desires cannot literally be embodied in networking technology. Here, literally functions as an intensifier emphasizing the tangible, physical nature of network connections, but the statement remains fundamentally non-literal. The model correctly identifies the pragmatic strengthening.
The two BASIC predictions, however, reveal the model’s uncertainty. Both academic instances, 1940’s meant literally breath and 1980’s translating literally from the german, receive confidence scores barely above chance (0.560 and 0.595 respectively). These are textbook cases of basic literal usage: the first discusses etymological meaning (“what the word literally means”), the second describes translation methodology (“word-for-word rendering”). Yet the model is hesitant. Why?
One possibility is context length. The 1940 instance appears in a passage discussing the metaphorical extension of anima from “breath” to “life force,” creating local ambiguity about whether literally is marking the etymological sense or serving a contrastive discourse function (“literally breath, but metaphorically life”). Similarly, the 1980 instance involves literally modifying a participle (translating), a syntactic context that might be less well-represented in our training data. The model may be detecting genuine ambiguity that human annotators would need context to resolve.
More troubling is the 1890 fiction instance: do you really think they will be literally fulfilled on the earth? The model confidently predicts NONLITERAL (0.846), but this deserves scrutiny. The context involves biblical prophecy: the speaker is asking whether prophetic statements will be fulfilled in their literal, concrete sense or should be interpreted figuratively. The sentence explicitly contrasts literal and figurative fulfillment. This is not an intensifier use; literally carries semantic weight here. The model has likely misclassified this instance.
The error is instructive. The model may have learned to associate question constructions containing literally with non-literal uses, or it may be responding to the genre marker (fiction often contains hyperbolic, non-literal language). Without access to the model’s internal representations, we can only speculate, but the prediction demonstrates the perils of relying on surface patterns rather than deep semantic understanding.
The 1840 magazine instance (taken a little more literally than it was meant) is correctly labeled NONLITERAL with moderate confidence (0.716). Here, literally appears in a metalinguistic reflection on interpretation, and while it does concern literal meaning, the adverb itself functions within a comparative structure (more literally than) that marks degree rather than asserting literal truth. The modest confidence score suggests the model recognizes this as a borderline case, which it is.
What these examples collectively reveal is that confidence scores encode useful information about instance difficulty. The high-confidence predictions (>0.80) tend to be clear cases where contextual cues strongly signal one category or the other. The mid-range predictions (0.55-0.75) cluster around genuinely ambiguous or contextually complex instances. This pattern suggests a pragmatic workflow for corpus analysis: accept high-confidence predictions provisionally, but subject low- and mid-confidence predictions to manual review. For a dataset of nearly 8,000 instances, this might mean carefully examining the roughly 2,000-3,000 instances where the model expresses uncertainty, a far more manageable task than annotating the full corpus.
Moreover, the distribution of confidence across genres and time periods could itself become an object of study. If the model consistently shows lower confidence for earlier time periods, this might indicate that the semantic shift in literally was genuinely more variable or context-dependent in the 19th century, stabilizing only in the 20th. If academic prose generates more mid-confidence predictions than fiction or magazines, this might reflect disciplinary conventions that preserve literal uses while other registers embrace intensification. The model’s uncertainty, in other words, is not mere noise. It is potentially informative signal about the sociolinguistic distribution and diachronic trajectory of the phenomenon.
Methodological reflections
This project brings into focus several perennial issues in corpus annotation and computational linguistics.
The reliability-validity trade-off
Reducing the annotation scheme from three categories to two improved inter-annotator reliability substantially, but at what cost to validity? The DUAL category was introduced to capture genuine linguistic ambiguity: instances where semantic change is incomplete, where both readings coexist. By eliminating this category, have we lost analytic resolution?
The answer depends on one’s theoretical commitments. If DUAL represents a psychologically real intermediate stage in semantic change (a moment where speakers genuinely entertain both literal and non-literal interpretations) then its loss is regrettable. However, if DUAL is merely an artifact of analyst uncertainty, a label applied when we cannot decide, then its removal clarifies rather than obscures. The low kappa scores suggest the latter interpretation. Students were not identifying a consistent class of ambiguous instances; they were exhibiting inconsistent judgment on the same data.
This is not a failure of the students but a feature of the phenomenon. Natural language categories are fuzzy, and the boundaries between semantic and pragmatic content are notoriously porous. Machine learning models trained on such data inherit this fuzziness. They do not resolve ambiguity; they learn to approximate human patterns of ambiguity.
Machine learning as a corpus annotation tool
The deployment of XLM-RoBERTa on unannotated data represents machine learning in its proper auxiliary role: not replacing human judgment but extending it to larger datasets. The model’s predictions should not be treated as ground truth but as working hypotheses, subject to spot-checking and validation.
In practice, one might adopt a tiered approach: accept high-confidence predictions (>90%) as reliable, flag low-confidence predictions (<70%) for manual review, and treat mid-range predictions as provisional. This workflow leverages the model’s ability to process thousands of instances while maintaining human oversight for difficult cases.
Moreover, the model’s systematic biases (especially its tendency to over-predict NONLITERA) become themselves objects of study. Why does the model struggle with BASIC instances? What linguistic features distinguish the instances it misclassifies? Error analysis on the confusion matrix can guide improvements to the annotation scheme or reveal blind spots in the training data.
Pedagogical outcomes
From a teaching perspective, the project succeeded in ways that transcend the raw accuracy numbers. Students confronted the practical difficulties of operationalizing theoretical distinctions, learned to tolerate disagreement and ambiguity, and saw firsthand how annotation quality directly impacts machine learning performance. The experience demystifies both corpus linguistics and NLP, showing that both fields rest on foundations of human judgment: fallible, inconsistent, but ultimately indispensable.
The kappa scores themselves became pedagogical tools. Seeing the improvement from 0.212 to 0.349 after category merger made concrete the principle that annotation schemes must balance granularity with reliability. The wide variation in kappa across pairs (from -0.050 to 0.720) sparked productive discussions about what makes certain instances harder to classify and how individual annotators develop idiosyncratic decision rules.
Limitations and future directions
While this project successfully showed that combining student annotation with transformer-based classification is valid, several limitations emerged that point toward productive avenues for improvement.
The most obvious limitation is the relatively small size of our training set. With only 362 agreed-upon instances after removing disagreements, we are operating at the lower bound of what is typically recommended for fine-tuning transformer models. This explains both the model’s modest F1 score (0.79) and its conservative prediction strategy. Future iterations of this project could address this in several ways.
First, rather than discarding all disagreed instances, we could implement a three-way annotation scheme where a third annotator adjudicates disagreements. This would recover many of the instances currently lost, potentially doubling our training data. The additional cost in time and effort would be modest, only disputed instances require the third annotation, while the gain in training data could be substantial.
Second, we could employ active learning strategies. After the initial model training, the model could be used to identify instances where it is most uncertain (those with prediction probabilities near 0.5). These ambiguous cases could then be prioritized for manual annotation, to direct human effort where it is most needed. This iterative approach (annotate, train, identify uncertain cases, annotate those, retrain) has been shown to achieve better performance with fewer total annotations than random sampling.
Third, the project could be extended over multiple cohorts. Each year’s Pragmatics seminar could annotate an additional 750 instances. Thus, a larger and more robust dataset would be built. Over three or four years, this would yield a training set of 2,000 to 3,000 instances, which is likely sufficient to achieve F1 scores above 0.85.
The initial low kappa scores (mean κ = 0.212 for three categories) suggest that our annotation guidelines, while theoretically motivated, were insufficiently detailed for practical application. Students needed clearer decision trees for borderline cases. Next year, we will definitely adopt the following fixes:
- more extensive training sessions with practice rounds on shared examples, followed by group discussion of disagreements before beginning the actual annotation task
- explicit decision heuristics for difficult cases (e.g., “If the sentence would be semantically coherent without literally, code as NONLITERAL”)
- exemplar sets showing prototypical instances of each category, which annotators can consult during the task
- mid-annotation check-ins after 25-30 instances to catch systematic misunderstandings before they propagate through the full dataset
This year, for lack of time, we did none of the above.
The improvement in kappa from 0.212 to 0.349 after category merger demonstrates that simpler can be better, but even the binary scheme showed considerable variation across annotator pairs (κ ranges from -0.050 to 0.720). This suggests that some annotators would benefit from additional calibration.
Although stratified sampling by century would ensured representation across historical periods, the model’s extreme skew in predictions (95.4% NONLITERAL) raises questions about whether we adequately sampled the full distribution of literally uses in COHA. Next time, we will implement deliberate oversampling of BASIC instances during annotation. If preliminary screening (even crude, heuristic-based screening) suggests that BASIC instances are rare, we could intentionally oversample them for annotation to create a more balanced training set. The model will then need to be calibrated to account for this sampling bias when making predictions, but standard techniques (e.g., adjusting decision thresholds or using class weights during training) can handle this.
Perhaps the most promising avenue for future research is to exploit the temporal dimension of COHA. Our current analysis treats all 8,600 instances as a homogeneous mass (although I made sure the years were kept during extraction; we simply ignored them), but literally has clearly undergone semantic change over the past two centuries. Future work should perhaps analyze prediction distributions by decade or quarter-century to track the historical progression from literal to non-literal uses. It should also train separate models for different time periods to see whether the linguistic features predictive of BASIC versus NONLITERAL shift over time. Another good idea is to identify the inflection point where non-literal uses become dominant, and investigate what sociolinguistic or cultural factors might correlate with this shift. Finally, we should also compare predictions against published scholarship on the historical development of literally to validate or challenge existing claims.
Conclusion
The rise of large language models and transformer architectures has not eliminated the need for carefully annotated corpora; if anything, it has made such resources more valuable. Transformers excel at pattern recognition and generalization, but they require training data that accurately represents the linguistic phenomena of interest. Garbage in, garbage out remains the cardinal rule of machine learning.
This experiment has taught the students that even modest annotation efforts (750 instances, 15 student annotators, three weeks) can yield usable (although not perfect) models for corpus analysis. The key requirements are clear guidelines, sufficient inter-annotator agreement, and realistic expectations about the model’s performance. An F1 score of 0.79 is not state-of-the-art, but it is perfectly adequate for exploratory corpus work, especially when predictions are treated as provisional and subjected to quality control.
Looking forward, the integration of human annotation and machine prediction is arguably becoming standard practice in corpus linguistics. Hopefully, the workflow described here offers a scalable and improvable path forward for large-scale corpus studies. As transformer models continue to improve, the amount of manual annotation required will decrease, but the need for human expertise will remain.
References
Davies, Mark. (2010) The Corpus of Historical American English (COHA). Available online at https://www.english-corpora.org/coha/.
Kostadinova, V. (2018). Attitudes to usage vs. actual language use: The case of literally in American English: American English speakers know how and why they use literally. English Today, 34(4), 29-38. https://doi.org/10.1017/S0266078418000366
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.
Data and code availability: The R scripts for annotation assignment and kappa calculation, along with the Python code for model training, are available upon request. The COHA dataset is accessible through subscription from Mark Davies.
The text only may be used under licence Creative Commons Attribution Non Commercial 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated.
OpenEdition suggests that you cite this post as follows:
Guillaume Desagulier (March 26, 2026). ‘Literally’ the most interesting project ever! Around the word. Retrieved May 21, 2026 from https://doi.org/10.58079/15y3b


orcid.org/0000-0003-4895-0788