Digital resources in the Social Sciences and Humanities OpenEdition Our platforms OpenEdition Books OpenEdition Journals Hypotheses Calenda Libraries OpenEdition Freemium Follow us

Spotting the bot: a corpus-based investigation of AI’s linguistic footprint in scholarly discourse

In a previous post, I argued that large language models were contaminating the linguistic ecosystem by producing a generic, formulaic style that spreads through the language like dye through water. A shorter article version of this post was published shortly after in The Conversation, an international nonprofit media outlet. The article drew some attention: fellow linguist Julie Neveux discussed it on national radio (France Inter), and Romain Ferrier wrote about it in a national French broadsheet (Le Figaro). But media coverage, however welcome, is no corpus study. What that post lacked, and what the existing literature has only begun to address, for lack of hindsight, was a rigorous, corpus-level investigation with a proper baseline and a statistical method that could survive scrutiny. This new study attempts to fill that gap. Here is the short story.

An intense moment of pride! (part one) – France Inter
An intense moment of pride! (part two) – Le Figaro

Why abstracts, and why arXiv?

The question I wanted to answer was simple to state and a bit harder to answer: can we detect the stylistic influence of LLMs in scholarly writing, and if so, when did it start and where is it strongest? The obvious place to look was scientific abstracts.

Indeed, abstracts are short, constrained, and high-stakes. They are the first thing a reader sees, which means authors take them seriously and, if they are going to reach help from a language model, they are likely to do so here. From the corpus linguist’s perspective, abstracts are also freely accessible through the arXiv API, which made it possible to collect a large enough sample at no cost to say something statistically meaningful.

The corpus I assembled contains 12,674 abstracts drawn from five disciplines: Natural Language Processing (cs.CL), Artificial Intelligence (cs.AI), Biology (q-bio.GN), Economics (econ.GN), and Mathematics (math.CO). They span the decade from 2015 to 2025, and I divided them into three periods: a baseline (2015–2019), a pre-LLM period (2020–October 2022), and a post-LLM period (April 2023–2025). The gap between October 2022 and April 2023, the window during which ChatGPT was released and GPT-4 became publicly available, was flagged and excluded from the main analyses. The one-publication-cycle lag after GPT-4’s release on 14 March 2023 is deliberate: papers submitted in March are unlikely to have made it through peer review before April.

Corpus composition by discipline and period. Baseline = pre-pandemic reference window (2015–2019); pre-LLM = immediate pre-ChatGPT period (2020–2022); post-LLM = April 2023 onward.
Corpus composition by discipline and period. Baseline = pre-pandemic reference window (2015–2019); pre-LLM = immediate pre-ChatGPT period (2020–2022); post-LLM = April 2023 onward.

Mathematics served as a control group. I may be wrong, but the reason is that mathematical writing is formula-heavy, has a low rhetorical load, and is governed by conventions that have little use for the kind of polished, assertive prose that LLMs tend to generate. If the effects I expected to find were genuine signals of LLM influence rather than some broader shift in academic writing fashion, Mathematics should show nothing at all.

What I measured

Because the study is tentative, it only tracks three families of markers. The first is lexical: I identified five sub-groups of words and expressions associated with AI-generated prose. Words that signal intellectual posturing (delveunderscorediscern, etc.), hyperbole (groundbreakingpivotalinvaluable, etc.), vague intensification (effectivelythoroughlyswiftly, etc.), opaque action verbs (showcasebolsterleverageutilize, etc.), and a set of formal nouns that tend to sound bookish in ways that feel slightly off (realminquiryamidst, etc.). The list was assembled from prior work:l Yakura et al. (2025) on spoken academic communication, Kaplan’s essay in Le Monde diplomatique (2025), Liang et al. (2024) on LLM prevalence in scientific papers. It was also extended through my own observations.

The second family is syntactic. I tracked three constructions. One is the sentence-final participial -ING clause: rather than ending a sentence with a finite verb, the author appends a dangling -ing participle that summarises a result or consequence. Consider this example from a 2024 Biology abstract:

Compared to state-of-the-art CPU software, HySortK achieves up to 2x speedup while reducing peak memory usage by 30% on 16 nodes. Finally, we integrated HySortK into an existing genome assembly pipeline and achieved up to 1.8x speedup, proving its flexibility and practicality in real-world scenarios.

The fragment I flagged is the close the sentence: “, proving its flexibility and practicality in real-world scenarios.” The construction is syntactically unimpeachable, but the pattern accumulates: a 2024 Computer Science abstract offers “, facilitating the reproduction of results”, and a 2023 Economics paper closes a clause with “, potentially leading to market dynamics that straddle the line between efficiency and collusion.” Each instance is fine on its own. The question is whether their frequency increased after April 2023, and by how much.

The other two syntactic constructions are what Kaplan calls the diptych pivot (not X, but rather Y) and the triptych rhythm(three parallel elements of roughly equal weight, A, B, and C). Kaplan identified these patterns as stylistic “attractors” of chatbot prose, though his argument was qualitative. Diptych examples in the corpus are easy to find: “not only rank preferences accurately but also effectively guide generation”“not just the current hypothesis, but an entire class of hypotheses”, and triptych patterns are just as common: “to identify, classify and track emerging virus variants”“understanding, reasoning, and coding”“to read, analyze, and interpret our genomes”. This study is, to my knowledge, the first empirical test of whether their frequency actually shifted after the LLM turning point.

The third family is stylometric: the Type-Token Ratio (a measure of lexical diversity, computed over a moving window to control for text length), mean sentence length, and passive voice frequency.

All scores were normalised per 100 words. Because marker scores tend to be zero-inflated and right-skewed (most abstracts contain none of a given marker, and a few contain many), I used Wilcoxon rank-sum tests rather than t-tests, reported effect sizes as r, and applied Benjamini-Hochberg correction to control the false discovery rate across the eight tests I ran simultaneously.

Five findings

Finding 1: there is a clear inflection point around late 2022 – early 2023. Across all non-mathematics disciplines, the density of AI-associated markers was essentially flat from 2015 to 2022. It then rose sharply. The increase from the baseline period to the post-LLM period represents a 123% rise in mean marker density, with a medium effect size (r = 0.33, p virtually zero after BH correction). This is not a small, technically significant result: it is visible with the naked eye in the time-series plot.

Time series of AI-associated marker density by discipline, 2015–2025. The yellow band marks the transition window; the dashed red line marks GPT-4's public release. Biology, Computer Science, NLP, and Economics all show a sharp rise after 2023. Mathematics remains flat throughout.
Time series of AI-associated marker density by discipline, 2015–2025. The yellow band marks the transition window; the dashed red line marks GPT-4’s public release. Biology, Computer Science, NLP, and Economics all show a sharp rise after 2023. Mathematics remains flat throughout.

Time series of AI-associated marker density by discipline, 2015–2025. The yellow band marks the transition window; the dashed red line marks GPT-4’s public release. Biology, Computer Science, NLP, and Economics all show a sharp rise after 2023. Mathematics remains flat throughout.

Finding 2: the rise is universal but its magnitude is discipline-specific. Biology, Computer Science, and NLP all show medium effect sizes (around r = 0.38–0.40). Economics shows a smaller but still significant effect (r = 0.30). Mathematics, the control group, shows an effect size of r = 0.04, which is both negligible in practical terms and non-significant after correction. The violin plot below makes this visible: the red violins (post-LLM period) swell noticeably in every discipline except Mathematics, which barely moves across all three periods.

Violin plots of AI-associated marker density by period and discipline. The red violins (2023–2025) are markedly wider and shifted upward in Biology, Computer Science, Economics, and NLP. In Mathematics, all three periods look essentially identical.
Violin plots of AI-associated marker density by period and discipline. The red violins (2023–2025) are markedly wider and shifted upward in Biology, Computer Science, Economics, and NLP. In Mathematics, all three periods look essentially identical.

If the post-2023 signal were simply a general trend in academic writing, for example a collective drift toward more assertive prose, we would expect to see it in Mathematics too. We do not. This is the strongest internal validity argument the study can make (my initial intuition was correct, yay!).

Finding 3: not all markers behave the same. The strongest individual signal comes from sentence-final participial -ING clauses (r = 0.239). Lexical markers collectively show r = 0.182. Passive voice, interestingly, decreases after 2023 (r = −0.107): LLMs have a known preference for active constructions, and the data confirm it. Disappointingly, the diptych pivot and triptych rhythm, which Kaplan had identified as hallmarks of chatbot style, showed negligible and non-significant changes (p > 0.22). This does not mean he was wrong (I observed this too, and qualitative observers are often right about things that are statistically weak), but it suggests these patterns are not robust enough to serve as detection features in a corpus of this kind.


Individual marker types over time (all disciplines). Passive voice (green) declines from 2023 onward. Sentence-final participial -ING (purple) and lexical markers (blue) rise sharply after the dashed GPT-4 line. Diptych and triptych patterns (red and orange) remain essentially flat throughout.

Finding 4: within the lexical markers, opaque verbs dominate the rise. The heatmap below shows mean marker density per 100 words broken down by discipline, marker type, and period. The darkest cells (i.e., the highest densities) consistently fall in the post-LLM column, and within each discipline they are driven above all by lexical markers and sentence-final -ING. The pattern for passive voice runs in the opposite direction, darker in the baseline and lighter in the post-LLM period, confirming the negative effect. Mathematics is the quiet exception: its column shows almost no change in colour from left to right.

Heatmap of AI-associated marker density by discipline, marker type, and period. Darker red = higher density. Lexical markers and SF participial -ING darken consistently post-2023 across all disciplines except Mathematics. Passive voice lightens post-2023 across all disciplines.
Heatmap of AI-associated marker density by discipline, marker type, and period. Darker red = higher density. Lexical markers and SF participial -ING darken consistently post-2023 across all disciplines except Mathematics. Passive voice lightens post-2023 across all disciplines.

Looking specifically within the lexical markers, showcasebolsterleverageutilize and their cousins account for the largest absolute increase. Posturing verbs like delve and underscore show the sharpest relative increase (they were almost absent before 2023!) but their absolute density remains low.

Semantic sub-group breakdown of lexical markers by period. The Opaque sub-group (showcase, bolster, leverage, utilize...) drives most of the post-LLM increase in absolute terms. Posturing verbs (delve, underscore...) show the sharpest relative rise from a near-zero baseline.
Semantic sub-group breakdown of lexical markers by period. The Opaque sub-group (showcase, bolster, leverage, utilize…) drives most of the post-LLM increase in absolute terms. Posturing verbs (delve, underscore…) show the sharpest relative rise from a near-zero baseline.

Finding 5: lexical diversity increases after 2023. This one is counterintuitive. The Type-Token Ratio goes up (r = 0.213, p virtually zero). If LLMs were simply making all abstracts sound the same, we would expect diversity to fall. Instead, it rises. The most plausible explanation is that LLMs introduce a range of formal synonyms and near-synonyms. For example, they promptly swap use for utilizeshow for showcaseimportant for pivotal, etc. which inflates the vocabulary count while actually narrowing the semantic register. The text becomes superficially more varied and substantively more homogeneous at the same time. This is the ouroboros I described in my November 2024 post, now in more precise empirical terms: more types, but a thinner world.

Sentence length, as the plot below shows, oscillates without any clear trend and the effect is non-significant (r = −0.027, p= 0.054). This rules out the simplest hypothesis, namely that AI just writes longer sentences, and points to something more specific going on at the level of word choice and clause structure.

Stylometric features over time (all disciplines). Top: Type-Token Ratio rises sharply from 2023. Bottom: Mean sentence length shows no coherent trend, oscillating around 23.7 words throughout. The dashed line marks GPT-4's public release.
Stylometric features over time (all disciplines). Top: Type-Token Ratio rises sharply from 2023. Bottom: Mean sentence length shows no coherent trend, oscillating around 23.7 words throughout. The dashed line marks GPT-4’s public release.

A cluster is not a switch

One further result deserves mention. When I ran a k-means clustering on the seven marker features simultaneously, I found that the two clusters that emerged did not map cleanly onto pre- and post-LLM periods.

a k-means clustering on the seven marker features simultaneously
A k-means clustering on the seven marker features simultaneously.

There is no clean boundary, no moment when scholarly abstracts suddenly became a new text type. What the data show instead is a gradient shift: LLM-associated features permeate the corpus progressively, raising the density of certain markers without creating a qualitatively new register. LLMs are being used in different ways, to different degrees, by different authors in different contexts. The signal is real, but it is diffuse.

What this does and does not establish

I want to be careful here. What this study shows is a footprint, not confirmed LLM use. Correlation is not causation, and it is possible in principle that some other factor explains part of the pattern. I can think of a cultural shift in what polished academic English sounds like, driven in part by non-native speakers modeling their writing on AI-generated text they have read, or a post-COVID surge in submissions. The arXiv corpus over-represents non-native English speakers, who may be more inclined to reach for formal, bookish vocabulary regardless of LLM assistance. The sentence-final -ING pattern has a precision of only around 70% in manual validation, meaning roughly three in ten instances are false positives. And the marker list has not yet been validated on a held-out dataset.

These are real limitations, and I have no interest in overstating the case. What the Mathematics control group does establish, quite firmly, is that something specific happened in disciplines with a high rhetorical load after April 2023, and that it did not happen in a discipline that has little use for the rhetorical repertoire that LLMs favour.

Reproducing this research

The data and code are openly available on GitHub. The corpus was assembled via the arXiv API, stratified by year and discipline (200–300 abstracts per year per discipline) to avoid recency bias. Marker detection was implemented in R: lexical markers via dictionary lookup on lemmatised tokens, syntactic patterns via regular expressions validated manually on 50 random examples per pattern, stylometric features via standard corpus linguistics tools. The statistical pipeline uses the coin package for Wilcoxon tests, Cohen’s r for effect sizes, and the p.adjust function with method = "BH" for multiple comparison correction. The PCA and k-means clustering were run on scaled feature matrices using base R and ggplot2 for visualisation. Anyone with a working R installation and access to the arXiv API can replicate the pipeline from scratch.

Video

References

Kaplan, F. (2025, July). « Diptyques », « attracteurs » et « tricolons » : L’étrange plume de ChatGPT. Le Monde diplomatique.

Liang, W., et al. (2024). Mapping the increasing use of LLMs in scientific papers. arXivhttps://arxiv.org/abs/2404.01268

Yakura, H., et al. (2025). Empirical evidence of large language models’ influence on human spoken communication. arXivhttps://arxiv.org/abs/2409.01754v3

Guillaume Desagulier
Guillaume Desagulier
Université Bordeaux-Montaigne, Laboratoire CLIMAS, Institut Universitaire de France

The text only may be used under licence Creative Commons Attribution Non Commercial 4.0 International. All other elements (illustrations, imported files) are “All rights reserved”, unless otherwise stated.


OpenEdition suggests that you cite this post as follows:
Guillaume Desagulier (March 13, 2026). Spotting the bot: a corpus-based investigation of AI’s linguistic footprint in scholarly discourse. Around the word. Retrieved May 20, 2026 from https://doi.org/10.58079/15veo


You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.