The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index; 26 of the 38 sources its claims cite were checked for support.
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Open accessPublished version of a preprint
Abstract
Large language models (LLMs) like ChatGPT can generate and revise text with human-level performance. These models come with clear limitations, can produce inaccurate information, and reinforce existing biases. Yet, many scientists use them for their scholarly writing. But how widespread is such LLM usage in the academic literature? To answer this question for the field of biomedical research, we present an unbiased, large-scale approach: We study vocabulary changes in more than 15 million biomedical abstracts from 2010 to 2024 indexed by PubMed and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. This excess word analysis suggests that at least 13.5% of 2024 abstracts were processed with LLMs. This lower bound differed across disciplines, countries, and journals, reaching 40% for some subcorpora. We show that LLMs have had an unprecedented impact on scientific writing in biomedical research, surpassing the effect of major world events such as the COVID pandemic.
Overview
CoChat Summary
TL;DR
This paper uses excess word frequencies in >15 million PubMed abstracts (2010–2024) to estimate LLM-assisted writing in biomedical publications. It finds an unprecedented post-ChatGPT rise in style words, implying at least 13.5% of 2024 abstracts were processed with LLMs.
Key findings
- At least 13.5% of 2024 PubMed abstracts were processed with LLMs, based on averaging two independent marker-word estimates: Δrare = 0.136 (291 rare style words) and Δcommon = 0.134 (10 common style words).
- Excess vocabulary rose sharply after ChatGPT: 454 excess words in 2024 versus 190 in 2021; counting unique lemmas, 343 in 2024 versus 180 in 2021.
- The 2024 excess vocabulary was mostly style words, not content words: of 379 excess style words in 2024, 66% were verbs and 14% adjectives.
- Lower bounds varied widely by subcorpus, reaching Δ = 0.41 for computation papers from China, Δ = 0.34 for South Korea papers in Sensors, Δ = 0.25 for Sensors, Δ = 0.21 for MDPI, Δ = 0.20 for Frontiers and Cureus, and Δ = 0.07 for Nature/Science/Cell. Gender differences were minor (0.11 male vs 0.10 female).
Methods
- Used the 2025 PubMed annual snapshot, keeping complete English-language abstracts of 250–4000 characters from 2010–2024, yielding 15,103,888 abstracts.
- Built a sparse binary word-occurrence matrix and focused on 273,112 words of at least four English letters; main excess analysis used 26,657 words with p > 10−4 in both 2023 and 2024.
- Computed counterfactual 2024 word frequencies by linear extrapolation from 2021–2022, then measured excess frequency gap δ = p − q and ratio r = p/q; combined excess style words into rare and common marker sets to estimate lower bounds on LLM usage.
Limitations
- The paper states the analysis is corpus-level and cannot identify individual abstracts processed by an LLM; it also cannot separate direct LLM usage from authors adopting LLM-preferred words, nor distinguish effects of different LLMs.
- The paper states the estimates are lower bounds because abstracts not containing selected marker words do not contribute; heterogeneity may reflect detection differences, author editing/censoring of LLM style, and faster publication timelines in some fields/journals.
- The paper states gender inference is approximate, based on first names, with limited US-based training data and only 55% of authors inferable; evident from the text, the method depends on style-word markers and may miss well-disguised LLM use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
11 Valid · 3 Not valid · 5 Not checked · 66 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This makes LLM outputs less diverse and novel than human-written text (43, 44).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > DISCUSSION > Implications and policies
What the sources say
- Partly supportedcites [43] V. Padmakumar, H. He, Does writing with language models reduce content diversity? arXiv:2309.05196 [cs.CL] (2023). sourceExperimental — accuracy not yet measured
The Introduction and conclusion report reduced content diversity in essays co-written with InstructGPT compared with essays written without model help; Section 6 also finds InstructGPT suggestions less diverse than GPT3 suggestions. The paper does not establish that LLM outputs generally are less diverse than human-written text: it found no statistically significant diversity reduction for GPT3-assisted writing. It also does not separately establish the claim about novelty. (LLM, conf=0.87)
- Partly supportedcites [44] A. J. Alvero, J. Lee, A. Regla-Vargas, R. F. Kizilec, T. Joachims, A. L. Antonio, Large language models, social demography, and hegemony: Comparing authorship in human and synthetic text. J. Big Data 11, 138 (2024). sourceExperimental — accuracy not yet measured
The Direct comparisons section (page unavailable) reports that AI-generated essays had lower variation across many writing-style features than human essays, supporting a narrower stylistic range. It does not establish that LLM outputs are less novel, and its comparison is limited to essays generated for college-admissions prompts. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This setup can introduce biases (16), as it requires assumptions on which models scientists use for their LLM-assisted writing, and how exactly they prompt them.”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > INTRODUCTION
What the sources say
- Partly supportedcites [16] R. Tang, Y.-N. Chuang, X. Hu, The science of detecting LLM-generated text. Commun. ACM 67, 50– 59 (2024). sourceExperimental — accuracy not yet measured
The “Limitations of black-box detection” section says data collection can introduce biases that harm a detector’s generalization. “Data acquisition” also says researchers must choose generation models and prompts when collecting LLM-generated text. The paper does not specifically discuss assumptions about which models scientists use for LLM-assisted writing or how those scientists prompt them, so that application is an extension of its point. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Computational fields like computation and bioinformatics showed Δ ≈ 0.20 (Fig. 5B).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Lower bounds differed between subcorpora
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 2020–2022, words like coronavirus, covid, lockdown, and pandemic showed very large excess usages (up to r > 1000 and δ = 0.06), in agreement with the observation that the COVID pandemic had an unprecedented effect on biomedical publishing (30).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Excess words indicate widespread LLM usage
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Content words were predominantly nouns (79.2%), and hence most excess words before 2024 were nouns.”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Excess words indicate widespread LLM usage
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
28 more claims
Create a free account to see all 85 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 33
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM text exhibits systematic lexical and syntactic patterns (Shaib et al., 2026; Kobak et al., 2025; Shaib et al., 2025)”
From StoryScope: Investigating idiosyncrasies in AI fiction (2026) · cites this paper as Kobak et al., 2025
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another study which estimated LLM usage in scientific writing by quantitating vocabulary shifts suggested that 13.5% of all 2024 PubMed abstracts were processed with LLMs 27 .”
From Reimagining biomedical science workflows in the age of large language models (2026) · cites this paper as [27]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Kobak et al. [16] reported that no clear increase in LLM-associated “excess vocabulary” was observed during the pre-ChatGPT period (e.g., 2020–2022) compared with two years earlier.”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“One study that analyzed over 15 million abstracts from the biomedical literature documented the emergence of a bona fide LLM lexicon, with specific model-favored words rising at rates that cannot be explained by natural linguistic drift 2 .”
From Fine-Grained Detection of AI-Generated Writing in the Biomedical Literature (2026) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“their analysis revealed an unprecedented post-2022 surge in stylistic words including ‘delve’ and ‘underscore’ [16].”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [16]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Lower bounds differed between subcorpora
“Among countries, some English-speaking countries like United Kingdom and Australia showed Δ ≈ 0.05, while countries like China, South Korea, and Taiwan showed Δ ≈ 0.20 (Fig. 5B).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Combining excess words puts a lower bound on LLM usage
“(B) Frequency gap estimates for various fields identified based on journal names (30). (C) Frequency gap estimates for various countries.”
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
(2008) · Journal of Machine Learning Research · Cited by 33,051
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Furthermore, LLMs can mimic biases and other deficiencies from their training data (Bender et al., 2021; Nav-igli et al., 2023; Bai et al., 2024; Choudhury, 2023), or even outright plagiarise (McCoy et al., 2023).”
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 2×
“…LLMs are infamous for making up references (Walters and Wilder, 2023), providing inaccurate summaries (Tang et al., 2024a; Kim et al., 2024), and making false claims that sound authoritative and convincing (Mittelstadt et al., 2023; Ji et al., 2023; Zhang et al., 2023; Zheng and Zhan, 2023).”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“Some have suggested to use retrieval-augmented LLMs that provide verifiable facts from trusted sources (Lewis et al., 2020; Borgeaud et al., 2022; Ahmed et al., 2023) or let the user provide all relevant facts to the LLM in order to protect scientific literature from accumulating subtle…”
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170
ChatGPT is fun, but not an author
Thorp, H. H. (2023) · Science · doi:10.1126/science.adg7879 · Cited by 1,437 · cited 1×
“As a result, publishers and funding agencies have put out various policies, banning LLMs in peer review (Kaiser, 2023; Brainard, 2023), as co-authors (Thorp, 2023), or undis-closed resource of any kind (Brainard, 2023).”
The WHO estimates of excess mortality associated with the COVID-19 pandemic
Msemburi, W., Karlinsky, A., Knutson, V., Aleshin-Guendel, S., Chatterji, S., & Wakefield, J. (2023) · Nature · doi:10.1038/s41586-022-05522-2 · Cited by 1,064
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891 · cited 1×
“Scientists use LLM-assisted writing because LLMs can improve grammar, rhetoric, and overall readability of their texts, help translate to English, and quickly generate summaries (Van Veen et al., 2024; Zhang et al., 2024).”
Research works printed but not linked: 1
- M. Geng R. Trotta “Is ChatGPT transforming academics’ writing style?” in Next Generation of AI Safety Workshop at ICML 2024 Vienna Austria 26 July 2024.
Cited by
AI and Its Impact on Creativity and Diversity: An Empirical Study of LLM-Generated Product Ideas
Terwiesch, C., Meincke, L., Girotra, K., Mollick, E. R., Nave, G., & Ulrich, K. T. (2023) · doi:10.2139/ssrn.4526071 · Cited by 174
Artificial intelligence tools expand scientists' impact but contract science's focus
Hao, Q., Xu, F., Li, Y., & Evans, J. (2026) · Nature · doi:10.1038/s41586-025-09922-y · Cited by 117
“Recent developments in large language models have also become increasingly used to assist scientific writing 23–26 and facilitate the distillation of scientific findings, but they raise concerns about weakened confidence in AI-generated content 21,22,27 .”
Resisting the Algorithmic Management of Science: Craft and Community After Generative AI
Bechky, B. A., & Davis, G. F. (2024) · Administrative Science Quarterly · doi:10.1177/00018392241304403 · Cited by 73
“One article opened with ‘‘Certainly, here is a possible introduction to your topic’’ (Coddington, 2024; see Kobak et al., 2024 for a systematic assessment in the medical literature).”
‘Stamp out paper mills’ — science sleuths on how to fight fake research
Abalkina, A., Aquarius, R., Bik, E., Bimler, D., Bishop, D., Byrne, J., Cabanac, G., Day, A., Labbé, C., & Wise, N. (2025) · Nature · doi:10.1038/d41586-025-00212-1 · Cited by 59
A Primer for Evaluating Large Language Models in Social-Science Research
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251325174 · Cited by 69
Metadata
- Authors
- Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J.
- Affiliations
- University of Tübingen (DE)
- Northwestern University (US)
- Corresponding author
- Kobak, Dmitry
- González-Márquez, Rita
- Horvát, Emőke-Ágnes
- Lause, Jan
- Venue
- Science Advances
- Published
- 2025-07-04
- Volume
- 11
- Issue
- 27
- Pages
- eadt3813-eadt3813
- ISSN
- 2375-2548 (ISSN-L)
- PMC
- PMC12219543
- OpenAlex
- W4411948293
- Cited by
- 213
- References
- 47
- Citations by year
- 2026: 145 · 2025: 54
- FWCI
- 126.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Science Advances (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Universitätsbibliothek Tübingen
- Funders
- National Science Foundation (1943506, CAREER)
- International Max Planck Research School for Environmental, Cellular and Molecular Microbiology
- Deutsche Forschungsgemeinschaft (390727645, KO6282/2-1)
- Gemeinnützige Hertie-Stiftung
- International Max Planck Research School for Advanced Methods in Process and Systems Engineering
- Leibniz-Gemeinschaft
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- large language models, ChatGPT, biomedical abstracts, scientific writing, scholarly publishing
- MeSH terms
- Humans, Language, Publications, Vocabulary, Writing, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Delving into LLM-assisted writing in biomedical publications through excess vocabulary. https://app.cochat.ai/papers/10.1126/sciadv.adt3813
- The paper
- Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2025). Delving into LLM-assisted writing in biomedical publications through excess vocabulary. Science Advances. https://doi.org/10.1126/sciadv.adt3813
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.