Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Open accessPublished version of a preprint
The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index; 26 of the 38 sources its claims cite were checked for support.
Abstract
Large language models (LLMs) like ChatGPT can generate and revise text with human-level performance. These models come with clear limitations, can produce inaccurate information, and reinforce existing biases. Yet, many scientists use them for their scholarly writing. But how widespread is such LLM usage in the academic literature? To answer this question for the field of biomedical research, we present an unbiased, large-scale approach: We study vocabulary changes in more than 15 million biomedical abstracts from 2010 to 2024 indexed by PubMed and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. This excess word analysis suggests that at least 13.5% of 2024 abstracts were processed with LLMs. This lower bound differed across disciplines, countries, and journals, reaching 40% for some subcorpora. We show that LLMs have had an unprecedented impact on scientific writing in biomedical research, surpassing the effect of major world events such as the COVID pandemic.
Overview
CoChat Summary
TL;DR
This paper uses excess word frequencies in >15 million PubMed abstracts (2010–2024) to estimate LLM-assisted writing in biomedical publications. It finds an unprecedented post-ChatGPT rise in style words, implying at least 13.5% of 2024 abstracts were processed with LLMs.
Key findings
- At least 13.5% of 2024 PubMed abstracts were processed with LLMs, based on averaging two independent marker-word estimates: Δrare = 0.136 (291 rare style words) and Δcommon = 0.134 (10 common style words).
- Excess vocabulary rose sharply after ChatGPT: 454 excess words in 2024 versus 190 in 2021; counting unique lemmas, 343 in 2024 versus 180 in 2021.
- The 2024 excess vocabulary was mostly style words, not content words: of 379 excess style words in 2024, 66% were verbs and 14% adjectives.
- Lower bounds varied widely by subcorpus, reaching Δ = 0.41 for computation papers from China, Δ = 0.34 for South Korea papers in Sensors, Δ = 0.25 for Sensors, Δ = 0.21 for MDPI, Δ = 0.20 for Frontiers and Cureus, and Δ = 0.07 for Nature/Science/Cell. Gender differences were minor (0.11 male vs 0.10 female).
Methods
- Used the 2025 PubMed annual snapshot, keeping complete English-language abstracts of 250–4000 characters from 2010–2024, yielding 15,103,888 abstracts.
- Built a sparse binary word-occurrence matrix and focused on 273,112 words of at least four English letters; main excess analysis used 26,657 words with p > 10−4 in both 2023 and 2024.
- Computed counterfactual 2024 word frequencies by linear extrapolation from 2021–2022, then measured excess frequency gap δ = p − q and ratio r = p/q; combined excess style words into rare and common marker sets to estimate lower bounds on LLM usage.
Limitations
- The paper states the analysis is corpus-level and cannot identify individual abstracts processed by an LLM; it also cannot separate direct LLM usage from authors adopting LLM-preferred words, nor distinguish effects of different LLMs.
- The paper states the estimates are lower bounds because abstracts not containing selected marker words do not contribute; heterogeneity may reflect detection differences, author editing/censoring of LLM style, and faster publication timelines in some fields/journals.
- The paper states gender inference is approximate, based on first names, with limited US-based training data and only 55% of authors inferable; evident from the text, the method depends on style-word markers and may miss well-disguised LLM use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
11 Valid · 3 Not valid · 5 Not checked · 66 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This makes LLM outputs less diverse and novel than human-written text (43, 44).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > DISCUSSION > Implications and policies
What the sources say
- Partly supportedcites [43] V. Padmakumar, H. He, Does writing with language models reduce content diversity? arXiv:2309.05196 [cs.CL] (2023). sourceExperimental — accuracy not yet measured
The Introduction and conclusion report reduced content diversity in essays co-written with InstructGPT compared with essays written without model help; Section 6 also finds InstructGPT suggestions less diverse than GPT3 suggestions. The paper does not establish that LLM outputs generally are less diverse than human-written text: it found no statistically significant diversity reduction for GPT3-assisted writing. It also does not separately establish the claim about novelty. (LLM, conf=0.87)
- Partly supportedcites [44] A. J. Alvero, J. Lee, A. Regla-Vargas, R. F. Kizilec, T. Joachims, A. L. Antonio, Large language models, social demography, and hegemony: Comparing authorship in human and synthetic text. J. Big Data 11, 138 (2024). sourceExperimental — accuracy not yet measured
The Direct comparisons section (page unavailable) reports that AI-generated essays had lower variation across many writing-style features than human essays, supporting a narrower stylistic range. It does not establish that LLM outputs are less novel, and its comparison is limited to essays generated for college-admissions prompts. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This setup can introduce biases (16), as it requires assumptions on which models scientists use for their LLM-assisted writing, and how exactly they prompt them.”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > INTRODUCTION
What the sources say
- Partly supportedcites [16] R. Tang, Y.-N. Chuang, X. Hu, The science of detecting LLM-generated text. Commun. ACM 67, 50– 59 (2024). sourceExperimental — accuracy not yet measured
The “Limitations of black-box detection” section says data collection can introduce biases that harm a detector’s generalization. “Data acquisition” also says researchers must choose generation models and prompts when collecting LLM-generated text. The paper does not specifically discuss assumptions about which models scientists use for LLM-assisted writing or how those scientists prompt them, so that application is an extension of its point. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Computational fields like computation and bioinformatics showed Δ ≈ 0.20 (Fig. 5B).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Lower bounds differed between subcorpora
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 2020–2022, words like coronavirus, covid, lockdown, and pandemic showed very large excess usages (up to r > 1000 and δ = 0.06), in agreement with the observation that the COVID pandemic had an unprecedented effect on biomedical publishing (30).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Excess words indicate widespread LLM usage
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Content words were predominantly nouns (79.2%), and hence most excess words before 2024 were nouns.”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Excess words indicate widespread LLM usage
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
28 more claims
Create a free account to see all 85 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 33
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM text exhibits systematic lexical and syntactic patterns (Shaib et al., 2026; Kobak et al., 2025; Shaib et al., 2025)”
From StoryScope: Investigating idiosyncrasies in AI fiction (2026) · cites this paper as Kobak et al., 2025
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another study which estimated LLM usage in scientific writing by quantitating vocabulary shifts suggested that 13.5% of all 2024 PubMed abstracts were processed with LLMs 27 .”
From Reimagining biomedical science workflows in the age of large language models (2026) · cites this paper as [27]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Kobak et al. [16] reported that no clear increase in LLM-associated “excess vocabulary” was observed during the pre-ChatGPT period (e.g., 2020–2022) compared with two years earlier.”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“One study that analyzed over 15 million abstracts from the biomedical literature documented the emergence of a bona fide LLM lexicon, with specific model-favored words rising at rates that cannot be explained by natural linguistic drift 2 .”
From Fine-Grained Detection of AI-Generated Writing in the Biomedical Literature (2026) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“their analysis revealed an unprecedented post-2022 surge in stylistic words including ‘delve’ and ‘underscore’ [16].”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [16]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Lower bounds differed between subcorpora
“Among countries, some English-speaking countries like United Kingdom and Australia showed Δ ≈ 0.05, while countries like China, South Korea, and Taiwan showed Δ ≈ 0.20 (Fig. 5B).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Combining excess words puts a lower bound on LLM usage
“(B) Frequency gap estimates for various fields identified based on journal names (30). (C) Frequency gap estimates for various countries.”
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
(2008) · Journal of Machine Learning Research · Cited by 33,051
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Furthermore, LLMs can mimic biases and other deficiencies from their training data (Bender et al., 2021; Nav-igli et al., 2023; Bai et al., 2024; Choudhury, 2023), or even outright plagiarise (McCoy et al., 2023).”
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 2×
“…LLMs are infamous for making up references (Walters and Wilder, 2023), providing inaccurate summaries (Tang et al., 2024a; Kim et al., 2024), and making false claims that sound authoritative and convincing (Mittelstadt et al., 2023; Ji et al., 2023; Zhang et al., 2023; Zheng and Zhan, 2023).”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“Some have suggested to use retrieval-augmented LLMs that provide verifiable facts from trusted sources (Lewis et al., 2020; Borgeaud et al., 2022; Ahmed et al., 2023) or let the user provide all relevant facts to the LLM in order to protect scientific literature from accumulating subtle…”
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170
ChatGPT is fun, but not an author
Thorp, H. H. (2023) · Science · doi:10.1126/science.adg7879 · Cited by 1,437 · cited 1×
“As a result, publishers and funding agencies have put out various policies, banning LLMs in peer review (Kaiser, 2023; Brainard, 2023), as co-authors (Thorp, 2023), or undis-closed resource of any kind (Brainard, 2023).”
The WHO estimates of excess mortality associated with the COVID-19 pandemic
Msemburi, W., Karlinsky, A., Knutson, V., Aleshin-Guendel, S., Chatterji, S., & Wakefield, J. (2023) · Nature · doi:10.1038/s41586-022-05522-2 · Cited by 1,064
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891 · cited 1×
“Scientists use LLM-assisted writing because LLMs can improve grammar, rhetoric, and overall readability of their texts, help translate to English, and quickly generate summaries (Van Veen et al., 2024; Zhang et al., 2024).”
Research works printed but not linked: 1
- M. Geng R. Trotta “Is ChatGPT transforming academics’ writing style?” in Next Generation of AI Safety Workshop at ICML 2024 Vienna Austria 26 July 2024.
Cited by
AI and Its Impact on Creativity and Diversity: An Empirical Study of LLM-Generated Product Ideas
Terwiesch, C., Meincke, L., Girotra, K., Mollick, E. R., Nave, G., & Ulrich, K. T. (2023) · doi:10.2139/ssrn.4526071 · Cited by 174
Artificial intelligence tools expand scientists' impact but contract science's focus
Hao, Q., Xu, F., Li, Y., & Evans, J. (2026) · Nature · doi:10.1038/s41586-025-09922-y · Cited by 117
“Recent developments in large language models have also become increasingly used to assist scientific writing 23–26 and facilitate the distillation of scientific findings, but they raise concerns about weakened confidence in AI-generated content 21,22,27 .”
Resisting the Algorithmic Management of Science: Craft and Community After Generative AI
Bechky, B. A., & Davis, G. F. (2024) · Administrative Science Quarterly · doi:10.1177/00018392241304403 · Cited by 111
“One article opened with ‘‘Certainly, here is a possible introduction to your topic’’ (Coddington, 2024; see Kobak et al., 2024 for a systematic assessment in the medical literature).”
‘Stamp out paper mills’ — science sleuths on how to fight fake research
Abalkina, A., Aquarius, R., Bik, E., Bimler, D., Bishop, D., Byrne, J., Cabanac, G., Day, A., Labbé, C., & Wise, N. (2025) · Nature · doi:10.1038/d41586-025-00212-1 · Cited by 59
A Primer for Evaluating Large Language Models in Social-Science Research
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251325174 · Cited by 98
Academic journals’ AI policies fail to curb the surge in AI-assisted academic writing
He, Y.-Y., & Bu, Y. (2025) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.2526734123 · Cited by 64
Will AI speed up literature reviews or derail them entirely?
Reynolds, S. A., Christie, A. P., Dicks, L. V., Jaffer, S., Madhavapeddy, A., Smith, R. K., & Sutherland, W. J. (2025) · Nature · doi:10.1038/d41586-025-02069-w · Cited by 32
Estimating the use of ChatGPT in dental research publications
Uribe, S. E., & Maldupa, I. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105275 · Cited by 27
Language barriers in conservation: consequences and solutions
Amano, T., & Berdejo‐Espinola, V. (2024) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2024.11.003 · Cited by 42
Zhou, Y., Liu, Q., Huang, J., & Li, G. (2026) · Technology in Society · doi:10.1016/j.techsoc.2025.103087 · Cited by 24
ChatGPT turns two: how the AI chatbot has changed scientists’ lives
Lenharo, M. (2024) · Nature · doi:10.1038/d41586-024-03940-y · Cited by 22
AI is complicating plagiarism. How should scientists respond?
Kwon, D. (2024) · Nature · doi:10.1038/d41586-024-02371-z · Cited by 25
Ergonomics & Human factors: fade of a discipline
de Winter, J., & Eisma, Y. B. (2024) · Ergonomics · doi:10.1080/00140139.2024.2416553 · Cited by 21
Drivers and barriers of AI adoption and use in scientific research
Bianchini, S., Müller, M., & Pelletier, P. (2025) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2025.124303 · Cited by 52
Jones, E. M., Newman, J. D., Kim, B., & Fogle, E. J. (2025) · JMIR Medical Education · doi:10.2196/80084 · Cited by 21
“Several studies have found LLM-generated writing tends to overuse adjectives [60] or create prose with an effu-sive, grandiose style [61,62].”
Tudino, G., & Qin, Y. (2024) · Lingua · doi:10.1016/j.lingua.2024.103838 · Cited by 39
Amano, T., Bowker, L., & Burton‐Jones, A. (2025) · PLoS Biology · doi:10.1371/journal.pbio.3003215 · Cited by 20
“Large language models are already widely used in academic writing, especially in countries where English is not widely spoken [7].”
Generative AI and the Perceived Quality of User-Generated Content: Evidence from Online ReviewsPreprint
Knight, S., Bart, Y., & Yang, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4621982 · Cited by 26
Language machines: Toward a linguistic anthropology of large language models
Lamoureaux, S., Castelle, M., & Weichselbraun, A. (2025) · Journal of Linguistic Anthropology · doi:10.1111/jola.70033 · Cited by 20
The adoption of Large Language Models in economics research
Feyzollahi, M., & Rafizadeh, N. (2025) · Economics Letters · doi:10.1016/j.econlet.2025.112265 · Cited by 10
Ahn, S. (2025) · Annals of Pediatric Endocrinology & Metabolism · doi:10.6065/apem.2448278.139 · Cited by 17
“In a recent report, at least 10% of PubMed abstracts published during the first half of 2024 showed signs of LLM usage when analyzing vocabulary patterns [1].”
The Epistemic Downside of Using LLM-Based Generative AI in Academic Writing
Tang, B. L. (2025) · Publications · doi:10.3390/publications13040063 · Cited by 24
“Several studies have revealed signs of fairly widespread undeclared use of AI on published works and preprints (Strzelecki, 2024; Glynn, 2024; Kwon, 2025b; Kobak et al., 2025; Maupin et al., 2025; Spick et al., 2025; Liang et al., 2025).”
Face to face: Comparing ChatGPT with human performance on face matching
Kramer, R. S. S. (2024) · Perception · doi:10.1177/03010066241295992 · Cited by 11
A ‘low-tech’ Academic Virtue Ethics in the Age of Generative AI
Daly, T. (2026) · Journal of Academic Ethics · doi:10.1007/s10805-025-09683-3 · Cited by 9
“Anecdotal and bibliometric evidence suggest that LLMs are being increasingly used by students and academics alike to produce papers, suggesting that an increasing amount of intellectual tasks are being delegated to AI (Joelving, 2024; Kobak et al., 2025).”
The diffusion of large language models in published academic articles
Siler, K. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2605754123 · Cited by 28
Bao, H., Sun, M., & Teplitskiy, M. (2025) · Quantitative Science Studies · doi:10.1162/qss_a_00368 · Cited by 7
Erden, M., Chiappetti, A., & Kho, R. M. (2026) · Journal of Minimally Invasive Gynecology · doi:10.1016/j.jmig.2026.02.014 · Cited by 8
ChatGPT and academic work: new psychological phenomena
de Winter, J., Hancock, P. A., & Eisma, Y. B. (2025) · AI & Society · doi:10.1007/s00146-025-02241-w · Cited by 9
Echoes in AI: Quantifying lack of plot diversity in LLM outputs
Xu, W., Jojić, N., Rao, S., Brockett, C., & Dolan, B. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2504966122 · Cited by 75
'There Has To Be a Lot That We're Missing': Moderating AI-Generated Content on Reddit
Lloyd, T., Reagle, J., & Naaman, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757445 · Cited by 40
AI content is tainting preprints: how moderators are fighting back
Watson, T. (2025) · Nature · doi:10.1038/d41586-025-02469-y · Cited by 7
Matsui, K. (2025) · Perspectives on Medical Education · doi:10.5334/pme.1929 · Cited by 6
“I included records that either specifically reported words or phrases frequently used by LLMs compared to human writing, or focused on terms with increased usage frequency following the emergence of ChatGPT, exemplified by the “excess vocabulary” reported by Kobak et al. [16].”
General-Purpose Models for the Chemical Sciences: LLMs and Beyond
Alampara, N., Aneesh, A., Ríos-García, M., Mirza, A., Schilling-Wilhelmi, M., Aghajani, A. A., Sun, M., Prastalo, G., & Jablonka, K. M. (2026) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00583 · Cited by 35
Fuente-Ballesteros, A., & Zeidler, V. Z. (2026) · Analytical Chemistry · doi:10.1021/acs.analchem.6c00352 · Cited by 6
Signs of AI-generated text found in 14% of biomedical abstracts last year
Mallapaty, S. (2025) · Nature · doi:10.1038/d41586-025-02097-6 · Cited by 8
Writing Scientific Review Articles in the Age of Generative Artificial Intelligence
Leong, S. X., & Loh, N. D. (2025) · ACS Applied Materials & Interfaces · doi:10.1021/acsami.5c15783 · Cited by 7
The rise of large language models and the direction and impact of US federal research funding
Qian, Y., Wen, Z., Furnas, A. C., Bai, Y., Shao, E., & Wang, D. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2601439123 · Cited by 5
Fine-Grained Detection of AI-Generated Writing in the Biomedical Literature
She, R. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.01.697311 · Cited by 4
“One study that analyzed over 15 million abstracts from the biomedical literature documented the emergence of a bona fide LLM lexicon, with specific model-favored words rising at rates that cannot be explained by natural linguistic drift 2 .”
Artificial intelligence for autoimmune diseases
Mahajan, A., LaChance, A., Rodman, A. M., Chung, L. S., & Powell, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02015-0 · Cited by 5
Large language models for peer review in biotechnology
Li, W., Zhao, R., Sun, P.-T., Lee, J., Lu, O., Yuan, S., Sureshkumar, P., Chen, Y., Xiao, Z., & Tang, Y. J. (2026) · New Biotechnology · doi:10.1016/j.nbt.2026.03.007 · Cited by 7
AI and the Future of Publishing: Not Detection, but Answerability
Floridi, L. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01148-8 · Cited by 6
“…volumes have risen sharply against a barely growing reviewer pool (Hanson et al., 2024); analysing more than 15 million PubMed abstracts, Kobak and colleagues estimate that at least 13.5 per cent of 2024 abstracts bear LLM signatures, rising to 40 per cent in some subgroups (Kobak et al., 2025).”
InfluentialUniversities are relying on AI-detection software to catch cheating. How well do the programs work?
McKie, A. (2026) · Nature · doi:10.1038/d41586-026-01358-2 · Cited by 4
Matsubara, S. (2025) · Journal of Investigative Medicine · doi:10.1177/10815589251352572 · Cited by 5
Establishing vocabulary tests as a benchmark for evaluating large language models
Martínez, G., Conde, J., Merino-Gómez, E., Margaretto, B. B., Hernández, J. A., Reviriego, P., & Brysbaert, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308259 · Cited by 10
“By varying these, it is possible to make more demanding vocabulary tests than the TOEFL to see what level various LLMs achieve.”
Shamsi, A., Wang, T., Amraei, M., & Raju, N. V. (2025) · Acta Informatica Pragensia · doi:10.18267/j.aip.293 · Cited by 5
Huh, S. (2026) · Journal of Korean Medical Association · doi:10.5124/jkma.26.0008 · Cited by 3
“이외 논문 작성에 특화된 무료로 이용 가능한 AI 플랫폼으로는 OpenEvidence ( https :// www . openevidence . com )와 STORM AI 활용한 것임을 알 수 있다[ 15 ].”
The Machine Consumes Itself: Artificial Intelligence and the End of Publish-or-Perish
Manabu, T. (2026) · Postdigital Science and Education · doi:10.1007/s42438-026-00666-0 · Cited by 6
Authorial Integrity in the Age of Artificial Intelligence: A Scientific Autobiography
Colenda, C. C. (2025) · American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2025.10.004 · Cited by 4
Hampson, T. (2026) · Digital studies in language and literature · doi:10.1515/dsll-2026-0012 · Cited by 6
“In the medical fi eld, using a similar method, Kobak et al. (2025) found that 13.5 % of abstracts indexed in PubMed had had some form of AI editing or writing.”
InfluentialYin, H. R., Ahn, W., Forster, P. M., & Rust, R. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.30.735556 · Cited by 5
“Because the corpus largely precedes the routine use of large language models in scientific writing, the trend from 2018 to 2024 also provides a reference point for assessing how preprint-to-publication revision changes as these tools become widespread 16,17 .”
Generative AI and Scientific Authorship: A New Paradigm for Biomedical Imaging Research
Hsieh, S.-C., Lee, K.-H., Karmakar, R., Kandalkar, A., Mukundan, A., & Wang, H. (2025) · International Journal of Software Science and Computational Intelligence · doi:10.4018/ijssci.394323 · Cited by 2
Emerging conventions in GenAI disclosure: how applied linguistics scholars disclose AI use
Moorhouse, B. L., Nejadghanbar, H., Wu, C., Kaur, H., & Yeo, M. A. (2026) · Applied Linguistics Review · doi:10.1515/applirev-2025-0291 · Cited by 6
“Despite widespread indicators that GenAI is integrated into academic research and writing for publication (Kobak et al. 2025; Li et al. 2024), only 4.2 % of the 3,384 articles examined included any disclosure related to GenAI use, with only 2.8 % declaring use and 1.4 % declaring non-use.”
InfluentialWhen AI Writes the Letters: Recognizing Synthetic Authorship Patterns in Medical Publishing
Lupon, E. L., & Micicoi, G. (2026) · Publications · doi:10.3390/publications14020021 · Cited by 3
(A)I Cannot See Them: A Situated Reflection on the Simulation of Historical Figures
Harder, L. R. (2026) · Open Library of Humanities · doi:10.16995/olh.28625 · Cited by 2
Bietti, L. M., & Bangerter, A. (2026) · PLoS Biology · doi:10.1371/journal.pbio.3003801 · Cited by 2
“As a result, it becomes easier (cheaper) to produce papers, lowering the threshold for lower-quality papers to enter the market, as evidenced by the increase in AI use in published articles [4,5].”
Large Language Model Authorship in Ophthalmic Publications
Fortenbach, C. R., Wu, Y. S., Mungra, P. M., & Gelder, R. N. V. (2026) · Ophthalmology · doi:10.1016/j.ophtha.2026.05.037 · Cited by 2
Joo, H. J., Radnaabaatar, M., & Jung, J. (2026) · Journal of Korean Medical Science · doi:10.3346/jkms.2026.41.e306 · Cited by 9
“3 This widespread adoption extends to all stakeholders in the publishing process, not only authors.”
Saleem, N. H., & Jansen, B. J. J. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8080223 · Cited by 3
“These studies are valuable, but they leave a further question unresolved: whether the observed vocabulary changes are only traces of possible LLM use or whether they reflect a broader diffusion of rhetorical habits across academic writing [1,2].”
InfluentialAI will reorganize science. Will research remain a human enterprise?
Hochberg, M., & Thrall, P. H. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2610088123 · Cited by 5
Challenges and Opportunities of Using Generative AI for Research: Opening the Discussion
Goretti, L., Küsters, A., Morgana, S., & Sharma, A. (2024) · The International Spectator · doi:10.1080/03932729.2024.2435176 · Cited by 5
“LLMs are becoming an integral part of scienti fi c writing and are reshaping our information practices (Kobak et al. 2024).”
Evolutionary Trajectories of Consciousness: From Biological Foundations to Technological Horizons
Gusev, E., Sarapultsev, A., & Komelkova, M. V. (2025) · Brain Sciences · doi:10.3390/brainsci15070734 · Cited by 6
“This unprecedented impact of machine-generated text on the core substrate of scientific communication illustrates how non-human agents are now actively shaping the evolution of knowledge and meaning in real time [377].”
InfluentialGenerating Authentic Grounded Synthetic Maintenance Work Orders
Lau, A., Feng, J., Hodkiewicz, M. R., Woods, C., Stewart, M. G., & Polpo, A. (2025) · IEEE Access · doi:10.1109/access.2025.3598751 · Cited by 2
“LLMs tend to generate certain words more frequently than others [54].”
The Role of Artificial Intelligence in Enabling Medical Research
Radecki, R. P. (2025) · Emergency Medicine Australasia · doi:10.1111/1742-6723.70148 · Cited by 2
Chakrabarty, T., & Dhillon, P. S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791276 · Cited by 12
“But it is merely a matter of time before LLMs reach that capacity.”
InfluentialEpistemic decay: Generative artificial intelligence and the recombination of culture
Tredinnick, L., & Laybats, C. (2025) · Business Information Review · doi:10.1177/02663821251369460 · Cited by 2
The Use of Artificial Intelligence in Academic Research
Graham, G., & Tucker, J. W. (2025) · Accounting Horizons · doi:10.2308/horizons-2025-281 · Cited by 5
Roth, N., Yang, Y., Coulon, F., Nieuwenhuijsen, M. J., & Covaci, A. (2026) · Environment International · doi:10.1016/j.envint.2026.110216 · Cited by 4
Mak, M. H. C., & Walasek, L. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100507 · Cited by 10
“Informed by prior studies (Gray, 2023; Kobak et al., 2025), we selected eight target stylistic words that were identified as being associated with ChatGPT-generated texts.”
MoDeST: A dataset for Multi Domain Scientific Title Generation
Bölücü, N., Bilge, Y. C., Çetintaş, D., & Yücel, Z. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113557 · Cited by 2
López, M. Á., & Petrelli, M. (2025) · Artificial Intelligence in Geosciences · doi:10.1016/j.aiig.2025.100153 · Cited by 3
The Architecture of Academic Overproduction: Toward Post-AI Scholarship
Lang, C., Moffett, C., & Vasudevan, L. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-11108-1_36 · Cited by 1
Arthur, A. W., Moore, A., Carmona-Rosado, L., Kassam, P., Logel, M., Nieves, C. I., Tope, P., Litaker, D. G., & Borrell, L. N. (2025) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2025.103334 · Cited by 2
“A recent analysis of GenAI-associated words in PubMed articles suggested that authors utilized large language models (LLMs) to prepare ≥ 13.5 % of biomedical literature in 2024 (Kobak et al., 2025).”
InfluentialOn the troubling rise of generative AI suspicion in academic publishing
Ciriello, R. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01178-z · Cited by 4
Larney, S. (2026) · Drug and Alcohol Review · doi:10.1111/dar.70128 · Cited by 2
“If trends in the use of LLMs in the peer-reviewed literature are a guide though [34], it can be assumed that even protocol writing would be outsourced in full to an LLM in at least some cases.”
Jeon, K. (2026) · Journal of Korean Medical Association · doi:10.5124/jkma.25.0163 · Cited by 2
Gray, A. D. (2026) · Bulletin of the Atomic Scientists · doi:10.1080/00963402.2026.2628491 · Cited by 2
Sevryugina, Y. V., Collins‐Thompson, K., & Walter, N. G. (2026) · AI in Education · doi:10.3390/aieduc2020014 · Cited by 2
“Perhaps not surprisingly, evidence has recently emerged that, since their emergence, LLMs have already had an unprecedented impact on scientific writing in biomedical research through modifying common vocabulary (Kobak et al., 2025).”
Linguistic shifts in dental scientific writing in the large language model era
Better, H., Haim, D., Asbi, T., Chaushu, G., & Better, K. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00226-x · Cited by 1
“…Kobak and colleagues analyzed over 15 million PubMed abstracts from 2010 to 2024 and found that vocabulary changes after 2022 were without precedent in any prior year, including during the COVID-19 pandemic period, supporting the temporal specificity of LLM-associ-ated stylometric drift [6].”
Grounding the right to refuse AI chatbot use in higher education
Daly, T. P. (2026) · AI and Ethics · doi:10.1007/s43681-026-01233-w · Cited by 2
Algorithmic scholarship and academic evaluation: governance misalignment in the age of generative AI
Baciu, C., Mohan, G., & Wilson, J. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1862980 · Cited by 2
“…empirical work suggests that this shift is 10.3389/frai.2026.1862980 Frontiers in Artificial Intelligence 02 frontiersin.org already visible in scholarly publishing, with large-scale analyses detecting substantial growth in LLM-assisted writing across scientific abstracts (Kobak et al., 2025).”
InfluentialThe Silent Author in Urology: Quantifying Large Language Model Influence at the Corpus LevelPreprint
Arezki, A., Tsui, J. M. G., & Kassouf, W. (2025) · medRxiv · doi:10.1101/2025.07.21.25331681 · Cited by 1
Lingard, L., Driessen, E. W., & Oswald, K. (2025) · medRxiv · doi:10.1101/2025.07.17.25331725 · Cited by 1
McLaughlin, L., Walz, M. S., & Arries, C. (2026) · medRxiv · doi:10.64898/2026.02.18.26346559 · Cited by 2
Liu, J., He, Y., Zheng, Z., Bu, Y., & Ni, C. (2025) · arXiv · doi:10.48550/arxiv.2511.15872 · Cited by 1
“In-depth text analytic estimates align with these trends, with a large-scale study characterizing the impact as unprecedented and estimating that at least 13.5% of 2024 abstracts in biomedical science involved some levels of LLM assistance (16).”
InfluentialThe Role of Human Agency in Human-AI Co-Creativity
Wu, S. H., Yang, Y., Lee, A. Y., Liebscher, A., Rapuano, K., Niederhoffer, K., & Hancock, J. T. (2026) · Proceedings of the 2026 Conference on Creativity and Cognition · doi:10.1145/3803784.3816857
StoryScope: Investigating idiosyncrasies in AI fiction
Russell, J., Rajendhran, R., Pham, C. M., Iyyer, M., & Wieting, J. (2026) · arXiv · doi:10.48550/arxiv.2604.03136 · Cited by 13
“LLM text exhibits systematic lexical and syntactic patterns (Shaib et al., 2026; Kobak et al., 2025; Shaib et al., 2025), and parallel-corpus studies show LLMs converge on a narrow grammatical and rhetorical style distinct from human variation (Reinhart et al., 2025).”
Han, Z., Battaglia, F., Mansuria, K., Heyman, Y., & Terlecky, S. R. (2024) · Research Square · doi:10.21203/rs.3.rs-5084169/v1
When Hiding Hurts: How Stealth Use of Generative AI Resources Impairs Task PerformancePreprint
Li, P., Ma, L., Dong, N., Xing, L., & Yuan, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5081173
Generative AI - Signal to Noise
Kane, A., Correia, R. A., Healy, K., & Jackson, A. L. (2025) · Digital Society · doi:10.1007/s44206-025-00209-3
Colloque CHR 2024 : les LLM (large language models) au service des humanités numériques ?
Lagarias, A. (2025) · Humanités numériques · doi:10.4000/1498y
Restructuring scientific papers for human and AI readersPreprint
Lin, Z. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/c46hs_v1
Making Adequate Use of Generative Language Models in Scientific ResearchPreprint
Bustio-Martínez, L., Herrera-Semenets, V., Feregrino-Uribe, C., & van den Berg, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5367076
Restructuring scientific papers for human and AI readersPreprint
Lin, Z. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/c46hs_v2
THE ETHNOGRAPHIC AI: EXPLORING THE FUTURE OF ACADEMIC WRITING WITH ARTIFICIAL INTELLIGENCE
(2025) · Education and New Developments 2022 – Volume I · doi:10.36315/2025v2end010
Bazac, A. (2025) · Qeios · doi:10.32388/n8ju5j · Cited by 1
Kreuz, R. J. (2025) · The Conversation · doi:10.64628/aai.pxevmf9r5
Gaining Perspective on Artificial Intelligence and Gerontological Nursing
Kagan, S. H. (2025) · International Journal of Older People Nursing · doi:10.1111/opn.70046
Ethical implications of artificial intelligence-assisted letter of recommendation editing
Keplinger, L. E., Kolodney, M. S., & Hu, G. (2025) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2025.09.068 · Cited by 1
Arthur, A. W., Moore, A., Carmona-Rosado, L., Kassam, P., Logel, M., Nieves, C. I., Tope, P., Litaker, D. G., & Borrell, L. N. (2025) · Preventive Medicine · doi:10.1016/j.ypmed.2025.108436
AI-Written Scientific Manuscripts
Quaia, E. (2025) · Tomography · doi:10.3390/tomography11110123
“According to Kobak et al. [2], among over 15 million biomedical abstracts from 2010–2024 indexed by PubMed, the appearance of LLMs led to an abrupt increase in the frequency of certain style words related to AI-written scientific manuscripts.”
Metadata
- Authors
- Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J.
- Affiliations
- University of Tübingen (DE)
- Northwestern University (US)
- Corresponding author
- Kobak, Dmitry
- González-Márquez, Rita
- Horvát, Emőke-Ágnes
- Lause, Jan
- Venue
- Science Advances
- Published
- 2025-07-04
- Volume
- 11
- Issue
- 27
- Pages
- eadt3813-eadt3813
- ISSN
- 2375-2548 (ISSN-L)
- PMC
- PMC12219543
- OpenAlex
- W4411948293
- Cited by
- 213
- References
- 47
- Citations by year
- 2026: 145 · 2025: 54
- FWCI
- 126.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Science Advances (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Universitätsbibliothek Tübingen
- Funders
- National Science Foundation (1943506, CAREER)
- International Max Planck Research School for Environmental, Cellular and Molecular Microbiology
- Deutsche Forschungsgemeinschaft (390727645, KO6282/2-1)
- Gemeinnützige Hertie-Stiftung
- International Max Planck Research School for Advanced Methods in Process and Systems Engineering
- Leibniz-Gemeinschaft
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- large language models, ChatGPT, biomedical abstracts, scientific writing, scholarly publishing
- MeSH terms
- Humans, Language, Publications, Vocabulary, Writing, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.