Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Open accessPublished version of a preprint
The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index; 26 of the 38 sources its claims cite were checked for support.
Abstract
Large language models (LLMs) like ChatGPT can generate and revise text with human-level performance. These models come with clear limitations, can produce inaccurate information, and reinforce existing biases. Yet, many scientists use them for their scholarly writing. But how widespread is such LLM usage in the academic literature? To answer this question for the field of biomedical research, we present an unbiased, large-scale approach: We study vocabulary changes in more than 15 million biomedical abstracts from 2010 to 2024 indexed by PubMed and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. This excess word analysis suggests that at least 13.5% of 2024 abstracts were processed with LLMs. This lower bound differed across disciplines, countries, and journals, reaching 40% for some subcorpora. We show that LLMs have had an unprecedented impact on scientific writing in biomedical research, surpassing the effect of major world events such as the COVID pandemic.
Overview
CoChat Summary
TL;DR
This paper uses excess word frequencies in >15 million PubMed abstracts (2010–2024) to estimate LLM-assisted writing in biomedical publications. It finds an unprecedented post-ChatGPT rise in style words, implying at least 13.5% of 2024 abstracts were processed with LLMs.
Key findings
- At least 13.5% of 2024 PubMed abstracts were processed with LLMs, based on averaging two independent marker-word estimates: Δrare = 0.136 (291 rare style words) and Δcommon = 0.134 (10 common style words).
- Excess vocabulary rose sharply after ChatGPT: 454 excess words in 2024 versus 190 in 2021; counting unique lemmas, 343 in 2024 versus 180 in 2021.
- The 2024 excess vocabulary was mostly style words, not content words: of 379 excess style words in 2024, 66% were verbs and 14% adjectives.
- Lower bounds varied widely by subcorpus, reaching Δ = 0.41 for computation papers from China, Δ = 0.34 for South Korea papers in Sensors, Δ = 0.25 for Sensors, Δ = 0.21 for MDPI, Δ = 0.20 for Frontiers and Cureus, and Δ = 0.07 for Nature/Science/Cell. Gender differences were minor (0.11 male vs 0.10 female).
Methods
- Used the 2025 PubMed annual snapshot, keeping complete English-language abstracts of 250–4000 characters from 2010–2024, yielding 15,103,888 abstracts.
- Built a sparse binary word-occurrence matrix and focused on 273,112 words of at least four English letters; main excess analysis used 26,657 words with p > 10−4 in both 2023 and 2024.
- Computed counterfactual 2024 word frequencies by linear extrapolation from 2021–2022, then measured excess frequency gap δ = p − q and ratio r = p/q; combined excess style words into rare and common marker sets to estimate lower bounds on LLM usage.
Limitations
- The paper states the analysis is corpus-level and cannot identify individual abstracts processed by an LLM; it also cannot separate direct LLM usage from authors adopting LLM-preferred words, nor distinguish effects of different LLMs.
- The paper states the estimates are lower bounds because abstracts not containing selected marker words do not contribute; heterogeneity may reflect detection differences, author editing/censoring of LLM style, and faster publication timelines in some fields/journals.
- The paper states gender inference is approximate, based on first names, with limited US-based training data and only 55% of authors inferable; evident from the text, the method depends on style-word markers and may miss well-disguised LLM use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
11 Valid · 3 Not valid · 5 Not checked · 66 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This makes LLM outputs less diverse and novel than human-written text (43, 44).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > DISCUSSION > Implications and policies
What the sources say
- Partly supportedcites [43] V. Padmakumar, H. He, Does writing with language models reduce content diversity? arXiv:2309.05196 [cs.CL] (2023). sourceExperimental — accuracy not yet measured
The Introduction and conclusion report reduced content diversity in essays co-written with InstructGPT compared with essays written without model help; Section 6 also finds InstructGPT suggestions less diverse than GPT3 suggestions. The paper does not establish that LLM outputs generally are less diverse than human-written text: it found no statistically significant diversity reduction for GPT3-assisted writing. It also does not separately establish the claim about novelty. (LLM, conf=0.87)
- Partly supportedcites [44] A. J. Alvero, J. Lee, A. Regla-Vargas, R. F. Kizilec, T. Joachims, A. L. Antonio, Large language models, social demography, and hegemony: Comparing authorship in human and synthetic text. J. Big Data 11, 138 (2024). sourceExperimental — accuracy not yet measured
The Direct comparisons section (page unavailable) reports that AI-generated essays had lower variation across many writing-style features than human essays, supporting a narrower stylistic range. It does not establish that LLM outputs are less novel, and its comparison is limited to essays generated for college-admissions prompts. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This setup can introduce biases (16), as it requires assumptions on which models scientists use for their LLM-assisted writing, and how exactly they prompt them.”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > INTRODUCTION
What the sources say
- Partly supportedcites [16] R. Tang, Y.-N. Chuang, X. Hu, The science of detecting LLM-generated text. Commun. ACM 67, 50– 59 (2024). sourceExperimental — accuracy not yet measured
The “Limitations of black-box detection” section says data collection can introduce biases that harm a detector’s generalization. “Data acquisition” also says researchers must choose generation models and prompts when collecting LLM-generated text. The paper does not specifically discuss assumptions about which models scientists use for LLM-assisted writing or how those scientists prompt them, so that application is an extension of its point. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Computational fields like computation and bioinformatics showed Δ ≈ 0.20 (Fig. 5B).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Lower bounds differed between subcorpora
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 2020–2022, words like coronavirus, covid, lockdown, and pandemic showed very large excess usages (up to r > 1000 and δ = 0.06), in agreement with the observation that the COVID pandemic had an unprecedented effect on biomedical publishing (30).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Excess words indicate widespread LLM usage
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Content words were predominantly nouns (79.2%), and hence most excess words before 2024 were nouns.”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Excess words indicate widespread LLM usage
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
28 more claims
Create a free account to see all 85 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 33
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM text exhibits systematic lexical and syntactic patterns (Shaib et al., 2026; Kobak et al., 2025; Shaib et al., 2025)”
From StoryScope: Investigating idiosyncrasies in AI fiction (2026) · cites this paper as Kobak et al., 2025
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another study which estimated LLM usage in scientific writing by quantitating vocabulary shifts suggested that 13.5% of all 2024 PubMed abstracts were processed with LLMs 27 .”
From Reimagining biomedical science workflows in the age of large language models (2026) · cites this paper as [27]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Kobak et al. [16] reported that no clear increase in LLM-associated “excess vocabulary” was observed during the pre-ChatGPT period (e.g., 2020–2022) compared with two years earlier.”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“One study that analyzed over 15 million abstracts from the biomedical literature documented the emergence of a bona fide LLM lexicon, with specific model-favored words rising at rates that cannot be explained by natural linguistic drift 2 .”
From Fine-Grained Detection of AI-Generated Writing in the Biomedical Literature (2026) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“their analysis revealed an unprecedented post-2022 surge in stylistic words including ‘delve’ and ‘underscore’ [16].”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [16]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Lower bounds differed between subcorpora
“Among countries, some English-speaking countries like United Kingdom and Australia showed Δ ≈ 0.05, while countries like China, South Korea, and Taiwan showed Δ ≈ 0.20 (Fig. 5B).”
§ Delving into LLM-assisted writing in biomedical publications through excess vocabulary > RESULTS > Combining excess words puts a lower bound on LLM usage
“(B) Frequency gap estimates for various fields identified based on journal names (30). (C) Frequency gap estimates for various countries.”
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
(2008) · Journal of Machine Learning Research · Cited by 33,051
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Furthermore, LLMs can mimic biases and other deficiencies from their training data (Bender et al., 2021; Nav-igli et al., 2023; Bai et al., 2024; Choudhury, 2023), or even outright plagiarise (McCoy et al., 2023).”
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 2×
“…LLMs are infamous for making up references (Walters and Wilder, 2023), providing inaccurate summaries (Tang et al., 2024a; Kim et al., 2024), and making false claims that sound authoritative and convincing (Mittelstadt et al., 2023; Ji et al., 2023; Zhang et al., 2023; Zheng and Zhan, 2023).”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“Some have suggested to use retrieval-augmented LLMs that provide verifiable facts from trusted sources (Lewis et al., 2020; Borgeaud et al., 2022; Ahmed et al., 2023) or let the user provide all relevant facts to the LLM in order to protect scientific literature from accumulating subtle…”
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170
ChatGPT is fun, but not an author
Thorp, H. H. (2023) · Science · doi:10.1126/science.adg7879 · Cited by 1,437 · cited 1×
“As a result, publishers and funding agencies have put out various policies, banning LLMs in peer review (Kaiser, 2023; Brainard, 2023), as co-authors (Thorp, 2023), or undis-closed resource of any kind (Brainard, 2023).”
The WHO estimates of excess mortality associated with the COVID-19 pandemic
Msemburi, W., Karlinsky, A., Knutson, V., Aleshin-Guendel, S., Chatterji, S., & Wakefield, J. (2023) · Nature · doi:10.1038/s41586-022-05522-2 · Cited by 1,064
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891 · cited 1×
“Scientists use LLM-assisted writing because LLMs can improve grammar, rhetoric, and overall readability of their texts, help translate to English, and quickly generate summaries (Van Veen et al., 2024; Zhang et al., 2024).”
Research works printed but not linked: 1
- M. Geng R. Trotta “Is ChatGPT transforming academics’ writing style?” in Next Generation of AI Safety Workshop at ICML 2024 Vienna Austria 26 July 2024.
Cited by
Items 101–197 of 197
A final word from the Editor in Chief
Martín, D. (2025) · Journal of the Intensive Care Society · doi:10.1177/17511437251396862
Opinion paper: generative AI and the future of scientometrics
Lepori, B., Andersen, J. P., & Donnay, K. (2026) · Scientometrics · doi:10.1007/s11192-026-05667-1 · Cited by 9
“…uses are acknowledged by authors (Kousha, 2024), linguistic analyses suggested that explicit acknowledgments represent only a fraction of usage and that between one-quarter and one-third of scholarly papers in some fields are already written with at least some GenAI assistance (Kobak et al., 2024).”
The Use of Artificial Intelligence in Academic ResearchPreprint
Graham, G. M., & Tucker, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5816222
Large language models in academic writing
Yeh, E. A., Levy, M., Hawkes, C. H., Lechner‐Scott, J., & Giovannoni, G. (2025) · Multiple Sclerosis and Related Disorders · doi:10.1016/j.msard.2025.106945
Abstract writing in the modern era of artificial intelligence
Chan, K., & Tang, P. (2025) · Cardiovascular Research · doi:10.1093/cvr/cvaf253
Henzler, H., & Tanasi, V. (2025) · FOM-Edition · doi:10.1007/978-3-658-49537-4_11
Generative artificial intelligence tools
Xu, Y., & Polio, C. G. (2026) · Research methods in applied linguistics · doi:10.1075/rmal.15.17xu
Bashashati, A., Sampath, A., Agrawal, D., & Bashashati, M. (2026) · Journal of Investigative Medicine · doi:10.1177/10815589261415897
A five-step guide for validating umbrella reviews with bibliometric and network indicators
Klaser, K., Casari, P., & Cuel, R. (2026) · Management Review Quarterly · doi:10.1007/s11301-025-00582-w
Use of AI in Research Journal Manuscript Writing and its ImplicationsPreprint
Weerakoon, M., Amarathunga, B., Weerasuriya, P., & Sivashankar, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6178360
AI Disclosure Policies: Are We Rearranging Deck Chairs on the Titanic?
Bradshaw, T. (2026) · Journal of Nuclear Medicine · doi:10.2967/jnumed.125.271870
Alhajailan, D. (2026) · World Journal of English Language · doi:10.2139/ssrn.6253748
Formalized scientific methodology enables rigorous AI-conducted research across domainsPreprint
Zhang, Y.-L., & Zhao, J. (2026) · bioRxiv · doi:10.64898/2026.03.02.709102
Kim, S. Y. (2026) · Journal of Korean Medical Association · doi:10.5124/jkma.26.0012
Reporting of Animal Studies on Craniofacial Bone Repair: 2004, 2014, and 2024
Liu, Y., Guo, F., Botelho, J., Tao, Z., Zhao, T., Degen, M., Pandis, N., & Hua, F. (2026) · Journal of Dental Research · doi:10.1177/00220345261424229 · Cited by 1
“However, it should be noted that although outputs from LLMs appear more fluent and standardized, they may undermine the authenticity of scientific writing (Kobak et al. 2025), which warrants further investigation in future research.”
On the makeup of high-impact reviews in ecology
Veresoglou, S. D., & Agathokleous, E. (2026) · Eco-Environment & Health · doi:10.1016/j.eehl.2026.100233
“Academic publishing currently undergoes a major breakthrough through the gradual integration of large language models into scientific documents [28,29].”
The Potential for Artificial Intelligence to Augment Peer Review
Milne, W. K., & Fatovich, D. M. (2026) · Emergency Medicine Australasia · doi:10.1111/1742-6723.70243
“This is fundamentally changing what reviewers and editors observe [5].”
Citation Inaccuracies and the Need for Multi-Level Oversight in AI-Assisted Medical Writing
Rajaratnam, V., Omar, U. F., Kee, K., & Kaliya-Perumal, A.-K. (2026) · Standards · doi:10.3390/standards6010010 · Cited by 3
“These tools are already widely used in clinical documentation and academic preparation across the globe [3,4].”
de Oliveira, R. T., & Toth‐Peter, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6467493 · Cited by 1
On the Vulnerability of Citation Metrics in the Era of Generative Artificial Intelligence
Smarsly, K. (2026) · Publications · doi:10.3390/publications14020023 · Cited by 1
“As reported by Haider et al. (2024), approximately two-thirds of examined manuscripts (as of 2024) contained undisclosed LLM-generated content, a practice notably more prevalent in journals with expedited review processes compared to high-prestige outlets (Kobak et al., 2025).”
Is polished writing enough? AI and the illusion of quality
Pievsky, M. A. (2026) · The Brown University Child and Adolescent Behavior Letter · doi:10.1002/cbl.30946
Moch, E. (2026) · Journal of Entrepreneurship and Global Business Strategy · doi:10.66705/zakg0c54
Sprachbarrieren in der Wissenschaft
Friederichs, H. (2026) · doi:10.59350/zz0p3-1en32
Sprachbarrieren in der Wissenschaft
Friederichs, H. (2026) · doi:10.59350/kz7sv-n7p95
Abdulsalam, A. J., Alansari, A., & Özçakar, L. (2026) · The American Journal of the Medical Sciences · doi:10.1016/j.amjms.2026.04.012 · Cited by 1
Generative AI and project scholarship: The collision is coming fast
Pinto, J. K. (2026) · International Journal of Project Management · doi:10.1016/j.ijproman.2026.102858
Lee, J. K., Choi, S., Park, S., Choi, Q., Hwang, S., & Cho, D. (2026) · Vox Sanguinis · doi:10.1111/vox.70280 · Cited by 1
“This instability likely reflects the model ’ s verbosity and reduced lexical precision, phenomena reported in large-scale linguistic analyses of generative LLMs [29].”
Espera, A. H. (2026) · Research Square · doi:10.21203/rs.3.rs-9612303/v1
Matsubara, S. (2026) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000005045
Moghe, G. D., Zimic-Sheen, A., Chen, D., Yadav, G., Cao, G., Tufan, H., Williams, J., Szymański, J., Kim, J., Busta, L., et al. (2026) · The Plant Cell · doi:10.1093/plcell/koag140 · Cited by 2
“One study demonstrated that style words such as “delves,” “no-tably,” “pivotal,” and others showed an unprecedented increase in biomedical abstracts in 2024—2 years after ChatGPT was released (Kobak et al. 2025).”
Self-citation in nursing research: Patterns, outliers, and empirical thresholds
Dardas, L. A., Al-leimon, O., Juweid, M. E., Malkawi, A. M. A., Sawair, F., Al-Halasah, W., & Pan, W. (2026) · Nursing Outlook · doi:10.1016/j.outlook.2026.102801
Huang, Z., & Liu, X. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6811523
Gale, R. P., & Lazarus, H. M. (2026) · Bone Marrow Transplantation · doi:10.1038/s41409-026-02915-z
Cai, Y. (2026) · International Conference on Big Data and Informatization Education (ICBDIE) · doi:10.1145/3806980.3807031 · Cited by 1
“” Level 3 includes common transitional words (“therefore,” “however,” “additionally”) flagged only when their frequency exceeds 3 per 500 words, following the excess frequency approach of Kobak et al. [7].”
Generative artificial intelligence in the writing process of accounting research
Gardner, J., Krupa, N. R., & Mammadov, B. (2026) · Accounting Open · doi:10.1016/j.accop.2026.100010
AI-associated linguistic shifts in Swedish university theses: a corpus-level analysis, 2020–2025
Hintze, A. (2026) · Social Sciences & Humanities Open · doi:10.2139/ssrn.6906371
McCreery, Z., & Naser, M. Z. (2026) · Journal of Informetrics · doi:10.1016/j.joi.2026.101842 · Cited by 6
Visualizing the Adoption of Large Language Models across Sociology Subfields
Cao, L. (2026) · Socius Sociological Research for a Dynamic World · doi:10.1177/23780231261460657 · Cited by 1
For Factors of AI Integration in Research: Case Study of Russian Laboratories
Sazonov, A., Popova, E. V., & Matsepuro, D. M. (2026) · Высшее образование в России · doi:10.31992/0869-3617-2026-35-6-149-168 · Cited by 1
Muslim, A., Asri, N. A. M., & Lyons, D. (2026) · Journal of Medical Internet Research · doi:10.2196/105453
Artificial Intelligence for Discovery in Life Sciences
Chanda, S., Rizzoli, S. O., & Shaib, A. (2026) · Bioconjugate Chemistry · doi:10.1021/acs.bioconjchem.6c00230 · Cited by 1
Qiu, W., & Pan, F. (2026) · Scientometrics · doi:10.1007/s11192-026-05701-2 · Cited by 2
Nakamura, S., Tada, K., Yamashita, Y., Inomata, N., Sakata, K., Ito, K., Kuroda, Y., Yoshizumi, F., Kouso, H., Iwaki, K., et al. (2026) · Surgery Today · doi:10.1007/s00595-026-03383-0
“Instead, we applied a population-level framework, already established in a large-scale bibliometric analysis of PubMed abstracts, to assess whether AI-associated lexical patterns exceeded the expected secular trends in two Japanese surgical journals [ 3, .”
InfluentialRosenkrantz, A. B. (2026) · American Journal of Roentgenology · doi:10.2214/ajr.26.35529 · Cited by 1
Surfacing the Invisible: AI and the IS Discipline
Agarwal, R., Padmanabhan, B., & Liu, Y. (2026) · ACM Transactions on Management Information Systems · doi:10.1145/3828556
Shoji, T. (2026) · Trends in Plant Science · doi:10.1016/j.tplants.2026.06.015
Linguistic Resistance to AI Writing Tools: A Review and Conceptual FrameworkPreprint
Isaev, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10348445/v1
Lexical Traces of AI : Linguistic Impact of Generative Tools in Academic Abstracts
Forgas, R. C., Torres, C. G., Garma, C. T., & Glynn, A. (2026) · Learned Publishing · doi:10.1002/leap.2067 · Cited by 2
“Matsui (2025) and Uribe and Maldupa (2024) highlight the disproportionate increase in words such as ‘delve’, ‘meticulous’, and ‘commendable’, while recent excess-vocabulary analyses in biomedical publishing likewise document abrupt post-ChatGPT lexical shifts at scale (Kobak et al. 2025).”
Solidjonov, D. (2026) · AI and Ethics · doi:10.1007/s43681-026-01284-z
“Kobak et al. [1] applied an excess-vocabulary method to 15 million PubMed abstracts, identifying style words (delve, underscore, showcasing) whose frequency increased abruptly in 2024,they estimate that at least 13.5% of 2024 abstracts were LLM-processed.”
InfluentialThe Invisible Author: Generative AI and the Auditability GapPreprint
Lucchi, N. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7024098
Citation and quotation accuracy in three anesthesia journals: a cross-sectional survey
Lukić, A., KHAMOOSHI, P., Patel, N., Hemmat, A., Modrić, M., Mirkarimi, S., Reiner, K., Candido, K. D., Suresh, S., Edelstein, S. B., et al. (2026) · Frontiers in Anesthesiology · doi:10.3389/fanes.2026.1868790
The rise of AI slop in transfusion medicine: Urgent need for oversight
Raza, S., Booth, G. S., & Jacobs, J. W. (2026) · Transfusion · doi:10.1111/trf.70347
Silva, A., & Silva, V. S. E. (2026) · European Journal of Cancer · doi:10.1016/j.ejca.2026.116963
Reply to Topaz and Bahl: Interpreting a post-2022 lexical shift in academic prose
Siler, K. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2621834123
Nepal, A., Nuredini, R., & Meixner, G. (2026) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3807503.3821552
Advancements in the Implementation of Artificial Intelligence for Imaging and Image Analysis
Niz, M. D., Kirchenbüechler, D., Dluhy, P., Piguel, N. H., & Arvanitis, C. (2026) · Microscopy and Microanalysis · doi:10.1093/mam/ozag051
Let's use ChatGPT for research: fictitious citations, who's responsible?
Abdullah, S., Anwar, F., Khalid, Z., & Siddique, A. (2026) · Consortium Psychiatricum · doi:10.17816/cp15879
AI Writing Vocabulary in Ophthalmology: Faster Non-Native Adoption, Unchanged Outcomes
Berni, A., Ting, D. S. W., Caputo, G., Russo, A., Gutiérrez-Sinisterra, L., Avitabile, A., Virgili, G., Reibaldi, M., Giansanti, F., Bacherini, D., et al. (2026) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2026.08.011
Dobrić, N., & Kuteeva, M. (2026) · Language Policy · doi:10.1007/s10993-026-09798-2 · Cited by 1
“…in assisted writing, increased use of LLM-style vocabulary and corpus-level traces of AI modification, while the precise contribution of L1-and L2-produced training data remains to be tested through targeted corpus-based work (e.g., Fitterer et al., 2025 ; Geng et al., 2025 ; Kobak et al., 2025 ).”
Diogo, J. E., & Andrade, C. S. (2026) · AI & Society · doi:10.1007/s00146-026-03332-y
Staggering 90% of biomedical papers now show signs of AI help
Glickman, K. (2026) · Nature · doi:10.1038/d41586-026-02551-z · Cited by 1
Expert preferences for AI-edited academic writingPreprint
Newbold, S. C., Curtin, F., Erikson, N., Estifenes, C. R., Giles, A., Kau, K., Melker, C., Patenaude, H., Prakash, M., & Zimmer, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7351203
The GenAI Spike: Proposal Submission Trends and Competition in Horizon Europe (2021-2026)
Teixeira, T., Soares, S. P., Maló, P., Soldatos, J., Almeida, B., & Mateus, M. (2026) · International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT) · doi:10.1109/dcoss-iot69657.2026.00157
“Kobak et al. [5] found that at least 13.5% of 2024 biomedical abstracts on PubMed were processed with LLMs — a lower bound reaching 40% in some subcorpora.”
InfluentialWho is mimicking whom? The mutual shaping effect between AI and college students’ English writing
Qin, X. (2026) · Computer Assisted Language Learning · doi:10.1080/09588221.2026.2723281
The influence of large language models on scientific publishers
Naser, M. Z. (2026) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2026.100358
Yang, H., Chang, F., Liu, Z., Zhang, W., & Cai, J. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103535
10 insights for harnessing artificial intelligence in scientific research: supervisor perspectives
Mitra, N. K., Vishnumukkala, T., & Chitra, E. (2026) · MedEdPublish · doi:10.12688/mep.21799.1
Reimagining biomedical science workflows in the age of large language models
Roberts, K. F., Koutarapu, S., Melendez, J., Landsness, E. C., Reese, J. T., Elbert, D. L., & Bateman, R. J. (2026) · npj Dementia · doi:10.1038/s44400-026-00126-3
“Another study which estimated LLM usage in scienti fi c writing by quantitating vocabulary shifts suggested that 13.5% of all 2024 PubMed abstracts were processed with LLMs 27 .”
InfluentialGüneş, Z., & Mergen, E. K. (2026) · Orthopaedic Journal of Sports Medicine · doi:10.1177/23259671261463933
“19 Words such as "delves," "show-casing," and "intricate" appeared with frequencies that exceeded historical baselines.”
InfluentialRai, P., Soni, N., Ora, M., Kassmeyer, B. A., Silvera, V. M., Agarwal, A., Black, D. F., Jain, R., Wintermark, M., & Bathla, G. (2026) · Neuroradiology · doi:10.1007/s00234-026-04176-5
From Property Prediction to Materials Narratives: A Conceptual Shift in Scientific Reporting
Nakamura, H., & Kato, Y. (2023) · Journal of Artificial Intelligence for Materials Science · doi:10.68159/g275479180
Declaring the machine: A disclosure policy on artificial intelligence for authors and reviewers
Iliescu, D., & Greiff, S. (2026) · Intelligence · doi:10.1016/j.intell.2026.102060
Yu, Y. (2026) · doi:10.2196/preprints.111260 · Cited by 1
Why we believe chatbots: trust calibration as a design problem
Jaidka, K., & Cai, M. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1935527 · Cited by 4
“…on whether users perceive the stylistic tropes typical of chatbot-generated text, such as its repeated formulaic expressions, phrases, and epithets (Kobak et al., 2025), as more or less trustworthy than text ostensibly written by humans; what is established is that readers cannot reliably tell the…”
Authors Must Have Intellectual Accountability and Thus Cannot Be ‘Brain-Dead’
Tang, B. L. (2026) · Medical Ethics and Bioethics · doi:10.3390/meb1010002
“For the field of biomedicine, it was estimated that at least 13.5% of abstracts of biomedical articles in 2024 were processed with LLMs, a large portion of it was undeclared [5].”
InfluentialPaklina, S. N., Паршаков, П. А., & Shakina, E. A. (2026) · Scientometrics · doi:10.1007/s11192-026-05805-9 · Cited by 1
“For example, Alsudais (2025) reports changes in scientific readability following the release of ChatGPT, while Kobak et al. (2025) identify excess vocabulary patterns linked to ChatGPT usage in academic writing.”
InfluentialAuthors, reviewers, editors… smitten and bitten by the artificial intelligence (AI) bug!
Kurdi, M. S., Gangakhedkar, G. R., & Sasidharan, S. (2026) · Indian Journal of Anaesthesia · doi:10.4103/ija.ija_1374_26
“[2] To write AI in medical writing off as a passing gimmick would be akin to Dr. Robert Metcalfe, the inventor of Ethernet, predicting the demise of the Internet in 1995!”
Guellec, B. L., Bentegeac, R., Boyer, L., Celi, L. A., Boussat, B., & Hamroun, A. (2026) · Journal of Epidemiology and Population Health · doi:10.1016/j.jeph.2026.203798 · Cited by 1
Finding the Fingerprints of Generative Artificial Intelligence in Psychology Publications
Botes, E., Teuber, Z., Vitali, N., Dewaele, J., & Colling, J. (2026) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459261472842 · Cited by 5
GenAI-assisted language teaching (GenAILT): Approach, design, and procedure
Lin, K. C., & Richards, J. C. (2026) · System · doi:10.1016/j.system.2026.104162
Inooka, T., Yuki, K., & Nishiguchi, K. M. (2026) · Ophthalmology Science · doi:10.1016/j.xops.2026.101418
Ünal, A. (2026) · Ethics and Information Technology · doi:10.1007/s10676-026-09921-1
“Kobak et al. (2025), in an independent analysis of biomedical abstracts, estimated that at least 13.5% of papers published in 2024 in their corpus showed evidence of LLM involve - ment.”
InfluentialHart, M. L., Walsh, M. C., Raimondo, C., Kato, R., & Rolauffs, B. (2026) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2026.1933082
del Giglio, A. (2026) · Journal of Hospital Librarianship · doi:10.1080/15323269.2026.2734722
Wang, Y.-Q., Yang, H., Chen, J.-J., Wang, W.-Z., Song, Y., Wang, A.-J., & Wang, H. (2026) · Environmental Science & Technology · doi:10.1021/acs.est.6c09278
Lipner, S. R. (2026) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2026.09.076
Badilla, C. A. C. (2026) · Biblios Journal of Librarianship and Information Science · doi:10.47456/biblios.2026.1425
The Readability of Generative AI Policies: An Empirical Analysis of OpenAI’s Informational Documents
Bassetta, J., Perpetuini, D., Giusti, M. T., & Merla, A. (2026) · Information · doi:10.3390/info17100953
Scientific Writing, Peer Review, and Artificial Intelligence
BaHammam, A. S. (2026) · Sleep Medicine Clinics · doi:10.1016/j.jsmc.2026.08.004
Positive Language in Ophthalmology Abstracts: A 30-Year Sentiment Analysis
Bak, E., & Kim, Y. K. (2026) · Ophthalmology · doi:10.1016/j.ophtha.2026.09.025
Barbagallo, S. P., Allegra, D., & Stanco, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39385-2_7
Arries, C., McLaughlin, L., & Walz, M. S. (2026) · doi:10.2196/preprints.112916
A Protocol to Identify Large Language Model Use in Undergraduate Chemistry Essays
Livingstone, K. (2026) · Journal of Chemical Education · doi:10.1021/acs.jchemed.5c01816
Pocheshkhova, E., Popova, N. G., Nurbaev, S. D., & Pocheshkhov, N. A. (2026) · Science Editor and Publisher · doi:10.24069/sep.26.11.49
Hernández, I., Souza, D., Toro, C. D., Parker, K., Falcon, A., Danton, G. H., Subhawong, T. K., & Alessandrino, F. (2026) · Academic Radiology · doi:10.1016/j.acra.2026.09.042
PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing
Żurawicki, K., Farganus, J., Gaweł, A., Bystroński, M., & Kajdanowicz, T. J. (2026) · arXiv · doi:10.48550/arxiv.2605.29815 · Cited by 5
Metadata
- Authors
- Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J.
- Affiliations
- University of Tübingen (DE)
- Northwestern University (US)
- Corresponding author
- Kobak, Dmitry
- González-Márquez, Rita
- Horvát, Emőke-Ágnes
- Lause, Jan
- Venue
- Science Advances
- Published
- 2025-07-04
- Volume
- 11
- Issue
- 27
- Pages
- eadt3813-eadt3813
- ISSN
- 2375-2548 (ISSN-L)
- PMC
- PMC12219543
- OpenAlex
- W4411948293
- Cited by
- 213
- References
- 47
- Citations by year
- 2026: 145 · 2025: 54
- FWCI
- 126.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Science Advances (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Universitätsbibliothek Tübingen
- Funders
- National Science Foundation (1943506, CAREER)
- International Max Planck Research School for Environmental, Cellular and Molecular Microbiology
- Deutsche Forschungsgemeinschaft (390727645, KO6282/2-1)
- Gemeinnützige Hertie-Stiftung
- International Max Planck Research School for Advanced Methods in Process and Systems Engineering
- Leibniz-Gemeinschaft
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- large language models, ChatGPT, biomedical abstracts, scientific writing, scholarly publishing
- MeSH terms
- Humans, Language, Publications, Vocabulary, Writing, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.