CoChat

Delving into LLM-assisted writing in biomedical publications through excess vocabulary

Open access

Published version of a preprint

The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index; 26 of the 38 sources its claims cite were checked for support.

Abstract

Large language models (LLMs) like ChatGPT can generate and revise text with human-level performance. These models come with clear limitations, can produce inaccurate information, and reinforce existing biases. Yet, many scientists use them for their scholarly writing. But how widespread is such LLM usage in the academic literature? To answer this question for the field of biomedical research, we present an unbiased, large-scale approach: We study vocabulary changes in more than 15 million biomedical abstracts from 2010 to 2024 indexed by PubMed and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. This excess word analysis suggests that at least 13.5% of 2024 abstracts were processed with LLMs. This lower bound differed across disciplines, countries, and journals, reaching 40% for some subcorpora. We show that LLMs have had an unprecedented impact on scientific writing in biomedical research, surpassing the effect of major world events such as the COVID pandemic.

Overview

CoChat Summary

TL;DR

This paper uses excess word frequencies in >15 million PubMed abstracts (2010–2024) to estimate LLM-assisted writing in biomedical publications. It finds an unprecedented post-ChatGPT rise in style words, implying at least 13.5% of 2024 abstracts were processed with LLMs.

Key findings

  • At least 13.5% of 2024 PubMed abstracts were processed with LLMs, based on averaging two independent marker-word estimates: Δrare = 0.136 (291 rare style words) and Δcommon = 0.134 (10 common style words).
  • Excess vocabulary rose sharply after ChatGPT: 454 excess words in 2024 versus 190 in 2021; counting unique lemmas, 343 in 2024 versus 180 in 2021.
  • The 2024 excess vocabulary was mostly style words, not content words: of 379 excess style words in 2024, 66% were verbs and 14% adjectives.
  • Lower bounds varied widely by subcorpus, reaching Δ = 0.41 for computation papers from China, Δ = 0.34 for South Korea papers in Sensors, Δ = 0.25 for Sensors, Δ = 0.21 for MDPI, Δ = 0.20 for Frontiers and Cureus, and Δ = 0.07 for Nature/Science/Cell. Gender differences were minor (0.11 male vs 0.10 female).

Methods

  • Used the 2025 PubMed annual snapshot, keeping complete English-language abstracts of 250–4000 characters from 2010–2024, yielding 15,103,888 abstracts.
  • Built a sparse binary word-occurrence matrix and focused on 273,112 words of at least four English letters; main excess analysis used 26,657 words with p > 10−4 in both 2023 and 2024.
  • Computed counterfactual 2024 word frequencies by linear extrapolation from 2021–2022, then measured excess frequency gap δ = p − q and ratio r = p/q; combined excess style words into rare and common marker sets to estimate lower bounds on LLM usage.

Limitations

  • The paper states the analysis is corpus-level and cannot identify individual abstracts processed by an LLM; it also cannot separate direct LLM usage from authors adopting LLM-preferred words, nor distinguish effects of different LLMs.
  • The paper states the estimates are lower bounds because abstracts not containing selected marker words do not contribute; heterogeneity may reflect detection differences, author editing/censoring of LLM style, and faster publication timelines in some fields/journals.
  • The paper states gender inference is approximate, based on first names, with limited US-based training data and only 55% of authors inferable; evident from the text, the method depends on style-word markers and may miss well-disguised LLM use.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1126/sciadv.adt3813

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1. Frequencies of PubMed abstracts containing several example words. Black lines show counterfactual extrapolations from 2021–2022 to 2023–2024. We manually selected words to illustrate the diversity of frequency time courses. The first six words are affected by LLMs; the last three relate to major events that influenced scientific writing and are shown for comparison.
Fig. 1. Frequencies of PubMed abstracts containing several example words. Black lines show counterfactual extrapolations from 2021–2022 to 2023–2024. We manually selected words to illustrate the diversity of frequency time courses. The first six words are affected by LLMs; the last three relate to major events that influenced scientific writing and are shown for comparison.CC BY · PubMed Central
Fig. 2. Words showing increased frequency in 2024. (A) Frequencies in 2024 and frequency ratios (r). Both axes are on a log scale. Only a subset of points are labeled for visual clarity. The dashed line shows the threshold defining excess words (see text). Words with r > 90 are shown at r = 90. Excess words were manually annotated into content words (blue) and style words (orange). (B) The same but with frequency gap (δ) as the vertical axis. Words with δ > 0.05 are shown at δ = 0.05.
Fig. 2. Words showing increased frequency in 2024. (A) Frequencies in 2024 and frequency ratios (r). Both axes are on a log scale. Only a subset of points are labeled for visual clarity. The dashed line shows the threshold defining excess words (see text). Words with r > 90 are shown at r = 90. Excess words were manually annotated into content words (blue) and style words (orange). (B) The same but with frequency gap (δ) as the vertical axis. Words with δ > 0.05 are shown at δ = 0.05.CC BY · PubMed Central
Fig. 3. Number of excess words per year. (A) Decomposed into the excess content words and excess style words. In each year, we show, as an example, the word with the highest frequency ratio r among excess words with P > 0.0015 and r > 3 (in blue). (B) Decomposed into nouns, verbs, and adjectives. This figure is designed after The Economist’s coverage of our preprint (https://economist.com/science-and-technology/2024/06/26/at-least-10-of-research-may-already-be-co-authored-by-ai).
Fig. 3. Number of excess words per year. (A) Decomposed into the excess content words and excess style words. In each year, we show, as an example, the word with the highest frequency ratio r among excess words with P > 0.0015 and r > 3 (in blue). (B) Decomposed into nouns, verbs, and adjectives. This figure is designed after The Economist’s coverage of our preprint (https://economist.com/science-and-technology/2024/06/26/at-least-10-of-research-may-already-be-co-authored-by-ai).CC BY · PubMed Central
Fig. 4. Combining excess style words yields a larger frequency gap. (A) Observed frequency (P) and counterfactual expected frequency (Q) in 2024 of abstracts containing at least one of the excess style words from 2024 with frequency p below a given threshold. (B) The frequency gap Δ = P − Q as a function of the threshold.
Fig. 4. Combining excess style words yields a larger frequency gap. (A) Observed frequency (P) and counterfactual expected frequency (Q) in 2024 of abstracts containing at least one of the excess style words from 2024 with frequency p below a given threshold. (B) The frequency gap Δ = P − Q as a function of the threshold.CC BY · PubMed Central
Fig. 5. Frequency gaps estimated for various subcorpora. (A) Frequency of abstracts containing at least one word from a given word group. (B) Frequency gap estimates for various fields identified based on journal names (30). (C) Frequency gap estimates for various countries. (D) Frequency gap estimates for various journals. Gray circles show multiple journals grouped together. (E) Frequencies as in (A) for various PubMed subsets; Δ=(Δcommon+Δrare)/2 values are shown.
Fig. 5. Frequency gaps estimated for various subcorpora. (A) Frequency of abstracts containing at least one word from a given word group. (B) Frequency gap estimates for various fields identified based on journal names (30). (C) Frequency gap estimates for various countries. (D) Frequency gap estimates for various journals. Gray circles show multiple journals grouped together. (E) Frequencies as in (A) for various PubMed subsets; Δ=(Δcommon+Δrare)/2 values are shown.CC BY · PubMed Central

Verification

Claims the paper makes

11 Valid · 3 Not valid · 5 Not checked · 66 Own results and interpretations

28 more claims

Create a free account to see all 85 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 30
  • It may state two separate claims as one. 33

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

3 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Research works printed but not linked: 1

  1. M. Geng R. Trotta “Is ChatGPT transforming academics’ writing style?” in Next Generation of AI Safety Workshop at ICML 2024 Vienna Austria 26 July 2024.

Cited by

See all 213

Metadata

Authors
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J.
Affiliations
  • University of Tübingen (DE)
  • Northwestern University (US)
Corresponding author
  • Kobak, Dmitry
  • González-Márquez, Rita
  • Horvát, Emőke-Ágnes
  • Lause, Jan
Venue
Science Advances
Published
2025-07-04
Volume
11
Issue
27
Pages
eadt3813-eadt3813
ISSN
2375-2548 (ISSN-L)
OpenAlex
W4411948293
Cited by
213
References
47
Citations by year
2026: 145 · 2025: 54
FWCI
126.21
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Science Advances (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
  • Universitätsbibliothek Tübingen
Funders
  • National Science Foundation (1943506, CAREER)
  • International Max Planck Research School for Environmental, Cellular and Molecular Microbiology
  • Deutsche Forschungsgemeinschaft (390727645, KO6282/2-1)
  • Gemeinnützige Hertie-Stiftung
  • International Max Planck Research School for Advanced Methods in Process and Systems Engineering
  • Leibniz-Gemeinschaft
Topics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
large language models, ChatGPT, biomedical abstracts, scientific writing, scholarly publishing
MeSH terms
Humans, Language, Publications, Vocabulary, Writing, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.