The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 4 of the 12 sources its claims cite were checked for support.
How much are LLMs changing the language of academic papers after ChatGPT? A multi-database and full text analysis
Open accessAbstract
This study investigates how Large Language Models (LLMs) are influencing the language of academic papers by tracking 12 LLM-associated terms across six major scholarly databases (Scopus, Web of Science, PubMed, PubMed Central (PMC), Dimensions, and OpenAlex) from 2015 to 2024. Using over 2.4 million PMC open-access publications (2021–July 2025), we also analysed full texts to assess changes in the frequency and co-occurrence of these terms before and after ChatGPT’s initial public release. Across databases, delve (+ 1,500%), underscore (+ 1,000%), and intricate (+ 700%) had the largest increases between 2022 and 2024. Growth in LLM-term usage was much higher in STEM fields than in social sciences and arts and humanities. In PMC full texts, the proportion of papers using underscore six or more times increased by over 10,000% from 2022 to 2025, followed by intricate (+ 5,400%) and meticulous (+ 2,800%). Nearly half of all 2024 PMC papers using any LLM term also included underscore, compared with only 3%–14% of papers before ChatGPT in 2022. Papers using one LLM term are now much more likely to include other terms. For example, in 2024, underscore strongly correlated with pivotal (0.449) and delve (0.311), compared with very weak associations in 2022 (0.032 and 0.018, respectively). These findings provide the first large-scale evidence based on full-text publications and multiple databases that some LLM-associated terms are now being used much more frequently and together in academic writing. However, the results do not provide direct causal evidence and cannot distinguish between LLM-generated text, LLM-edited text, or broader adoption of LLM-associated writing or publishing styles. The rapid uptake of LLMs to support scholarly publishing is a welcome development reducing the language barrier to academic publishing for non-English speakers.
Overview
CoChat Summary
TL;DR
This study tracks 12 LLM-associated terms across six scholarly databases (2015–2024) and over 2.4 million PMC full texts (2021–July 2025), finding sharp post-ChatGPT increases in terms such as underscore, delve, and intricate. It reports increased frequency, co-occurrence, and correlation of these terms, especially in STEM and PMC full texts, but cannot establish causality or distinguish LLM-generated from LLM-edited text.
Key findings
- Across databases, delve (+1,500%), underscore (+1,000%), and intricate (+700%) had the largest increases between 2022 and 2024; in Scopus specifically, delve rose 1,360% and underscore 1,062%, compared with traditional terms investigate (+8.9%) and highlight (+85%).
- In PMC full texts, papers using underscore six or more times increased by over 10,000% from 2022 to 2025, followed by intricate (+5,400%) and meticulous (+2,800%).
- Co-occurrence rose sharply: in 2024, 59.3% of PMC papers mentioning delve also mentioned underscore, compared with 1.3% in 2022; papers using intricate, pivotal, or meticulous co-occurred with other LLM terms in 30%–45% of cases by 2024, versus typically below 8% in 2022.
- Correlations strengthened: in 2024, underscore correlated with pivotal (r = 0.449) and delve (r = 0.311), compared with 2022 values of 0.032 and 0.018; growth was much higher in STEM fields than in social sciences and arts and humanities.
Methods
- The authors tracked 12 LLM-associated terms across Scopus, Web of Science, PubMed, PMC, Dimensions, and OpenAlex from 2015 to 2024, normalising term counts by total publications indexed per year.
- For full-text analysis, they downloaded over 2.4 million PMC open-access publications in XML from 2021 to 23 July 2025 and counted term frequencies in the main body, excluding titles, abstracts, references, and supplementary sections.
- Analyses included geometric means for average usage, conditional probabilities for co-occurrence, and Pearson correlations between term frequencies before and after ChatGPT’s public release.
Limitations
- Paper-stated: the results are limited to 12 LLM-associated terms and six databases, may overlook other LLM-suggested terms or models, rely on English-language terms and mostly metadata, and may be biased by non-English articles with translated abstracts or by non-native English speakers using LLMs for proofreading, editing, and translation.
- Paper-stated: some terms may be discipline-specific (e.g., foster, heighten), the PMC full-text dataset covers only open-access biomedical and life-science research, and the study did not assess semantic or stylistic changes in sentence structure.
- Paper-stated and evident from the design: the findings provide no direct causal evidence and cannot distinguish LLM-generated text, LLM-edited text, or broader adoption of LLM-associated writing or publishing styles.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures















Verification
Claims the paper makes
3 Valid · 1 Not valid · 9 Not checked · 124 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Growth in LLM-term usage was much higher in STEM fields than in social sciences and arts and humanities.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Papers using one LLM term are now much more likely to include other terms.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“only 16.1% of papers mentioning underscore (138,183 in total) also mentioned delve (22,241 papers).”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“underscore had weak correlations with delve (r = 0.018), showcase (r = 0.014), and intricate (r = 0.020)”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 2022, the Pearson correlations between most terms were very low, mostly below 0.05.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
54 more claims
Create a free account to see all 137 claims and to verify the 4 references still open.
- Probably not a substantial claim. 38
- It may state two separate claims as one. 53
Citations
Cites
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
Using artificial intelligence in academic writing and research: An essential productivity tool
Khalifa, M., & Albadawy, M. (2024) · Computer Methods and Programs in Biomedicine Update · doi:10.1016/j.cmpbup.2024.100145 · Cited by 580
AI and science: what 1,600 researchers think
Van Noorden, R., & Perkel, J. M. (2023) · Nature · doi:10.1038/d41586-023-02980-0 · Cited by 383
Cabanac, G., Labbé, C., & Magazinov, A. (2021) · arXiv · doi:10.48550/arxiv.2107.06751 · Cited by 224
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07016 · Cited by 214
The Use of Artificial Intelligence in Writing Scientific Review Articles
Kacena, M. A., Plotkin, L. I., & Fehrenbacher, J. C. (2024) · Current Osteoporosis Reports · doi:10.1007/s11914-023-00852-0 · Cited by 193
Artificial intelligence to support publishing and peer review: A summary and review
Kousha, K., & Thelwall, M. (2024) · Learned Publishing · doi:10.1002/leap.1570 · Cited by 151
Self-citationEppler, M., Ganjavi, C., Ramacciotti, L. S., Piazza, P., Rodler, S., Checcucci, E., Gomez Rivas, J., Kowalewski, K. F., Belenchón, I. R., Puliatti, S., et al. (2024) · European Urology · doi:10.1016/j.eururo.2023.10.014 · Cited by 117
Can ChatGPT be used to generate scientific hypotheses?
Park, Y. J., Kaplan, D., Ren, Z., Hsu, C.-W., Li, C., Xu, H., Li, S., & Li, J. (2023) · Journal of Materiomics · doi:10.1016/j.jmat.2023.08.007 · Cited by 82
Huang, Y., Wu, R., He, J., & Xiang, Y. (2024) · Journal of Global Health · doi:10.7189/jogh.14.04070 · Cited by 72
Research works printed but not linked: 2
- Elsevier. (2024). Insights 2024: Attitudes toward AI – Full report. Elsevier. https://www.elsevier.com/insights/attitudes-toward-ai
- OUP (2024). How are researchers responding to AI? Oxford University Press, May 23, 2024, https://corp.oup.com/news/how-are-researchers-responding-to-ai/
Cited by
Which abstract stylistic features fool ChatGPT research evaluations?
Kousha, K., & Thelwall, M. A. (2026) · Scientometrics · doi:10.1007/s11192-026-05756-1 · Cited by 5
Self-citationMcCreery, Z., & Naser, M. Z. (2026) · Journal of Informetrics · doi:10.1016/j.joi.2026.101842 · Cited by 6
Xu, Q., Ren, Y., Zhao, Y., Hu, Y., & Yuan, J. (2026) · Scientometrics · doi:10.1007/s11192-026-05696-w
Reconciling Efficiency and Scientific Integrity Through Human-Centered LLM Integration
Holzer, T., & Michahelles, F. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-34044-3_7
Paklina, S. N., Паршаков, П. А., & Shakina, E. A. (2026) · Scientometrics · doi:10.1007/s11192-026-05805-9 · Cited by 1
Metadata
- Authors
- Kousha, K., & Thelwall, M.
- Affiliations
- University of Wolverhampton (GB)
- University of Sheffield (GB)
- Corresponding author
- Thelwall, Mike
- Venue
- Scientometrics
- Published
- 2026-04-03
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- OpenAlex
- W7148780284
- Cited by
- 10
- References
- 20
- Citations by year
- 2026: 10
- FWCI
- 46.77
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientometrics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Wolverhampton Intellectual Repository and E-Theses (University of Wolverhampton) (submittedVersion)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- large language models, ChatGPT, academic writing style, full-text analysis, PubMed Central, research databases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How much are LLMs changing the language of academic papers after ChatGPT? A multi-database and full text analysis. https://app.cochat.ai/papers/10.1007/s11192-026-05601-5
- The paper
- Kousha, K., & Thelwall, M. (2026). How much are LLMs changing the language of academic papers after ChatGPT? A multi-database and full text analysis. Scientometrics. https://doi.org/10.1007/s11192-026-05601-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.