Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models
Open accessAbstract
Scientific English is currently undergoing rapid change, with words like "delve," "intricate," and "underscore" appearing far more frequently than just a few years ago. It is widely assumed that scientists' use of large language models (LLMs) is responsible for such trends. We develop a formal, transferable method to characterize these linguistic changes. Application of our method yields 21 focal words whose increased occurrence in scientific abstracts is likely the result of LLM usage. We then pose "the puzzle of lexical overrepresentation": WHY are such words overused by LLMs? We fail to find evidence that lexical overrepresentation is caused by model architecture, algorithm choices, or training data. To assess whether reinforcement learning from human feedback (RLHF) contributes to the overuse of focal words, we undertake comparative model testing and conduct an exploratory online study. While the model testing is consistent with RLHF playing a role, our experimental results suggest that participants may be reacting differently to "delve" than to other focal words. With LLMs quickly becoming a driver of global language change, investigating these potential sources of lexical overrepresentation is important. We note that while insights into the workings of LLMs are within reach, a lack of transparency surrounding model development remains an obstacle to such research.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The lexical preferences of large language models may have been shaped by reinforcement learning from human feedback (RLHF).
“Recent discussions suggest that the lexical preferences of large language models may have been shaped by reinforcement learning from human feedback (RLHF) [18]”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [18]
Such exposure could then be amplified by the RLHF during model fine-tuning, which is considered a primary cause of the increased frequency in LLMs' use of specific expressions.
“Such exposure could then be amplified by the RLHF during model fine-tuning, which is considered a primary cause of the increased frequency in LLMs’ use of specific expressions [18].”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [18]
Cites
Still being checked.
Cited by
Sorted by citations
Bao, T., Zhao, Y., Mao, J., & Zhang, C. (2025) · Scientometrics · doi:10.1007/s11192-025-05341-y · Cited by 25 · see citing works
Matsui, K. (2025) · Perspectives on Medical Education · doi:10.5334/pme.1929 · Cited by 6 · see citing works
“These Recent discussions suggest that the lexical preferences of large language models may have been shaped by reinforcement learning from human feedback (RLHF) [18], which rewards stylistic features favored by human evaluators [19].”
InfluentialKousha, K., & Thelwall, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05601-5 · Cited by 10 · see citing works
Tacit knowledge in large language models
Lu, J. (2025) · The Review of Austrian Economics · doi:10.1007/s11138-025-00710-5 · Cited by 2 · see citing works
(A)I Cannot See Them: A Situated Reflection on the Simulation of Historical Figures
Harder, L. R. (2026) · Open Library of Humanities · doi:10.16995/olh.28625 · Cited by 2 · see citing works
1. Beyond the Hype: Humanities Perspectives on AI
Kyselo, M., Wilde, L. R. A., Brandt, T., Hanem, G., & Søraa, R. A. (2026) · Universitetsforlaget eBooks · doi:10.18261/9788294167234-26-01
Paklina, S. N., Паршаков, П. А., & Shakina, E. A. (2026) · Scientometrics · doi:10.1007/s11192-026-05805-9
- Authors
- Juzek, T. S., & Ward, Z. B.
- Affiliations
- Florida State University (US)
- ORCID
- Ward, Zina B.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2412.11385
- OpenAlex
- W4405469941
- Cited by
- 5
- References
- 0
- Citations by year
- 2026: 3 · 2025: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- large language models, ChatGPT, scientific English, language change, reinforcement learning from human feedback, scientific abstracts, word usage trends, model testing, online study, model transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
