Playing with words: Comparing the vocabulary and lexical diversity of ChatGPT and humans
Open accessAbstract
The introduction of Artificial Intelligence (AI) generative language models such as GPT (Generative Pre-trained Transformer) and conversational tools such as ChatGPT has triggered a revolution that can transform how text is generated. This has many implications, for example, as AI-generated text becomes a significant fraction of the text, would this affect the language capabilities of readers and also the training of newer AI tools? Would it affect the evolution of languages? Focusing on one specific aspect of the language: words; will the use of tools such as ChatGPT increase or reduce the vocabulary used or the lexical diversity? This has implications for words, as those not included in AI-generated content will tend to be less and less popular and may eventually be lost. In this work, we perform an initial comparison of the vocabulary and lexical diversity of ChatGPT and humans when performing the same tasks. In more detail, two datasets containing the answers to different types of questions answered by ChatGPT and humans, and a third dataset in which ChatGPT paraphrases sentences and questions are used. The analysis shows that ChatGPT-3.5 tends to use fewer distinct words and lower diversity than humans while ChatGPT-4 has a similar lexical diversity as humans and in some cases even larger. These results are very preliminary and additional datasets and ChatGPT configurations have to be evaluated to extract more general conclusions. Therefore, further research is needed to understand how the use of ChatGPT and more broadly generative AI tools will affect the vocabulary and lexical diversity in different types of text and languages.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MTLD is not sensitive to text length, making it suitable for analysing documents of varied length.
“Unlike traditional TTR, MTLD is not sensitive to text length, making it suitable for analysing document of varied length 52 , which is important here because the three types of student report in the WPS Corpus vary significantly in length.”
From Style, sentiment, and quality of undergraduate writing in the AI era: A cross-sectional and longitudinal analysis of 4,820 authentic empirical reports (2025) · cites this paper as [52]
The Vocabulary analysis section (p. 4) says traditional TTR depends on text length, supporting that part of the comparison. But the paper discusses RTTR and Maas as less length-dependent alternatives; our copy does not mention MTLD or establish its suitability for documents of varied length. (LLM, conf=0.88)
Experimental — accuracy not yet measuredNominalisation is often used in academic and more formal writing, intended to convey abstraction and objectivity.
“Nominalisation is often used in academic and more formal writing 52 , intended to convey abstraction and objectivity.”
From Style, sentiment, and quality of undergraduate writing in the AI era: A cross-sectional and longitudinal analysis of 4,820 authentic empirical reports (2025) · cites this paper as [52]
The Vocabulary analysis section (p. 4) discusses part-of-speech tagging and lexical diversity, but not nominalisation as a feature of academic or formal writing. Searches for nominalisation/nominalization, abstraction, and objectivity found no passage addressing the claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 2× · see citing works
“The rise of Generative AI with text-to-image generators such as DALL-E (Ramesh, Dhariwal, Nichol, Chu, & Chen, 2022) or Stable Diffusion (Esser et al., 2024b), text-to-video tools such as Sora, and in particular LLMs has transformed how interactive applications are developed with LLMs becoming a new piece in the softw…”
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 1× · see citing works
“This has huge implications for many industries and jobs as noted by different analysts and experts (Keller, 2023; Ray, 2023).”
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 507 · see citing works
A large-scale comparison of human-written versus ChatGPT-generated essays
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-45644-9 · Cited by 338 · see citing works
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.03206 · Cited by 140 · see citing works
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · cited 1× · see citing works
“Additionally, since these are open datasets, they can be integrated into the model’s training phase, altering the evaluation results (McIntosh, Susnjak, Liu, Watters, & Halgamuge, 2024).”
Mapping the Increasing Use of LLMs in Scientific Papers
Liang, W., Zhang, Y., Wu, Z., Lepp, H., Ji, W., Zhao, X., Cao, H., Liu, S., He, S., Huang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.01268 · Cited by 55 · see citing works
A Tale of Tails: Model Collapse as a Change of Scaling Laws
Dohmatob, E., Feng, Y., Yang, P., Charton, F., & Kempe, J. (2024) · arXiv · doi:10.48550/arxiv.2402.07043 · Cited by 20 · see citing works
Flagged first, then by citations
Research works printed but not linked: 48
- Brysbaert, M., Stevens, M., Mandera, P., & Keuleers, E. (2016). How many words do we know? Practical estimates of vocabulary size dependent on word definition, the degree of language input and the participant's age. Frontiers in Psychology , 7 . http://refhub.elsevier.com/S2666-8270(2400078-1/sb2)
- Cai, Z. G., Haslett, D. A., Duan, X., Wang, S., & Pickering, M. J. (2023). Does ChatGPT resemble humans in language use? arXiv preprint arXiv:2303.08014. · arXiv
- Campbell, L. (2013). Historical linguistics . Edinburgh University Press. http://refhub.elsevier.com/S2666-8270(2400078-1/sb4)
- Conde, J., González, M., Melero, N., Ferrando, R., Martínez, G., Merino-Gómez, E., et al. (2024). Open source conversational LLMs do not know most spanish words. Procesamiento del Lenguaje Natural , 73 , 95-108. http://refhub.elsevier.com/S2666-8270(2400078-1/sb6)
- Conde, J., Reviriego, P., Salvachúa, J., Martínez, G., Hernández, J. A., & Lombardi, F. (2024). Understanding the impact of artificial intelligence in academic writing: Metadata to the rescue. Computer , 57 (1), 105-109. http://refhub.elsevier.com/S2666-8270(2400078-1/sb7)
- Costa-jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., et al. (2022). No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. · arXiv
- Crystal, D. (2011). Internet linguistics: A student guide . Routledge. http://refhub.elsevier.com/S2666-8270(2400078-1/sb9)
- de la Mata, C. B. (2019). La décima edición del Diccionario de la lengua castellana de la Real Academia Española (1852): el aumento y la supresión de voces . http://refhub.elsevier.com/S2666-8270(2400078-1/sb10)
- Diamond, J. (2023). "Genlangs" and Zipf's law: Do languages generated by ChatGPT statistically look human? arXiv:2304.12191. · arXiv
- Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., et al. (2024a). Scaling rectified flow transformers for high-resolution image synthesis. http://refhub.elsevier.com/S2666-8270(2400078-1/sb14)
- Geng, M., & Trotta, R. (2024). Is ChatGPT transforming academics' writing style?. http://refhub.elsevier.com/S2666-8270(2400078-1/sb17)
- Gloeckle, F., Idrissi, B. Y., Rozière, B., Lopez-Paz, D., & Synnaeve, G. (2024). Better & faster large language models via multi-token prediction. http://refhub.elsevier.com/S2666-8270(2400078-1/sb18)
- Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., et al. (2023). How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597. · arXiv
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., et al. (2021). Measuring massive multitask language understanding. http://refhub.elsevier.com/S2666-8270(2400078-1/sb20)
- Herring, S. C. (2005). Computer-mediated discourse. In The handbook of discourse analysis (pp. 612-634). Wiley Online Library. http://refhub.elsevier.com/S2666-8270(2400078-1/sb22)
- Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., et al. (2024). Training compute-optimal large language models. In Proceedings of the 36th international conference on neural information processing systems . Red Hook, NY, USA: Curran Associates Inc.. http://refhub.elsevier.com/S2666-8270(2400078-1/sb23)
- Hout, R., & Vermeer, A. (2007). Comparing measures of lexical richness. In H. Daller, J. Milton, & J. Treffers-Daller (Eds.), Modelling and assessing vocabulary knowledge (pp. 93-116). Cambridge: Cambridge University Press. http://refhub.elsevier.com/S2666-8270(2400078-1/sb24)
- Jarvis, S., & Daller, M. (2013). Defining and measuring lexical diversity. Vocabulary knowledge: Human ratings and automated measures. Amsterdam, the Netherlands. Johansson, V. (2008). Lexical diversity and lexical density in speech and writing: A developmental perspective. Working papers/Lund University, Department of Linguistics and Phonetics , 53 , 61-79. http://refhub.elsevier.com/S2666-8270(2400078-1/sb27)
- [Keller, J. (2023). Chat, anyone? [President's message]. IEEE Computational Intelligence Magazine , 18 (3), 4-5.](http://refhub.elsevier.com/S2666-8270(24)00078-1/sb28)
- Kyle, K. (2019). Measuring lexical richness. In The Routledge handbook of vocabulary studies (pp. 454-475). Routledge. http://refhub.elsevier.com/S2666-8270(2400078-1/sb29)
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th international conference on neural information processing systems . Red Hook, NY, USA: Curran Associates Inc.. http://refhub.elsevier.com/S2666-8270(2400078-1/sb30)
- Liu, Y., Zhang, K., Li, Y., Yan, Z., Gao, C., Chen, R., et al. (2024). Sora: A review on background, technology, limitations, and opportunities of large vision models. http://refhub.elsevier.com/S2666-8270(2400078-1/sb32)
- Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., et al. (2024). The era of 1-bit LLMs: All large language models are in 1.58 bits. http://refhub.elsevier.com/S2666-8270(2400078-1/sb33)
- Marchi, M., Soatto, S., Chaudhari, P., & Tabuada, P. (2024). Heat Death of Generative Models in Closed-Loop Learning. arXiv preprint arXiv:2404.02325. · arXiv
- Marcus, M., Santorini, B., & Marcinkiewicz, M. A. (1993). Building a large annotated corpus of English: The penn treebank. http://refhub.elsevier.com/S2666-8270(2400078-1/sb35)
- Martínez, G., Watson, L., Reviriego, P., Hernández, J. A., Juarez, M., & Sarkar, R. (2024). Towards understanding the interplay of generative artificial intelligence and the internet. In F. Cuzzolin, & M. Sultana (Eds.), Epistemic uncertainty in artificial intelligence (pp. 59-73). Cham: Springer Nature Switzerland. http://refhub.elsevier.com/S2666-8270(2400078-1/sb36)
- Miller, G. A. (1995). WordNet: a lexical database for English. Communications of the ACM , 38 (11), 39-41. http://refhub.elsevier.com/S2666-8270(2400078-1/sb38)
- Muñoz-Ortiz, A., Gómez-Rodríguez, C., & Vilares, D. (2023). Contrasting linguistic patterns in human and LLM-generated text. arXiv preprint arXiv:2308.09067. · arXiv
- Naeem, M. F., Oh, S. J., Uh, Y., Choi, Y., & Yoo, J. (2020). Reliable fidelity and diversity metrics for generative models. In International conference on machine learning (pp. 7176-7185). PMLR. http://refhub.elsevier.com/S2666-8270(2400078-1/sb40)
- Ortega-Martín, M., García-Sierra, Ó., Ardoiz, A., Álvarez, J., Armenteros, J. C., & Alonso, A. (2023). Linguistic ambiguity analysis in ChatGPT. arXiv preprint arXiv: 2302.06426. · arXiv
- Ortega-Martín, M., García-Sierra, Ó., Ardoiz, A., Armenteros, J. C., Garrido, I., Álvarez, J., et al. (2024). Building another Spanish dictionary, this time with GPT-4. arXiv preprint arXiv:2406.11218. · arXiv
- Petrov, S., Das, D., & McDonald, R. (2012). A universal part-of-speech tagset. In Proceedings of the eighth international conference on language resources and evaluation (pp. 2089-2096). Istanbul, Turkey: European Language Resources Association (ELRA). http://refhub.elsevier.com/S2666-8270(2400078-1/sb43)
- Pierre, L., & Cerquiglini, B. (2018). Les mots disparus de Pierre Larousse . Larousse. http://refhub.elsevier.com/S2666-8270(2400078-1/sb44)
- Reviriego, P., & Merino-Gómez, E. (2022). Text to image generation: leaving no language behind. arXiv preprint arXiv:2208.09333. · arXiv
- Schmitt, N. (2010). Researching vocabulary: A vocabulary research manual . Springer. Shijaku, R., & Canhasi, E. (2023). ChatGPT generated text detection.
- Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023). The curse of recursion: Training on generated data makes models forget. arXiv: 2305.17493. · arXiv
- Siepmann, D., Bürgel, C., & Sascha, D. (2016). Le corpus de référence du français contemporain (CRFC), un corpus massif du français largement diversifié par genres. SHS Web of Conferences , 27 , 11002. http://refhub.elsevier.com/S2666-8270(2400078-1/sb51)
- Skowron, D., & Bączkowska, A. (2023). Assessment of various GPT models versus the human text: a quantitative analysis of lexis and cohesion. In Forum filologiczne ateneum , (no. 1 (11)). http://refhub.elsevier.com/S2666-8270(2400078-1/sb52)
- Tagliamonte, S. A., & Denis, D. (2008). Linguistic ruin? Lol! instant messaging and teen language. American Speech , 83 (1), 3-34. http://refhub.elsevier.com/S2666-8270(2400078-1/sb53)
- Toro, J. M. (2023). Emergence of a phonological bias in ChatGPT. arXiv preprint arXiv:2305.15929. · arXiv
- Touvron, H., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. http://refhub.elsevier.com/S2666-8270(2400078-1/sb55)
- Tweedie, F. J., & Baayen, R. H. (1998). How variable may a constant be? Measures of lexical richness in perspective. Computers and the Humanities , 32 , 323-352. http://refhub.elsevier.com/S2666-8270(2400078-1/sb56)
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., et al. (2023). Attention is all you need. http://refhub.elsevier.com/S2666-8270(2400078-1/sb57)
- Vladimir Vorobev, M. K. (2023). ChatGPT paraphrases dataset . http://refhub.elsevier.com/S2666-8270(2400078-1/sb58)
- Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q.-L., et al. (2023). A brief overview of ChatGPT: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica , 10 (5), 1122-1136. http://refhub.elsevier.com/S2666-8270(2400078-1/sb59)
- Yu, P., Chen, J., Feng, X., & Xia, Z. (2023). CHEAT: A large-scale dataset for detecting ChatGPT-written AbsTracts. arXiv:2304.12008. · arXiv
- Zhang, X., Li, S., Hauer, B., Shi, N., & Kondrak, G. (2023). Don't trust ChatGPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Proceedings of the 2023 conference on empirical methods in natural language processing (pp. 7915-7927). http://refhub.elsevier.com/S2666-8270(2400078-1/sb61)
- Zipf, G. K. (2013). Selected studies of the principle of relative frequency in language . Harvard University Press. http://refhub.elsevier.com/S2666-8270(2400078-1/sb62)
Cited by
Sorted by citations
Human-LLM Coevolution: Evidence from Academic Writing
Geng, M., & Trotta, R. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.657 · Cited by 8 · see citing works
Mak, M. H. C., & Walasek, L. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100507 · Cited by 1 · see citing works
- Authors
- Reviriego, P., Conde, J., Merino-Gómez, E., Martínez, G., & Hernández, J. A.
- Affiliations
- Universidad Politécnica de Madrid (ES)
- Universidad de Valladolid (ES)
- Universidad Carlos III de Madrid (ES)
- Corresponding author
- Reviriego, Pedro
- Venue
- Machine Learning with Applications
- Publisher
- Elsevier BV
- Published
- 2024-12
- Volume
- 18
- Pages
- 100602
- ISSN
- 2666-8270 (ISSN-L)
- OpenAlex
- W4404277103
- Cited by
- 37
- References
- 9
- Citations by year
- 2026: 17 · 2025: 15 · 2024: 3 · 2023: 2
- FWCI
- 5.17
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Machine Learning with Applications (publishedVersion, cc-by-nc-nd)
- Hispana (acceptedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- UPM Digital Archive (Technical University of Madrid) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Agencia Estatal de Investigación (TED2021-130118B-I00, PID2022-136684OB-C22)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Language and Linguistics › Linguistic research and analysis
- Keywords
- ChatGPT, lexical diversity, vocabulary, generative AI, AI-generated text, human-AI comparison, ChatGPT-3.5, ChatGPT-4, paraphrase, language evolution, word usage
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
