CoChat
Sign Up
Cited by 11Topic ModelingNatural Language Processing TechniquesSpeech and dialogue systems

Benchmarking Linguistic Diversity of Large Language Models

Open access
Guo, Yanzhu, Shang, Guokan, Clavel, Chloé·2025·Transactions of the Association for Computational Linguistics
DOI 10.1162/tacl.a.47
Open this paper in CoChat — free accountOpen PDF

Abstract

The development and evaluation of Large Language Models (LLMs) has primarily focused on their task-solving capabilities, with recent models even surpassing human performance in some areas. However, this focus often neglects whether machine-generated language matches the human level of diversity, in terms of vocabulary choice, syntactic construction, and expression of meaning, raising questions about whether the fundamentals of language generation have been fully addressed. This paper emphasizes the importance of examining the preservation of human linguistic richness by language models, given the concerning surge in online content produced or aided by LLMs. We adapt a comprehensive framework for evaluating LLMs from various linguistic diversity perspectives including lexical, syntactic, and semantic dimensions. Using this framework, we benchmark several state-of-the-art LLMs across all diversity dimensions, and conduct an in-depth analysis for syntactic diversity. Finally, we analyze how the design, development, and deployment choices of LLMs impact the linguistic diversity of their outputs, focusing on the creative task of story generation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/tacl.a.47

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Guo et al. (2025) showed that model outputs lack the richness of human text across lexical, syntactic, and semantic dimensions.

    “Guo et al. (2025) analyze linguistic diversity across lexical, syntactic, and semantic dimensions, showing that model outputs lack the richness of human text.”

    From On Epistemic Diversity in Large Language Models (2026) · cites this paper as Guo et al. (2025)

    The “Model Output Diversity” passage in Section 5.2 reports that LLMs lag behind humans in lexical, syntactic, and semantic diversity for story generation. The claim omits that task qualification: the same passage says models exceed human references in lexical and semantic diversity for next-utterance generation. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 36 · Cited by 11

Research works printed but not linked: 60

  1. Arshiya Aggarwal, Jiao Sun, and Nanyun Peng. 2022. Towards robust NLG bias evaluation with syntactically-diverse prompts. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 6022-6032, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. https://doi .org/10.18653/v1/2022.findings -emnlp.445 · DOI
  2. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. The falcon series of open language models. arXiv preprint arXiv:2311.16867. · arXiv
  3. Yves Bestgen. 2023. Measuring lexical diversity in texts: The twofold length problem. Lan- guage Learning. https://doi.org/10 .1111/lang.12630 · DOI
  4. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statis- tical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Mary- land, USA. Association for Computational Linguistics. https://doi.org/10.3115 /v1/W14-3302, https://doi.org/10 .3115/v1/W14-33 · DOI
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Infor- mation Processing Systems, 33:1877-1901.
  6. Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2020. Language gans falling short. In International Conference on Learning Representations.
  7. Tuhin Chakrabarty, Vishakh Padmakumar, and He He. 2022. Help me write a poem -instruction tuning as a vehicle for collaborative poetry writing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6848-6863, Abu Dhabi, United Arab Emirates. Association for Compu- tational Linguistics. https://doi.org/10 .18653/v1/2022.emnlp-main.460 · DOI
  8. Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloé Clavel. 2022. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In Proceed- ings of the 29th International Conference on Computational Linguistics, pages 5794-5836, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  9. Bastien De Clercq and Alex Housen. 2017. A cross-linguistic perspective on syntactic com- plexity in l2 development: Syntactic elaboration and diversity. The Modern Language Jour- nal, 101(2):315-334. https://doi.org /10.1111/modl.12396 · DOI
  10. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. 2024. Ultrafeedback: Boosting language mod- els with scaled AI feedback. In Forty-first International Conference on Machine Learning. arXiv:2407.21783. · arXiv
  11. Guillaume Dubuisson Duplessis, Chloé Clavel, and Frédéric Landragin. 2017. Automatic measures to characterise verbal alignment in human-agent interaction. In Proceed- ings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 71-81, Saarbrücken, Germany. Association for Com- putational Linguistics. https://doi.org /10.18653/v1/W17-5510 · DOI
  12. Susan Edwards and Roelien Bastiaanse. 1998. Diversity in the lexical and syntactic abil- ities of fluent aphasic speakers. Aphasiol- ogy, 12(2):99-117. https://doi.org/10 .1080/02687039808250466 · DOI
  13. G. Fergadiotis, H. H. Wright, and T. M. West. 2013. Measuring lexical diversity in narra- tive discourse of people with aphasia. Amer- ican Journal of Speech-language Pathology, 22(2):S397-408. https://doi.org/10 .1044/1058-0360(2013/12-0083), PubMed: 23695912 · DOI
  14. Mingmeng Geng and Roberto Trotta. 2024. Is chatgpt transforming academics' writing style? arXiv preprint arXiv:2404.08627. · arXiv
  15. Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023. What comes next? Evaluating uncertainty in neu- ral text generators against human production variability. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 14349-14371, Singapore. Association for Computational Lin- guistics. https://doi.org/10.18653 /v1/2023.emnlp-main.887 · DOI
  16. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. 2024. OLMo: Ac- celerating the science of language mod- els. https://doi.org/10.18653/v1 /2024.acl-long.841 · DOI · Paper page
  17. Yanzhu Guo, Simone Conia, Zelin Zhou, Min Li, Saloni Potdar, and Henry Xiao. 2024a. Do large language models have an English accent? Evaluating and improving the natu- ralness of multilingual LLMs. arXiv preprint arXiv:2410.15956. · arXiv
  18. Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. 2024b. The curious decline of linguistic diversity: Training language models on synthetic text. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3589-3604, Mexico City, Mexico. Association for Com- putational Linguistics. https://doi.org/10 .18653/v1/2024.findings-naacl.228 · DOI
  19. Seungju Han, Beomsu Kim, and Buru Chang. 2022. Measuring and improving semantic di- versity of dialogue generation. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 934-950, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  20. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilin- gual abstractive summarization for 44 lan- guages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693-4703, Online. Association for Com- putational Linguistics. https://doi.org /10.18653/v1/2021.findings-acl.413 · DOI
  21. Shirley Anugrah Hayati, Minhwa Lee, Dheeraj Rajagopal, and Dongyeop Kang. 2023. How far can we extract diverse perspectives from large language models? Criteria-based diversity prompting! arXiv preprint arXiv:2311.09799. · arXiv
  22. Patrick G. T. Healey, Matthew Purver, and Christine Howes. 2014. Divergence in di- alogue. PloS One, 9(6):e98598. https://doi .org/10.1371/journal.pone.0098598, PubMed: 24919186 · DOI
  23. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring mas- sive multitask language understanding. In International Conference on Learning Repre- sentations.
  24. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
  25. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. · arXiv
  26. Wendell Johnson. 1944. Studies in language behavior: A program of research. Psycholog- ical Monographs, 56(2):1-15. https://doi .org/10.1037/h0093508 · DOI
  27. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large lan- guage models struggle to learn long-tail knowl- edge. In International Conference on Machine Learning, pages 15696-15707.
  28. PMLR.
  29. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. Understanding the effects of RLHF on LLM generalization and diversity. In The Twelfth International Conference on Learning Representations.
  30. Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghavendra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, and Jilin Chen. 2023. Improving diversity of demographic representation in large language models via collective-critiques and self-voting. In Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Pro- cessing, pages 10383-10405, Singapore. Association for Computational Linguis- tics. https://doi.org/10.18653/v1 /2023.emnlp-main.643 · DOI
  31. Florian Le Bronnec, Alexandre Verine, Benjamin Negrevergne, Yann Chevaleyre, and Alexandre Allauzen. 2024. Exploring precision and recall to assess the quality and diversity of LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11418-11441, Bangkok, Thailand. Association for Computa- tional Linguistics. https://doi.org/10 .18653/v1/2024.acl-long.616 · DOI
  32. Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christopher D. Manning, and James Y. Zou. 2024. Mapping the increasing use of LLMs in scientific papers. In First Conference on Language Modeling.
  33. Jiaming Luo, Colin Cherry, and George Foster. 2024. To diverge or not to diverge: A morphosyntactic perspective on machine trans- lation vs human translation. Transactions of the Association for Computational Linguis- tics, 12:355-371. https://doi.org/10 .1162/tacl_a_00645 · DOI
  34. Danielle S. McNamara, Scott A. Crossley, and Philip M. McCarthy. 2010. Linguistic features of writing quality. Written Commu- nication, 27(1):57-86. https://doi.org /10.1177/0741088309351547 · DOI · Paper page
  35. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mix- ture models. In International Conference on Learning Representations.
  36. J. F. Miller. 1981. Assessing Language Pro- duction in Children: Experimental Procedures. Assessing communicative behavior. University Park Press.
  37. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Mea- suring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems, vol- ume 34, pages 4816-4828.
  38. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Pro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Sys- tem Demonstrations, pages 101-108, Online. Association for Computational Linguistics.
  39. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimiza- tion: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.
  40. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702, Online. Association for Com- putational Linguistics. https://doi.org /10.18653/v1/2020.emnlp-main.213 · DOI
  41. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Associa- tion for Computational Linguistics. https:// doi.org/10.18653/v1/D19-1410 · DOI · Paper page
  42. Nils Reimers and Iryna Gurevych. 2020. Mak- ing monolingual sentence embeddings mul- tilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525, On- line. Association for Computational Linguis- tics. https://doi.org/10.18653/v1 /2020.emnlp-main.365 · DOI
  43. Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. Transactions of the Association for Computational Linguistics, 8:810-827. https://doi.org/10.1162 /tacl_a_00347 · DOI
  44. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language mod- els' sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations.
  45. Chantal Shaib, Yanai Elazar, Junyi Jessy Li, and Byron C. Wallace. 2024b. Detection and measurement of syntactic templates in generated text. In Proceedings of the 2024 Conference on Empirical Methods in Natu- ral Language Processing, pages 6416-6431, Miami, Florida, USA. Association for Compu- tational Linguistics. https://doi.org/10 .18653/v1/2024.emnlp-main.368 · DOI
  46. Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M. Borgwardt. 2011. Weisfeiler-lehman graph ker- nels. Journal of Machine Learning Research, 12(null):2539-2561.
  47. Giannis Siglidis, Giannis Nikolentzos, Stratis Limnios, Christos Giatsidis, Konstantinos Skianis, and Michalis Vazirgiannis. 2020. Grakel: A graph kernel library in python. Jour- nal of Machine Learning Research, 21(54):1-5.
  48. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. 2024. Dolma: An open corpus of three trillion to- kens for language model pretraining research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725-15788, Bangkok, Thailand. Association for Computa- tional Linguistics. https://doi.org/10 .18653/v1/2024.acl-long.840 · DOI · Paper page
  49. Rafael Alberto Rivera Soto, Kailin Koch, Aleem Khan, Barry Y. Chen, Marcus Bishop, and Nicholas Andrews. 2024. Few-shot detection of machine-generated text using style representa- tions. In The Twelfth International Conference on Learning Representations.
  50. Katherine Stasaski and Marti Hearst. 2022. Se- mantic diversity in dialogue with natural language inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 85-98, Seattle, United States. Association for Compu- tational Linguistics. https://doi.org/10 .18653/v1/2022.naacl-main. · DOI
  51. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open lan- guage models at a practical size. arXiv preprint arXiv:2408.00118. · arXiv
  52. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  53. Adaku Uchendu, Jooyoung Lee, Hua Shen, Thai Le, Dongwon Lee. 2023. Does hu- man collaboration enhance the accuracy of identifying LLM-generated deepfake texts? In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol- ume 11, pages 163-174. https://doi.org /10.1609/hcomp.v11i1.27557 · DOI
  54. Jan Philip Wahle, Terry Ruas, Yang Xu, and Bela Gipp. 2024. Paraphrase types elicit prompt engi- neering capabilities. In Proceedings of the 2024 Conference on Empirical Methods in Natu- ral Language Processing, pages 11004-11033, Miami, Florida, USA. Association for Compu- tational Linguistics. https://doi.org/10 .18653/v1/2024.emnlp-main.617 · DOI
  55. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. MMLU-pro: A more robust and challenging multi-task lan- guage understanding benchmark. arXiv preprint arXiv:2406.01574. · arXiv
  56. Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. · arXiv
  57. Denis Yarats and Mike Lewis. 2018. Hierarchical text generation and planning for strategic dia- logue. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5591-5599. PMLR.
  58. Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2021. Trad- ing off diversity and quality in natural language generation. In Proceedings of the Work- shop on Human Evaluation of NLP Systems (HumEval), pages 25-33, Online. Association for Computational Linguistics.
  59. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
  60. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024. SafetyBench: Evaluating the safety of large language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15537-15553, Bangkok, Thailand. Association for Computa- tional Linguistics. https://doi.org/10 .18653/v1/2024.acl-long.830 · DOI

Authors
Guo, Y., Shang, G., & Clavel, C.
Affiliations
  • Institut national de recherche en sciences et technologies du numérique (FR)
  • Institut français de la mode (FR)
ORCID
Clavel, Chloé
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2025
Volume
13
Pages
1507-1526
ISSN
2307-387X (ISSN-L)
DOI
10.1162/tacl.a.47
OpenAlex
W7106340822
Cited by
11
References
36
Citations by year
2026: 11
FWCI
4.45
Type
journal article
Language
en
Open access
Licence
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
Agence Nationale de la Recherche (ANR-23-CE23-0033)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
linguistic diversity, large language models, lexical diversity, syntactic diversity, semantic diversity, language generation, benchmarking, story generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review