Evaluating the Evaluation of Diversity in Natural Language Generation
Open accessAbstract
Despite growing interest in natural language generation (NLG) models that produce diverse outputs, there is currently no principled method for evaluating the diversity of an NLG system.In this work, we propose a framework for evaluating diversity metrics.The framework measures the correlation between a proposed diversity metric and a diversity parameter, a single parameter that controls some aspect of diversity in generated text.For example, a diversity parameter might be a binary variable used to instruct crowdsourcing workers to generate text with either low or high content diversity.We demonstrate the utility of our framework by: (a) establishing best practices for eliciting diversity judgments from humans, (b) showing that humans substantially outperform automatic metrics in estimating content diversity, and (c) demonstrating that existing methods for controlling diversity by tuning a "decoding parameter" mostly affect form but not meaning.Our framework can advance the understanding of different diversity metrics, an essential step on the road towards better NLG systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The diversity metrics (distinct N-grams, Sentence-BERT, NLI) align well with human diversity evaluations and underlying diversity generators.
“All of these metrics have been shown to align well with human diversity evaluations and with underlying diversity generators by Tevet & Berant (2021).”
From Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023) · cites this paper as Tevet & Berant (2021)
The paper’s Set B overview reports that n-gram metrics correlate well with decoding parameters, and its Results section finds that neural metrics—especially Sentence-BERT—perform better on content diversity. But the Results also say n-gram metrics perform poorly on content diversity and humans substantially outperform every automatic metric; the claim that all these metrics align well with human evaluations is therefore overstated. The paper identifies an NLI-trained Sentence-BERT model, not a separately evaluated NLI metric. Page numbers were not available in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816 · see citing works
A Learning Algorithm for Boltzmann Machines
Ackley, D., Hinton, G. E., & Sejnowski, T. (1985) · Cognitive Sciences · doi:10.1016/b978-0-08-051581-6.50053-2 · Cited by 4,462 · see citing works
(2005) · Cited by 4,152 · see citing works
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Yu, L., Zhang, W., Wang, Y.-X., & Yu, Y. H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10804 · Cited by 2,342 · see citing works
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 1,981 · see citing works
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2001 · Cited by 1,562 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242 · see citing works
MASS: Masked Sequence to Sequence Pre-training for Language GenerationPreprint
Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02450 · Cited by 834 · see citing works
A Survey of Text Similarity Approaches
Gomaa, W. H., & Fahmy, A. A. (2013) · International Journal of Computer Applications · doi:10.5120/11638-7118 · Cited by 823 · see citing works
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · see citing works
A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., & Allen, J. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1098 · Cited by 598 · see citing works
(2020) · arXiv (Cornell University) · Cited by 461 · see citing works
Texygen: A Benchmarking Platform for Text Generation Models
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., & Yu, Y. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210080 · Cited by 408 · see citing works
Deal or No Deal? End-to-End Learning of Negotiation Dialogues
Lewis, M., Yarats, D., Dauphin, Y., Parikh, D., & Batra, D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1259 · Cited by 347 · see citing works
Danescu-Niculescu-Mizil, C., & Lee, L. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1106.3077 · Cited by 329 · see citing works
ELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319 · see citing works
A Simple, Fast Diverse Decoding Algorithm for Neural GenerationPreprint
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv.org · doi:10.48550/arxiv.1611.08562 · Cited by 261 · see citing works
Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge
Dušek, O., Novikova, J., & Rieser, V. (2019) · Computer Speech & Language · doi:10.1016/j.csl.2019.06.009 · Cited by 194 · see citing works
Texygen: A Benchmarking Platform for Text Generation ModelsPreprint
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., & Yu, Y. H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01886 · Cited by 168 · see citing works
Unifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161 · see citing works
Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., & Charlin, L. (2020) · International Conference on Learning Representations · Cited by 103 · see citing works
Enhancing Topic-to-Essay Generation with External Commonsense Knowledge
Yang, P., Li, L., Luo, F., Liu, T., & Sun, X. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1193 · Cited by 91 · see citing works
Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., & Charlin, L. (2018) · arXiv · doi:10.48550/arxiv.1811.02549 · Cited by 83 · see citing works
Recent Advances in Neural Question GenerationPreprint
Pan, L., Lei, W., Chua, T., & Kan, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08949 · Cited by 79 · see citing works
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog SystemsPreprint
Ghandeharioun, A., Shen, J. H., Jaques, N., Ferguson, C., Jones, N. C., Lapedriza, À., & Picard, R. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09308 · Cited by 77 · see citing works
Syntax-Infused Variational Autoencoder for Text Generation
Zhang, X., Yang, Y., Yuan, S., Shen, D., & Carin, L. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1199 · Cited by 51 · see citing works
PaperRobot: Incremental Draft Generation of Scientific Ideas
Wang, Q., Huang, L., Jiang, Z., Knight, K. K., Ji, H., Bansal, M., & Luan, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1191 · Cited by 48 · see citing works
Generating Diverse Translations with Sentence Codes
Shu, R., Nakayama, H., & Cho, K. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1177 · Cited by 35 · see citing works
Li, Z., Ding, X., & Liu, T. (2018) · International Conference on Computational Linguistics · Cited by 33 · see citing works
Generating Long and Informative Reviews with Aspect-Aware Coarse-to-Fine Decoding
Li, J., Zhao, W. X., Wen, J.-R., & Song, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1190 · Cited by 31 · see citing works
Boosting Dialog Response Generation
Du, W., & Black, A. W. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1005 · Cited by 28 · see citing works
Quality Signals in Generated Stories
Sagarkar, M., Wieting, J., Tu, L., & Gimpel, K. (2018) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2024 · Cited by 16 · see citing works
Evaluating Text GANs as Language Models
Tevet, G., Habib, G., Shwartz, V., & Berant, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1233 · Cited by 12 · see citing works
Self-citationDiversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text CollectionsPreprint
Lai, Y.-A., Zhu, X., Zhang, Y., & Diab, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08529 · Cited by 2 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 1
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 110-119.
Cited by
Sorted by citations
GenAug: Data Augmentation for Finetuning Text Generators
Feng, S. Y., Gangal, V., Kang, D., Mitamura, T., & Hovy, E. H. (2020) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2020.deelio-1.4 · Cited by 69 · see citing works
Result Diversification in Search and Recommendation: A Survey
Wu, H., Zhang, Y., Ma, C., Lyu, F., He, B., Mitra, B., & Liu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3382262 · Cited by 54 · see citing works
Are Some Words Worth More than Others?
Dudy, S., & Bedrick, S. D. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.13 · Cited by 33 · see citing works
Paraphrase Identification With Deep Learning: A Review of Datasets and Methods
Zhou, C., Qiu, C. J., Liang, L., & Acuna, D. E. (2025) · IEEE Access · doi:10.1109/access.2025.3556899 · Cited by 33 · see citing works
Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs
Wang, S., Wei, Z., Choi, Y., & Ren, X. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.406 · Cited by 32 · see citing works
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation
Ke, P., Zhou, H., Lin, Y., Li, P., Zhou, J., Zhu, X., & Huang, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.164 · Cited by 27 · see citing works
Multi-Teacher Distillation With Single Model for Neural Machine Translation
Liang, X., Wu, L., Li, J., Qin, T., Zhang, M., & Liu, T. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3153264 · Cited by 25 · see citing works
Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts
Yu, W., Zhu, C., Qin, L., Zhang, Z., Zhao, T., & Jiang, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.dlg4nlp-1.1 · Cited by 23 · see citing works
Decoding and Diversity in Machine TranslationPreprint
Roberts, N. P., Liang, D., Neubig, G., & Lipton, Z. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13477 · Cited by 20 · see citing works
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17 · see citing works
“Our work extends these works by making diversity evaluation a primary focus and using diversity metrics beyond self-BLEU which have been externally validated (Tevet & Berant, 2021) and measure diversity in a range of different ways.”
MethodDecoding Methods for Neural Narrative Generation
DeLucia, A., Mueller, A., Li, X. L., & Sedoc, J. (2021) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2021.gem-1.16 · Cited by 17 · see citing works
Semantic Diversity in Dialogue with Natural Language Inference
Stasaski, K., & Hearst, M. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.6 · Cited by 17 · see citing works
DECODING AI AND HUMAN AUTHORSHIP: NUANCES REVEALED THROUGH NLP AND STATISTICAL ANALYSIS
Akinwande, M. J., Yussuph, T., & Adeliyi, O. (2024) · International Journal on Cybernetics & Informatics · doi:10.5121/ijci.2024.130408 · Cited by 14 · see citing works
Benchmarking Linguistic Diversity of Large Language Models
Guo, Y., Shang, G., & Clavel, C. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.47 · Cited by 11 · see citing works
Exploring diversity in back translation for low-resource machine translation
Burchell, L., & Heafield, A. B. A. K. (2022) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/2022.deeplo-1.8 · Cited by 10 · see citing works
Measuring and Improving Semantic Diversity of Dialogue Generation
Han, S., Kim, B., & Chang, B. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.66 · Cited by 9 · see citing works
One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity
Murthy, S. K., Ullman, T., & Hu, J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.561 · Cited by 8 · see citing works
Calderon, N., Mukherjee, S., Reichart, R., & Kantor, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.818 · Cited by 8 · see citing works
"When AI Writes Personas": Analyzing Lexical Diversity in LLM-Generated Persona Descriptions
Sethi, S., Salminen, J. O., Amin, D., & Jansen, B. J. J. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719712 · Cited by 7 · see citing works
Lee, H., Yoo, K. M., Park, J., Lee, H., & Jung, K. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.76 · Cited by 7 · see citing works
Giulianelli, M., Baan, J., Aziz, W. F., Fernández, R., & Plank, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.887 · Cited by 7 · see citing works
PROTEGE: Prompt-based Diverse Question Generation from Web Articles
Puranik, V. S., Majumder, A., & Chaoji, V. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.362 · Cited by 4 · see citing works
BioSpark: Beyond Analogical Inspiration to LLM-augmented Transfer
Kang, H. B., Lin, D. C.-E., Chen, Y.-Y., Hong, M. K., Martelaro, N., & Kittur, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714053 · Cited by 4 · see citing works
Zhao, J., Song, Y., & Cohen, E. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112493 · Cited by 4 · see citing works
4W1H Keyword Extraction based Summarization Model
Lee, S., Park, T., & Lee, M. (2021) · International Conference on Electronics, Information, and Communication (ICEIC) · doi:10.1109/iceic51217.2021.9369820 · Cited by 4 · see citing works
ChatGPT4PCG 2 Competition: Prompt Engineering for Science Birds Level Generation
Taveekitworachai, P., Abdullah, F., Dewantoro, M. F., Xia, Y., Suntichaikul, P., Thawonmas, R., Togelius, J., & Renz, J. (2024) · IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645641 · Cited by 4 · see citing works
MultiTalk: A Highly-Branching Dialog Testbed for Diverse Conversations
Dou, Y., Forbes, M., Holtzman, A., & Choi, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17510 · Cited by 4 · see citing works
DivHSK: Diverse Headline Generation using Self-Attention based Keyword Selection
Venkatesh, E., Maurya, K. K., Kumar, D., & Desarkar, M. S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.118 · Cited by 3 · see citing works
CASSI: Contextual and Semantic Structure-based Interpolation Augmentation for Low-Resource NER
Surana, T., Ho, T.-N., Tun, K., & Chng, E. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.651 · Cited by 3 · see citing works
Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and SamplingPreprint
Yang, Z., Xu, P., & Wan, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03641 · Cited by 3 · see citing works
DeepGen: Diverse Search Ad Generation and Real-Time Customization
Golobokov, K., Chai, J., Dong, V. Y., Gu, M., Chi, B., Cao, J., Yan, Y., & Liu, Y. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.19 · Cited by 2 · see citing works
Sun, W., He, S., Zhao, J. W., & Liu, K. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3546556 · Cited by 2 · see citing works
Preference-Controlled Multi-Objective Reinforcement Learning for Conditional Text Generation
Chen, W., Tian, J., Fan, C., Li, Y., He, H., & Jin, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26490 · Cited by 2 · see citing works
Evaluating the Diversity and Quality of LLM Generated Content
Shypula, A., Li, S., Zhang, B., Padmakumar, V., Yin, K., & Bastani, O. (2025) · arXiv · doi:10.48550/arxiv.2504.12522 · Cited by 1 · see citing works
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment
Lake, T., Choi, E., & Durrett, G. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.346 · Cited by 1 · see citing works
LiftFuzz: Validating Binary Lifters through Context-aware Fuzzing with GPT
Zhou, Y., Yang, F., Song, Z., Zhang, K., Chen, J., & Zhang, K. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3670276 · Cited by 1 · see citing works
Better Red Teaming via Searching with Large Language Model
Chen, Y., Zhao, C. K., Jianwentian, J., Cao, G., Li, H., & Kuang, X. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.257 · Cited by 1 · see citing works
Ming, L., Zhong, S.-H., & Li, Y. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.45 · Cited by 1 · see citing works
HAUSER: Towards Holistic and Automatic Evaluation of Simile Generation
He, Q., Zhang, Y., Liang, J., Huang, Y., Xiao, Y., & Chen, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.702 · Cited by 1 · see citing works
Machine Reading Comprehension using Case-based Reasoning
Thai, D., Agarwal, D., Chaudhary, M., Zhao, W., Das, R., Lee, J. Y., Hajishirzi, H., Zaheer, M., & McCallum, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.564 · Cited by 1 · see citing works
Neural Text Generation with Part-of-Speech Guided Softmax.Preprint
Yang, Z., & Wan, X. (2021) · arXiv (Cornell University) · Cited by 1 · see citing works
Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac SignalsPreprint
Kiyasseh, D., Zhu, T., & Clifton, D. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11011
Large Scale Fine-Tuned Transformers Models Application for Business Names Generation
Lukauskas, M., Rasymas, T., Minelga, M., & Vaitmonas, D. (2023) · Computing and Informatics · doi:10.31577/cai_2023_3_525
Allein, L., Truşcǎ, M. M., & Moens, M. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104234
Li, M., Lu, X., & Zhao, Y. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.380
Learning to vary: Teaching LMs to reproduce human linguistic variability in next-word prediction
Groot, T., Lacunes, S., & Ilia, E. (2025) · Workshop on Uncertainty-Aware NLP (UncertaiNLP) · doi:10.18653/v1/2025.uncertainlp-main.9
Toward Optimal LLM Alignments Using Two-Player Games
Zheng, R., Guo, H., Liu, Z., Zhang, X., Yao, Y., Xu, X., Wang, Z., Xi, Z., Gui, T., Zhang, Q., et al. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.6
Afzal, A., Mercier, A., & Matthes, F. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-0887-7_27
Wang, W., & Zheng, W. (2025) · International Conference on Educational Technology (ICET) · doi:10.1109/icet67421.2025.11380603
IVCN: A Siamese Network for Quantifying the Inspirational Value of Text
Zhang, J., Yang, W., & Ji, Y. (2025) · International Symposium on Artificial Intelligence and Big Data (AIBDF) · doi:10.1109/aibdf67964.2025.11440887
Pastrián, D., Hidalgo, N., Reyes, V., & Rosas, E. (2026) · Applied Sciences · doi:10.3390/app16083623
A Persona-Augmented Multi-Agent System for Varied Narrative Generation
Sillano, A., Arbore, G., & Russis, L. D. (2026) · ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) · doi:10.1145/3807968.3810931
Menghan, Z., Khan, I., Zifan, Y., & Thawonmas, R. (2026) · Entertainment Computing · doi:10.1016/j.entcom.2026.101172
Salemi, A., Samarinas, C., & Zamani, H. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820415
Goudarzi, A., & Zonouz, A. N. (2026) · Digital studies in language and literature · doi:10.1515/dsll-2026-0052
- Authors
- Tevet, G., & Berant, J.
- Affiliations
- Tel Aviv University (IL)
- ORCID
- Tevet, Guy
- Venue
- Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
- Published
- 2021
- Pages
- 326-346
- OpenAlex
- W3015962271
- Cited by
- 52
- References
- 41
- Citations by year
- 2026: 6 · 2025: 14 · 2024: 6 · 2023: 9 · 2022: 8 · 2021: 6 · 2020: 3
- FWCI
- 3.23
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- European Commission
- Israel Science Foundation
- H2020 European Research Council (802800)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- natural language generation, output diversity, diversity indices, content diversity, crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
