A Survey of Evaluation Metrics Used for NLG Systems
Open accessAbstract
In the last few years, a large number of automatic evaluation metrics have been proposed for evaluating Natural Language Generation (NLG) systems. The rapid development and adoption of such automatic evaluation metrics in a relatively short time has created the need for a survey of these metrics. In this survey, we (i) highlight the challenges in automatically evaluating NLG systems, (ii) propose a coherent taxonomy for organising existing evaluation metrics, (iii) briefly describe different existing metrics, and finally (iv) discuss studies criticising the use of automatic evaluation metrics. We then conclude the article highlighting promising future directions of research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Intrinsic metrics lack the capability to capture semantics, context, distant dependencies, semantically critical ordering change, and human perspectives, particularly in real-world scenarios.
“These metrics lack the capability to capture essential elements such as semantics19,20, context19,21, distant dependencies22,23, semantically critical ordering change21, and human perspectives, particularly in real-world scenarios.”
From Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI (2024) · cites this paper as [19]
Intrinsic evaluation metrics solely rely on surface-form similarity and language-specific perspectives, rendering them inadequate for healthcare chatbots.
“However, they solely rely on surface-form similarity and language-specific perspectives, rendering them inadequate for healthcare chatbots.”
From Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI (2024) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Draper, N. R., & Smith, H. (2005) · Technometrics · doi:10.1198/tech.2005.s303 · Cited by 17,993 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Indexing by latent semantic analysis
Deerwester, S. C., Dumais, S., Furnas, G. W., Landauer, T. K., & Harshman, R. A. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6 · Cited by 12,758 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
(2016) · Cited by 11,679 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1× · see citing works
“Indeed, the Turing test [160], widely considered to be the ultimate test of a machine’s ability to exhibit human-like intelligent behaviour requires a machine to have natural language conversations with a human evaluator.”
Background(2004) · Cited by 9,718 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Landauer, T. K., & Dumais, S. T. (1997) · Psychological Review · doi:10.1037/0033-295x.104.2.211 · Cited by 6,282 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Computing Machinery and Intelligence (1950)
Turing, A. M. (2004) · doi:10.1093/oso/9780198250791.003.0017 · Cited by 6,120 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
(2005) · Cited by 4,152 · see citing works
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
Recent Trends in Deep Learning Based Natural Language ProcessingPreprint
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · IEEE Computational Intelligence Magazine · doi:10.48550/arxiv.1708.02709 · Cited by 3,133 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Understanding natural language
Winograd, T. A. (1972) · Cognitive Psychology · doi:10.1016/0010-0285(72)90002-3 · Cited by 2,803 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,765 · see citing works
A Study of Translation Edit Rate with Targeted Human Annotation
Snover, M., Dorr, B. J., Schwartz, R. M., Micciulla, L., & Makhoul, J. I. (2006) · Cited by 2,380 · see citing works
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361 · see citing works
A Call for Clarity in Reporting BLEU Scores
Post, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6319 · Cited by 2,112 · see citing works
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · see citing works
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., & Bordes, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1070 · Cited by 2,049 · see citing works
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning
Lu, J., Xiong, C., Parikh, D., & Socher, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.345 · Cited by 1,724 · see citing works
From Word Embeddings To Document Distances
Kusner, M. J., Sun, Y., Kolkin, N., & Weinberger, K. Q. (2015) · PolyPublie (École Polytechnique de Montréal) · Cited by 1,695 · see citing works
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Serban, I. V., Sordoni, A., Bengio, Y., Courville, A. C., & Pineau, J. (2016) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v30i1.9883 · Cited by 1,639 · see citing works
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics
Doddington, G. R. (2002) · International Conference on Human Language Technology Research · doi:10.3115/1289189.1289273 · Cited by 1,603 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · see citing works
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,349 · see citing works
Cer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · arXiv · doi:10.48550/arxiv.1803.11175 · Cited by 1,267 · see citing works
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,262 · see citing works
A Primer on Neural Network Models for Natural Language Processing
Goldberg, Y. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4992 · Cited by 1,259 · see citing works
Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., & Moy, L. P. (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859894 · Cited by 1,195 · see citing works
MS MARCO: A Human Generated MAchine Reading COmprehension DatasetPreprint
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T. G., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.giscience.2025.5 · Cited by 1,190 · see citing works
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. Z. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1152 · Cited by 1,186 · see citing works
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177 · see citing works
Understanding Back-Translation at Scale
Edunov, S., Ott, M., Auli, M., & Grangier, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1045 · Cited by 1,044 · see citing works
(2016) · doi:10.18653/v1/d16-1147 · Cited by 1,035 · see citing works
chrF: character n-gram F-score for automatic MT evaluation
Popović, M. (2015) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w15-3049 · Cited by 1,026 · see citing works
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005 · see citing works
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005
Gao, Y., Beijbom, O., Zhang, N., & Darrell, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.41 · Cited by 912 · see citing works
The power of outliers (and why researchers should ALWAYS check for them)
Osborne, J. W., & Overbay, A. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/qf69-7k43 · Cited by 898 · see citing works
Adversarial Learning for Neural Dialogue Generation
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., & Jurafsky, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1230 · Cited by 880 · see citing works
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gašić, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. J. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1199 · Cited by 876 · see citing works
WordNet: similarity - measuring the relatedness of concepts
Pedersen, T., Patwardhan, S., & Michelizzi, J. (2004) · Cited by 854 · see citing works
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776 · see citing works
Findings of the 2014 Workshop on Statistical Machine Translation
Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., et al. (2014) · Conference on Machine Translation (WMT) · doi:10.3115/v1/w14-3302 · Cited by 705 · see citing works
The NarrativeQA Reading Comprehension Challenge
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00023 · Cited by 596 · cited 1× · see citing works
“WikiMovies[109], VQA[5] [36] ROUGE, CIDEr, METEOR, BLEU, CS D2T WikiBio[157] [19] BLEU, ROUGE, METEOR, VecSim QA/AS MS MARCO[112], CNN/DailyMail[60,110] [20] BLEU, ROUGE, SMS, BERTScore QA NarrativeQA[71], SemEval[117], ROPES[83] [84] BLEU, ROUGE, METEOR, Greedy matching, vector extrema, vector avg DG Twitter[133], Ub…”
BackgroundSummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587 · see citing works
Findings of the 2016 Conference on Machine Translation
Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., et al. (2016) · Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers · doi:10.18653/v1/w16-2301 · Cited by 539 · see citing works
Challenges in Data-to-Document Generation
Wiseman, S., Shieber, S., & Rush, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1239 · Cited by 524 · see citing works
Procedures as a representation for data in a computer program for understanding natural language
Winograd, T. (1971) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 521 · see citing works
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1053 · Cited by 490 · see citing works
Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability
Clark, J. H., Dyer, C., Lavie, A., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473090 · Cited by 443 · see citing works
(2017) · doi:10.18653/v1/p17-1017 · Cited by 436 · see citing works
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational AgentsPreprint
Wolf, T., Sanh, V., Chaumond, J., & Delangue, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08149 · Cited by 435 · see citing works
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Liu, S., Zhu, Z., Ye, N., Guadarrama, S., & Murphy, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.100 · Cited by 430 · see citing works
An efficient framework for learning sentence representations
Logeswaran, L., & Lee, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02893 · Cited by 427 · see citing works
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · see citing works
Unsupervised Modeling of Twitter Conversations
Ritter, A., Cherry, C., & Dolan, B. (2010) · NPARC · Cited by 419 · see citing works
Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk
Callison-Burch, C. (2009) · Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 1 - EMNLP '09 · doi:10.3115/1699510.1699548 · Cited by 412 · see citing works
Deep Learning in Natural Language Processing
(2018) · Springer eBooks · doi:10.1007/978-981-10-5209-5 · Cited by 408 · see citing works
A Structured Review of the Validity of BLEU
Reiter, E. (2018) · Computational Linguistics · doi:10.1162/coli_a_00322 · Cited by 386 · see citing works
Coarse-to-Fine Decoding for Neural Semantic Parsing
Dong, L., & Lapata, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1068 · Cited by 378 · see citing works
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., & Pineau, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1103 · Cited by 341 · see citing works
Neural Text Summarization: A Critical Evaluation
Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1051 · Cited by 335 · see citing works
Hierarchical Recurrent Neural Networks for Long-Term Dependencies
Hihi, S. E., & Bengio, Y. (1995) · Cited by 319 · see citing works
Wieting, J., & Gimpel, K. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1042 · Cited by 318 · see citing works
(2004) · doi:10.3115/1220355.1220427 · Cited by 294 · see citing works
chrF++: words helping character n-grams
Popović, M. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4770 · Cited by 286 · see citing works
An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research
Nießen, S., Och, F. J., Leusch, G., et al. (2000) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/56t9t4rchhar · Cited by 284 · see citing works
Image Description using Visual Dependency Representations
Elliott, D., & Keller, F. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1128 · Cited by 282 · see citing works
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
Lowe, R., Pow, N., Serban, I. V., & Pineau, J. (2015) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w15-4640 · Cited by 282 · see citing works
DailyDialog: A Manually Labelled Multi-Turn Dialogue Dataset
Li, Y. (2025) · TIB Data Manager · doi:10.57702/l9x4bl6j · Cited by 262 · see citing works
Neural AMR: Sequence-to-Sequence Models for Parsing and Generation
Konstas, I., Iyer, S. V., Yatskar, M., Choi, Y., & Zettlemoyer, L. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1014 · Cited by 255 · see citing works
Evaluation of Machine Translation and its Evaluation
Turian, J. P., Shea, L., & Melamed, I. D. (2006) · doi:10.21236/ada453509 · Cited by 250 · see citing works
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247 · see citing works
Accelerated DP based search for statistical translation
Tillmann, C., Vogel, S., Ney, H., Zubiaga, A., & Sawaf, H. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-673 · Cited by 233 · see citing works
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics
Mathur, N., Baldwin, T., & Cohn, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.448 · Cited by 206 · see citing works
Reiter, E., & Belz, A. (2009) · Computational Linguistics · doi:10.1162/coli.2009.35.4.35405 · Cited by 204 · see citing works
Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge
Dušek, O., Novikova, J., & Rieser, V. (2019) · Computer Speech & Language · doi:10.1016/j.csl.2019.06.009 · Cited by 194 · see citing works
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity
Wieting, J., Berg-Kirkpatrick, T., Gimpel, K., & Neubig, G. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1427 · Cited by 190 · see citing works
Interpreting BLEU/NIST Scores: How Much Improvement do We Need to Have a Better System?
Zhang, Y., Vogel, S., & Waibel, A. (2004) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2qqhpv7hcigk · Cited by 167 · see citing works
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · arXiv · doi:10.48550/arxiv.1801.07243 · Cited by 165 · see citing works
Summarization Evaluation: An Overview.
Mani, I. (2001) · Cited by 156 · see citing works
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Dhingra, B., Faruqui, M., Parikh, A., Chang, M.-W., Das, D., & Cohen, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1483 · Cited by 150 · see citing works
Re-evaluating Automatic Metrics for Image Captioning
Kilickaya, M., Erdem, A., İkizler-Cinbiş, N., & Erdem, E. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1019 · Cited by 149 · see citing works
Can machine translation systems be evaluated by the crowd alone
Graham, Y., Baldwin, T., Moffat, A., & Zobel, J. (2015) · Natural Language Engineering · doi:10.1017/s1351324915000339 · Cited by 148 · see citing works
Continuous Measurement Scales in Human Evaluation of Machine Translation
Graham, Y., Baldwin, T., Moffat, A., & Zobel, J. (2013) · Trinity's Access to Research Output (TARA) (Trinity College Dublin) · Cited by 146 · see citing works
Evaluation Measures for Text Summarization
Steinberger, J., & Ježek, K. (2012) · Cited by 145 · see citing works
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges
Ma, Q., Wei, J. T.-Z., Bojar, O., & Graham, Y. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5302 · Cited by 144 · see citing works
Learning to Evaluate Image Captioning
Cui, Y., Yang, G., Veit, A., Huang, X., & Belongie, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00608 · Cited by 140 · see citing works
Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts
Clark, E., Çelikyılmaz, A., & Smith, N. A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1264 · Cited by 138 · see citing works
Comparing Automatic Evaluation Measures for Image Description
Elliott, D., & Keller, F. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2074 · Cited by 137 · see citing works
CharacTer: Translation Edit Rate on Character Level
Wang, W., Peter, J.-T., Rosendahl, H., & Ney, H. (2016) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w16-2342 · Cited by 134 · see citing works
Results of the WMT16 Metrics Shared Task
Bojar, O., Graham, Y., Kamran, A., & Stanojević, M. (2016) · Conference on Machine Translation · doi:10.18653/v1/w16-2302 · Cited by 128 · see citing works
Evaluation metrics for generation
Bangalore, S., Rambow, O., & Whittaker, S. (2000) · doi:10.3115/1118253.1118255 · Cited by 127 · see citing works
Evaluating Evaluation Methods for Generation in the Presence of Variation
Stent, A., Marge, M., & Singhai, M. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30586-6_38 · Cited by 126 · see citing works
Results of the WMT17 Metrics Shared Task
Bojar, O., Graham, Y., & Kamran, A. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4755 · Cited by 122 · see citing works
Correlating automated and human assessments of machine translation quality
Coughlin, D. A. (2003) · Cited by 118 · see citing works
Phrase-Based Statistical Language Generation Using Graphical Models and Active Learning
Mairesse, F., Gašić, M., Jurčíček, F., Keizer, S., Thomson, B., Yu, K., & Young, S. J. (2010) · Cited by 118 · see citing works
Rus, V., & Lintean, M. C. (2012) · Cited by 117 · see citing works
RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog SystemsPreprint
Tao, C., Mou, L., Zhao, D., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.03079 · Cited by 117 · see citing works
SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge
Ostermann, S., Roth, M., Modi, A., Thater, S., & Pinkal, M. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1119 · Cited by 114 · see citing works
Lo, C.-K. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5358 · Cited by 114 · see citing works
A New Quantitative Quality Measure for Machine Translation Systems
Su, K.-Y., Wu, M.-W., & Chang, J.-S. (1992) · International Conference on Computational Linguistics · doi:10.3115/992133.992137 · Cited by 112 · see citing works
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn ComparisonsPreprint
Li, M., Weston, J., & Roller, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03087 · Cited by 111 · see citing works
CDER: Efficient MT Evaluation Using Block Movements.
Leusch, G., Ueffing, N., & Ney, H. (2006) · RWTH Publications (RWTH Aachen) · Cited by 109 · see citing works
Towards a Better Metric for Evaluating Question Generation Systems
Nema, P., & Khapra, M. M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1429 · Cited by 106 · see citing works
Self-citationResults of the WMT14 Metrics Shared Task
Macháċek, M., & Bojar, O. (2014) · Conference on Machine Translation (WMT) · doi:10.3115/v1/w14-3336 · Cited by 105 · see citing works
Evaluating Question Answering Evaluation
Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5817 · Cited by 105 · see citing works
Image Understanding using vision and reasoning through Scene Description Graph
Aditya, S., Yang, Y., Baral, C. R., Aloimonos, Y., & Fermuller, C. M. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.12.004 · Cited by 103 · see citing works
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., & Dolan, B. (2015) · doi:10.3115/v1/p15-2073 · Cited by 101 · see citing works
Results of the WMT15 Metrics Shared Task
Stanojević, M., Kamran, A., Koehn, P., et al. (2015) · Workshop on Statistical Machine Translation · doi:10.18653/v1/w15-3031 · Cited by 98 · see citing works
The price of debiasing automatic metrics in natural language evalaution
Chaganty, A., Mussmann, S., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1060 · Cited by 96 · see citing works
The challenging task of summary evaluation: an overview
Lloret, E., Plaza, L., & Aker, A. (2017) · Language Resources and Evaluation · doi:10.1007/s10579-017-9399-2 · Cited by 92 · see citing works
Reasoning Over Paragraph Effects in Situations
Lin, K., Tafjord, O., Clark, P. E., & Gardner, M. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5808 · Cited by 92 · see citing works
Asiya: An Open Toolkit for Automatic Machine Translation (Meta-)Evaluation
Giménez, J., & Màrquez, L. (2010) · The Prague Bulletin of Mathematical Linguistics · doi:10.2478/v10108-010-0022-6 · Cited by 91 · see citing works
BEER: BEtter Evaluation as Ranking
Stanojević, M., & Sima’an, K. (2014) · Conference on Machine Translation (WMT) · doi:10.3115/v1/w14-3354 · Cited by 91 · see citing works
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance
Ma, Q., Bojar, O., & Graham, Y. (2018) · doi:10.18653/v1/w18-6450 · Cited by 90 · see citing works
RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation
Shimanaka, H., Kajiwara, T., & Komachi, M. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6456 · Cited by 89 · see citing works
Lo, C.-K., & Wu, D. (2011) · Cited by 87 · see citing works
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings
Ghazarian, S., Wei, J. T.-Z., Galstyan, A., & Peng, N. (2019) · doi:10.18653/v1/w19-2310 · Cited by 84 · see citing works
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog SystemsPreprint
Ghandeharioun, A., Shen, J. H., Jaques, N., Ferguson, C., Jones, N. C., Lapedriza, À., & Picard, R. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09308 · Cited by 77 · see citing works
METEOR-NEXT and the METEOR Paraphrase Tables: Improved Evaluation Support for Five Target Languages
Denkowski, M., & Lavie, A. (2010) · Cited by 72 · see citing works
Fully Automatic Semantic MT Evaluation
Lo, C.-K., Tumuluru, A. K., & Wu, D. (2012) · Cited by 67 · see citing works
Improving summaries by revising them
Mani, I., Gates, B., & Bloedorn, E. (1999) · doi:10.3115/1034678.1034761 · Cited by 66 · see citing works
Learning an Unreferenced Metric for Online Dialogue Evaluation
Sinha, K., Parthasarathi, P., Wang, J., Lowe, R., Hamilton, W. L., & Pineau, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.220 · Cited by 64 · see citing works
Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation
Tian, R., Narayan, S., Sellam, T., & Parikh, A. P. (2019) · arXiv · doi:10.48550/arxiv.1910.08684 · Cited by 60 · see citing works
Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining
Sai, A. B., Mohankumar, A. K., Arora, S., & Khapra, M. M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00347 · Cited by 58 · see citing works
Self-citationFitting Sentence Level Translation Evaluation with Many Dense Features
Stanojević, M., & Sima’an, K. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1025 · Cited by 56 · see citing works
Results of the WMT13 Metrics Shared Task
Macháċek, M., & Bojar, O. (2013) · Workshop on Statistical Machine Translation · Cited by 56 · see citing works
Introducing Shared Tasks to NLG: The TUNA Shared Task Evaluation Challenges
Albert Gatt, A., & Belz, A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15573-4_14 · Cited by 55 · see citing works
Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation
Mathur, N., Baldwin, T., & Cohn, T. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1269 · Cited by 54 · see citing works
Hutchins, J. (1997) · Machine Translation · doi:10.1023/a:1007969630568 · Cited by 53 · see citing works
MEANT 2.0: Accurate semantic MT evaluation for any output language
Lo, C.-K. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4767 · Cited by 52 · see citing works
Ma, Q., Graham, Y., Wang, S., & Liu, Q. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4768 · Cited by 45 · see citing works
Rethinking the Agreement in Human Evaluation Tasks
Amidei, J., Piwek, P., & Willis, A. (2018) · Open Research Online (The Open University) · Cited by 40 · see citing works
Survey of the state of the art in natural language generation
GattAlbert, & KrahmerEmiel (2018) · doi:10.5555/3241691.3241693 · Cited by 40 · see citing works
Designing Precise and Robust Dialogue Response Evaluators
Zhao, T., Lala, D., & Kawahara, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.4 · Cited by 38 · see citing works
The price of debiasing automatic metrics in natural language evaluationPreprint
Chaganty, A. T., Mussmann, S., & Liang, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02202 · Cited by 36 · see citing works
What makes a good conversation? How controllable attributes affect human judgmentsPreprint
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08654 · Cited by 34 · see citing works
Evaluating Rewards for Question Generation Models
Hosking, T., & Riedel, S. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1237 · Cited by 34 · see citing works
Re-Evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
Sai, A. B., Gupta, M. D., Khapra, M. M., & Srinivasan, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016220 · Cited by 33 · see citing works
Self-citationNUBIA: NeUral Based Interchangeability Assessor for Text Generation
Kane, H., Kocyigit, M. Y., Abdalla, A., Ajanoh, P., & Coulibali, M. (2020) · arXiv · doi:10.48550/arxiv.2004.14667 · Cited by 32 · see citing works
EED: Extended Edit Distance Measure for Machine Translation
Stanchev, P., Wang, W., & Ney, H. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5359 · Cited by 31 · see citing works
Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation
Guo, Y., & Hu, J. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5357 · Cited by 30 · see citing works
XMEANT: Better semantic MT evaluation without reference translations
Lo, C.-K., Beloucif, M., Saers, M., et al. (2014) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/v1/p14-2124 · Cited by 30 · see citing works
Focused Evaluation for Image Description with Binary Forced-Choice Tasks
Hodosh, M., & Hockenmaier, J. (2016) · doi:10.18653/v1/w16-3203 · Cited by 28 · see citing works
AMR Similarity Metrics from Principles
Opitz, J., Pârcălăbescu, L., & Frank, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00329 · Cited by 27 · see citing works
ROUGE-C: A fully automated evaluation method for multi-document summarization
He, T., Chen, J., Ma, L., et al. (2008) · IEEE International Conference on Granular Computing (GrC) · doi:10.1109/grc.2008.4664680 · Cited by 25 · see citing works
Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments
Fomicheva, M., & Specia, L. (2019) · Computational Linguistics · doi:10.1162/coli_a_00356 · Cited by 24 · see citing works
Learning-based Composite Metrics for Improved Caption Evaluation
Sharif, N., White, L., Bennamoun, M., & Shah, S. A. A. (2018) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/p18-3003 · Cited by 23 · see citing works
Factorization of synchronous context-free grammars in linear time
Zhang, H., & Gildea, D. (2007) · doi:10.3115/1626281.1626285 · Cited by 22 · see citing works
NNEval: Neural Network Based Evaluation Metric for Image Captioning
Sharif, N., White, L., Bennamoun, M., & Shah, S. A. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_3 · Cited by 22 · see citing works
WMDO: Fluency-based Word Mover’s Distance for Machine Translation Evaluation
Chow, J., Specia, L., & Madhyastha, P. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5356 · Cited by 22 · see citing works
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR
Opitz, J., & Frank, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.129 · Cited by 22 · see citing works
Unsupervised vs. supervised weight estimation for semantic MT evaluation metrics
Lo, C.-K., & Wu, D. (2012) · Cited by 21 · see citing works
Machine Translation Evaluation with BERT RegressorPreprint
Shimanaka, H., Kajiwara, T., & Komachi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.12679 · Cited by 21 · see citing works
WebNLG Challenge: Human Evaluation Results
Shimorina, A., Gardent, C., Narayan, S., et al. (2018) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 16 · see citing works
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized EmbeddingsPreprint
Ghazarian, S., Wei, J. T.-Z., Galstyan, A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.10635 · Cited by 15 · see citing works
ITER: Improving Translation Edit Rate through Optimizable Edit Costs
Panja, J., & Naskar, S. K. (2018) · doi:10.18653/v1/w18-6455 · Cited by 13 · see citing works
Query-based summarization of discussion threads
Verberne, S., Krahmer, E., Wubben, S., & van den Bosch, A. P. J. (2019) · Natural Language Engineering · doi:10.1017/s1351324919000123 · Cited by 12 · see citing works
Linguistic Appropriateness and Pedagogic Usefulness of Reading Comprehension Questions.
Horbach, A., Aldabe, I., Bexte, M., de Lacalle, O. L., & Maritxalar, M. (2020) · Universitätsbibliographie, Universität Duisburg-Essen · Cited by 11 · see citing works
Speaker Sensitive Response Evaluation Model
Bak, J., & Oh, A. H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.568 · Cited by 11 · see citing works
Definitional and human constraints on structural annotation of English
Sampson, G. R., & Babarczy, A. (2008) · Natural Language Engineering · doi:10.1017/s1351324908004695 · Cited by 10 · see citing works
Meteor++: Incorporating Copy Knowledge into Machine Translation Evaluation
Guo, Y., Ruan, C., & Hu, J. (2018) · doi:10.18653/v1/w18-6454 · Cited by 10 · see citing works
Word Embedding-Based Automatic MT Evaluation Metric using Word Position Information
Echizen‐ya, H., Araki, K., & Hovy, E. H. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1186 · Cited by 9 · see citing works
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog EvaluationPreprint
Liang, W., Zou, J., & Zhou, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.10716 · Cited by 7 · see citing works
Evaluating Rewards for Question Generation ModelsPreprint
Hosking, T., & Riedel, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.11049 · Cited by 6 · see citing works
Parthasarathi, P., Pineau, J., & Chandar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10427 · Cited by 6 · see citing works
Designing Precise and Robust Dialogue Response EvaluatorsPreprint
Zhao, T., Lala, D., & Kawahara, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.04908 · Cited by 5 · see citing works
Speaker Sensitive Response Evaluation ModelPreprint
Bak, J., & Oh, A. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07015
Flagged first, then by citations
Research works printed but not linked: 10
- Anja Belz and Eric Kow. 2011. Discrete vs. Continuous Rating Scales for Language Evaluation in NLP. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA -Short Papers. The Association for Computer Linguistics, 230-235. https://www.aclweb.org/anthology/P11-2040/
- Anja Belz and Ehud Reiter. 2006. Comparing Automatic and Human Evaluation of NLG Systems. In 11th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Trento, Italy. https://www.aclweb.org/anthology/E06-1040
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Manuscript submitted to ACM
- Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the Role of Bleu in Machine Translation Research. In 11th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Trento, Italy. https://www.aclweb.org/anthology/E06-1032
- Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. 2009. Introduction to algorithms. MIT press.
- Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why We Need New Evaluation Metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, Martha
- Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, Satinder P. Singh and Shaul Markovitch (Eds.). AAAI Press, 3295-3301. http://aaai.org/ocs/index. php/AAAI/AAAI17/paper/view/14567
- E.J. Williams. 1959. Regression Analysis. Wiley. https://books.google.co.in/books?id=uWkNogEACAAJ
- Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 5754-5764. http://papers.nips.cc/paper/8812-xlnet-generalized-autoregressive-pretraining- for-language-understanding
- Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019,
Cited by
Sorted by citations
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · see citing works
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
“These metrics lack the capability to capture essential elements such as semantics, 29,46 context, 46,47 distant dependencies, 48,49 semantically-critical ordering change, 47 and human perspectives, particularly in real-world scenarios.”
BackgroundKuhn, L., Gal, Y., & Farquhar, S. (2023) · arXiv · doi:10.48550/arxiv.2302.09664 · Cited by 52 · see citing works
“Similarly, Sai et al. (2022) discuss how paraphrase detection can be used to evaluate NLG and other related methods might transfer to uncertainty estimation.”
BackgroundHuman Evaluation and Correlation with Automatic Metrics in Consultation Note Generation
Moramarco, F., Papadopoulos Korfiatis, A., Perera, M., Juric, D., Flann, J., Reiter, E., Belz, A., & Savkov, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.394 · Cited by 40 · see citing works
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
Gu, Z., Zhu, X., Ye, H., Zhang, L., Wang, J., Zhu, Y., Jiang, S., Xiong, Z., Li, Z., Wu, W., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29767 · Cited by 28 · see citing works
Croxford, E., Gao, Y., First, E., Pellegrino, N., Schnier, M., Caskey, J., Oguss, M., Wills, G., Chen, G., Dligach, D., et al. (2025) · doi:10.1101/2025.04.22.25326219 · Cited by 25 · see citing works
QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation
Fu, W., Wei, B., Hu, J., Cai, Z., & Liu, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.658 · Cited by 16 · see citing works
“Human evaluation is widely acknowledged as the gold standard for evaluating QG (Wang et al., 2022), with most automatic metrics striving to align their results with human evaluation results (Amidei et al., 2018; Sai et al., 2022).”
BackgroundYESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
D'Souza, J., Giglou, H. B., & Münch, Q. (2025) · arXiv · doi:10.48550/arxiv.2505.14279
“Based on a comprehensive review of NLG metrics (Sai et al., 2022), we applied eight diverse similarity metrics: four for verbatim matching, one edit-distance-based, and three embedding-based, e.g., Mover-Score (2019), BERTScore (2020).”
Chen, Y., Zhang, C., Luo, D., D'Haro, L. F., Tan, R. T., & Li, H. (2024) · arXiv · doi:10.48550/arxiv.2405.14646
- Authors
- Sai, A. B., Mohankumar, A. K., & Khapra, M. M.
- Affiliations
- Indian Institute of Technology Madras (IN)
- Venue
- ACM Computing Surveys
- Published
- 2022-01-18
- Volume
- 55
- Issue
- 2
- Pages
- 1-39
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3080855795
- Cited by
- 235
- References
- 189
- Citations by year
- 2026: 52 · 2025: 53 · 2024: 29 · 2023: 56 · 2022: 27 · 2021: 17 · 2020: 1
- FWCI
- 13.52
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Department of Computer Science and Engineering
- Robert Bosch Center for Data Science and Artificial Intelligence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- natural language generation, automatic evaluation metrics, evaluation taxonomy, human evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
