SummEval: Re-evaluating Summarization Evaluation
Open accessThe DOI matches its Crossref record; 99 of the 102 references it cites were found in a registry or an index.
Abstract
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SummEval (Fabbri et al., 2021) is a news domain dataset, where the task is abstractive summarization.The outputs are systemgenerated summaries of CNN/Daily Mail articles, with human ratings provided on coherence, consistency, fluency, and relevance.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Human evaluation has long been considered the gold standard for evaluating the quality of natural language generation (NLG) systems (Belz and Reiter, 2006;Lai and Tetreault, 2018;Fabbri et al., 2021).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indirectly, these metrics also evaluate composition, since it has been shown that ROUGE and BLEU correlate with human judgments on fluency of a text [67].”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“Data in AG-GREFACT comes from 9 factual consistency evaluation datasets on CNN or XSum, including widely used ones such as SummaC (Laban et al., 2022), FRANK (Pagnoni et al., 2021), and SummEval (Fabbri et al., 2021).”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Fabbri et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al.”
BackgroundMethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“ROUGE-WE (Ng and Abrecht, 2015) extends ROUGE by using soft lexical matching based on the cosine similarity of Word2Vec (Mikolov et al., 2013) embeddings.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“However, it has recently been shown that the mentioned datasets contain human judgments for model outputs scoring on a lower scale compared to current summarization systems putting into question the true performance of those metrics in the new setting (Peyrard, 2019).”
BackgroundInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani ∗ Equal contributions from authors et al., 2017) as well as the availability of largescale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et…”
BackgroundProceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Research works printed but not linked: 2
- Dang, 2009, Overview of the TAC 2009 summarization track
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
Cited by
Items 101–200 of 762
DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence
Zhao, W., Strube, M., & Eger, S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.278 · Cited by 81
Xiao, Z., Zhang, S., Lai, V. W.-M., & Liao, Q. V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.676 · Cited by 90
“Since Coherence and Relevance are distinct by definition (Fabbri et al., 2021), the conflated validity structure indicates potential issues in the expert rating process.”
BackgroundMethodInfluentialExploring Neural Models for Query-Focused Summarization
Vig, J., Fabbri, A. R., Kryściński, W., Wu, C.-S., & Liu, W. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.109 · Cited by 69
Self-citationSummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization
Laban, P., Kryściński, W., Agarwal, D., Fabbri, A. R., Xiong, C., Joty, S., & Wu, C.-S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.600 · Cited by 119
“The DialSummEval (Gao and Wan, 2022) benchmark is a summarization evaluation benchmark created following the format of SummEval (Fabbri et al., 2021) for the domain of dialogue summarization.”
BackgroundMethodInfluentialSelf-citationRevisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE?
Akter, M., Bansal, N., & Karmaker, S. K. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.122 · Cited by 64
“We further performed meta-evaluation of the Sem-nCG metric using data provided by (Fabbri et al., 2021) 6 .”
MethodText Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges
Becker, J., Wahle, J. P., Gipp, B., & Ruas, T. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.22258 · Cited by 43
BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language ModelsPreprint
Leong, W. Q., Ngui, J. G., Susanto, Y., Rengarajan, H., Sarveswaran, K., & Tjhi, W. C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.06085 · Cited by 68
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation
Moramarco, F., Papadopoulos Korfiatis, A., Perera, M., Juric, D., Flann, J., Reiter, E., Belz, A., & Savkov, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.394 · Cited by 40
Leveraging generative AI for clinical evidence synthesis needs to ensure trustworthiness
Zhang, G., Jin, Q., Jered McInerney, D., Chen, Y., Wang, F., Cole, C. L., Yang, Q., Wang, Y., Malin, B. A., Peleg, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104640 · Cited by 40
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40
Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment
Raina, V., Liusie, A., & Gales, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14016 · Cited by 40
“Smaller LLMs (FlanT5-xl, Llama2-7b and Mistral-7b) demonstrate reasonable performance on SummEval and TopicalChat, but larger models (GPT3.”
MethodInfluentialPan, X., Li, X., Li, Q., Hu, Z., & Bao, J. (2024) · Journal of Engineering Design · doi:10.1080/09544828.2023.2301230 · Cited by 50
“To this end, multiple studies indicate that obtaining high-quality design datasets for MMKG is essential (Fabbri et al. 2021).”
MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics
Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.528 · Cited by 22
“In a similar vein, SummEval is a recently released dataset that evaluates a number of evaluation metrics for summarization (Fabbri et al., 2020).”
BackgroundTransparent Human Evaluation for Image Captioning
Kasai, J., Sakaguchi, K., Dunagan, L., Morrison, J., Bras, R. J. L., Choi, Y., & Smith, N. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.254 · Cited by 65
Human-like Summarization Evaluation with ChatGPT
Gao, M., Ruan, J., Sun, R., Yin, X., Yang, S., & Wan, X. (2023) · arXiv · doi:10.48550/arxiv.2304.02554 · Cited by 40
“For SummEval, the four dimensions are relevance, faithfulness 5 , fluency, and coherence.”
MethodInfluentialPerturbation CheckLists for Evaluating NLG Evaluation Metrics
Sai, A. B., Dixit, T., Sheth, D. Y., Mohan, S., & Khapra, M. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.575 · Cited by 80
“For SMS (Clark et al., 2019), WMDo (Chow et al., 2019), and Mover-Score (Zhao et al., 2019), we use the implementation provided by Fabbri et al. (2020).”
MethodA Comprehensive Survey on Legal Summarization: Challenges and Future Directions
Akter, M., Çano, E., Weber, E., Dobler, D., & Habernal, I. (2025) · ACM Computing Surveys · doi:10.1145/3776586 · Cited by 47
“This issue is not unique to the legal domain; it has also been observed in the broader summarization community [41].”
CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization
Lin, H., Ma, L., Zhu, J., Xiang, L., Zhou, Y., Zhang, J., & Zong, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.365 · Cited by 59
“These three aspects are frequently used in the summarization community (Zhu et al., 2019; Fabbri et al., 2021), and we also refer to some researches on NLG evaluation (Howcroft et al.”
MethodOf Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story GenerationPreprint
Chhun, C., Colombo, P., Clavel, C., & Suchanek, F. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.11646 · Cited by 112
“, 2015) Novelty (Fabbri et al., 2021) Repetition (Fabbri et al.”
BackgroundMethodInfluentialStruc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?Preprint
Tang, X., Zong, Y., Phang, J., Zhao, Y., Zhou, W., Cohan, A., & Gerstein, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.08963 · Cited by 50
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics
Deutsch, D., Dror, R., & Roth, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.442 · Cited by 65
Detecting and Mitigating Hallucinations in Multilingual Summarisation
Qiu, Y., Ziser, Y., Korhonen, A., Ponti, E. M., & Cohen, S. B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.551 · Cited by 73
Personalized Impression Generation for PET Reports Using Large Language Models
Tie, X., Shin, M., Pirasteh, A., Ibrahim, N., Huemann, Z., Castellino, S. M., Kelly, K. M., Garrett, J., Hu, J., Cho, S. Y., et al. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-00985-3 · Cited by 46
“The evaluation codes are partially adapted from [49] and made available on GitHub: https:// github. com/ xtie97/ PET- Report-Summa rizat ion/ tree/ main/ evalu ation_ metri cs. [1] Large metabolically active anterior mediastinal mass with metabolically active left supraclavicular and left axillary…”
A Review of Faithfulness Metrics for Hallucination Assessment in Large Language Models
Malin, B., Kalganova, T., & Boulgouris, N. V. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3579203 · Cited by 47
“These classifications are typically grouped into entailment, contradiction or neutral, though other studies have desired other outputs such as with a Likert 1-5 scale [26,27].”
InfluentialEvaluation of Automatic Legal Text Summarization Techniques for Greek Case Law
Koniaris, M., Galanis, D., Giannini, E. N., & Tsanakas, P. D. (2023) · Information · doi:10.3390/info14040250 · Cited by 42
Learning to Revise References for Faithful Summarization
Adams, G., Shing, H.-C., Sun, Q., Winestock, C., McKeown, K. R., & Elhadad, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.296 · Cited by 49
Which is better? Exploring Prompting Strategy For LLM-based Metrics
Kim, J., Lee, S., Han, S. H., Park, S.-Y., Lee, J., Jeong, K., & Kang, P. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.14 · Cited by 47
Learning Opinion Summarizers by Selecting Informative Reviews
Bražinskas, A., Lapata, M., & Titov, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.743 · Cited by 51
“Generating input faithful summaries is crucial for practical applications, however, it remains an open problem in summarization (Maynez et al., 2020; Fabbri et al., 2020; Wang et al., 2020).”
BackgroundAssemble Foundation Models for Automatic Code Summarization
Gu, J., Salza, P., & Gall, H. C. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00112 · Cited by 55
Shi, Y., Ren, P., Wang, J., Han, B., ValizadehAslani, T., Agbavor, F., Zhang, Y., Hu, M., Zhao, L., & Liang, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104533 · Cited by 40
Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers
Subbiah, M., Zhang, S., Chilton, L. B., & McKeown, K. R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00702 · Cited by 53
“Work like Fabbri et al. (2021a) has shown the flaws in many of the traditional automatic metrics, prompting a move toward model-based reference-free methods and fine-grained span analysis.”
BackgroundMethodInfluentialTrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C., & Szpektor, I. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.127 · Cited by 33
Balachandran, V., Hajishirzi, H., Cohen, W., & Tsvetkov, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.667 · Cited by 30
Can Large Language Models Serve as Evaluators for Code Summarization?
Wu, Y., Wan, Y., Chu, Z., Zhao, W., Liu, Y., Zhang, H., Shi, X., Jin, H., & Yu, P. S. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3595283 · Cited by 42
“We evaluate generated code summaries across four key dimensions [42]: • Coherence (0-4).”
Utama, P., Bambrick, J., Moosavi, N., & Gurevych, I. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.199 · Cited by 24
Zhang, S., Wan, D., & Bansal, M. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.120 · Cited by 10
Salience Allocation as Guidance for Abstractive Summarization
Wang, F., Song, K., Zhang, H., Jin, L., Cho, S., Yao, W., Wang, X., Chen, M., & Yu, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.409 · Cited by 48
Kouris, P., Alexandridis, G., & Stafylopatis, A. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00950-5 · Cited by 38
“Finally, we note that the Blue metric is the primary evaluation measure for machine translation and this is also used for assessing the generated summaries in the field of automatic TS [86].”
MethodLENS: Layers of Evaluation of Hallucination in GenAI Systems
Gu, C., Zhang, W., Huang, Z., Kou, J., Liu, Z., Zhao, C., Liu, C., Zhang, L., Lin, W., Wang, Z., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189150 · Cited by 36
Investigating Entropy for Extractive Document Summarization
Khurana, A., & Bhatnagar, V. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115820 · Cited by 38
“…surge in study of evaluation metrics for automatic summaries is a clear and strong testimony to the considerable complexity of the task (Peyrard, 2019a,b; Ermakova et al., 2019; Bhandari et al., 2020a; Vasilyev & Bohannon, 2020; Fabbri et al., 2020; Huang et al., 2020; Bhandari et al., 2020b).”
BackgroundMethodA Survey of Deep Learning Methods on Semantic Similarity and Sentence Modeling
Zad, S., Heidari, M., Hajibabaee, P., & Malekzadeh, M. (2021) · IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) · doi:10.1109/iemcon53756.2021.9623078 · Cited by 41
King, D., Shen, Z., Subramani, N., Weld, D. S., Beltagy, I., & Downey, D. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.51 · Cited by 48
Conditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A. P., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00583 · Cited by 67
“We further validated our ANLI entailment scores against human judgements of faithfulness elicited as part of SummEval (Fabbri et al., 2021), a recently released dataset for assessing automated summarization metrics.”
MethodLongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.121 · Cited by 29
Ensemble methods for improving extractive summarization of legal case judgements
Deroy, A., Ghosh, K., & Ghosh, S. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09349-8 · Cited by 48
Adapting Pretrained Text-to-Text Models for Long Text Sequences
Xiong, W., Gupta, A., Toshniwal, S., Mehdad, Y., & Yih, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.370 · Cited by 33
Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End
Chen, Y., & Eger, S. (2023) · Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems · doi:10.18653/v1/2023.eval4nlp-1.6 · Cited by 24
On the Limitations of Reference-Free Evaluations of Generated Text
Deutsch, D., Dror, R., & Roth, D. (2022) · arXiv · doi:10.48550/arxiv.2210.12563 · Cited by 29
SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation
Su, W., Xie, A., Ai, Q., Long, J., Chen, X., Mao, J., Ye, Z., & Liu, Y. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808598 · Cited by 29
Zhang, F., Yu, X., Keung, J., Li, F., Xie, Z., Yang, Z., Ma, C., & Zhang, Z. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106922 · Cited by 32
Rehman, T., Sanyal, D. K., Chattopadhyay, S., Bhowmick, P. K., & Das, P. P. (2023) · IEEE Access · doi:10.1109/access.2023.3292300 · Cited by 31
“These are the standard metrics used to measure the performance of summarization models [42].”
MethodLeveraging Locality in Abstractive Text Summarization
Liu, Y., Ni, A., Nan, L., Deb, B., Zhu, C., Awadallah, A. H., & Radev, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.408 · Cited by 37
Self-citationÚbeda, P. L., Martín‐Noguerol, T., Díaz-Angulo, C., & Luna, A. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105443 · Cited by 30
CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization
Choubey, P. K., Fabbri, A. R., Vig, J., Wu, C.-S., Liu, W., & Rajani, N. F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.685 · Cited by 40
Self-citationSummary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
Laban, P., Fabbri, A. R., Xiong, C., & Wu, C.-S. (2024) · arXiv · doi:10.48550/arxiv.2407.01370 · Cited by 26
Self-citationHow well do you know your summarization datasets?
Tejaswin, P., Naik, D., & Liu, P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.303 · Cited by 39
Hierarchical Heterogeneous Graph Attention Network for Syntax-Aware Summarization
Song, Z., & King, I. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21385 · Cited by 30
“We further access the proposed model by eliciting human judgment (Fabbri et al. 2021).”
MethodAnalyzing and Evaluating Faithfulness in Dialogue Summarization
Wang, B., Zhang, C., Zhang, Y., Chen, Y., & Li, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.325 · Cited by 40
Gurusamy, B. M., Kumar, R. P., & Srinivasan, P. (2023) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v13i6.pp6663-6672 · Cited by 28
“The evaluation model for text summarization has its shortcoming in using neural networks [30].”
FFCI: A Framework for Interpretable Automatic Evaluation of Summarization
Koto, F., Baldwin, T., & Lau, J. H. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13167 · Cited by 42
PASS: Perturb-and-Select Summarizer for Product Reviews
Oved, N., & Levy, R. (2021) · doi:10.18653/v1/2021.acl-long.30 · Cited by 34
“We chose to use the experts’ annotations only, as they are considered to be more accurate and reliable for coherence and fluency (Fabbri et al., 2021).”
MethodInfluentialMultilingual Text Summarization for German Texts Using Transformer Models
Alcantara, T. H. M., Krütli, D., Ravada, R., & Hanne, T. (2023) · Information · doi:10.3390/info14060303 · Cited by 24
“An evaluation of different approaches for evaluating the quality of text summarization methods is provided by Fabbri et al. [29].”
Finding a Balanced Degree of Automation for Summary Evaluation
Zhang, S., & Bansal, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.531 · Cited by 15
“…most of them cannot reliably substitute human evaluation due to the unstable performance across datasets (Bhandari et al., 2020), weak to moderate correlations with human judgment (Fabbri et al., 2021), or more indication of topic similarity than information overlap (Deutsch and Roth, 2021).”
BackgroundConditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2022) · arXiv · doi:10.48550/arxiv.2207.00397 · Cited by 24
“We further validated our ANLI entailment scores against human judgements of faithfulness elicited as part of SummEval (Fabbri et al., 2021), a recently released dataset for assessing automated summarization metrics.”
MethodEvaluation metrics on text summarization: comprehensive survey
Davoodijam, E., & Meybodi, M. A. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02217-0 · Cited by 31
“SummEval [85] and REAL-Summ [87] datasets are used to analyze summary quality, both of which were collected from CNN/DailyMail.”
A scalable framework for evaluating health language models
Mallinar, N., Heydari, A. A., Liu, X., Faranesh, A. Z., Winslow, B. D., Hammerquist, N., Graef, B., Speed, C. A., Malhotra, M., Patel, S., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02492-x · Cited by 35
“Further details of the rubric question classi fi cation process using automatic evaluation methods are provided in the Automatic Evaluation Setup section of Methods.”
On Improving Summarization Factual Consistency from Natural Language Feedback
Liu, Y., Deb, B., Teruel, M., Halfaker, A., Radev, D., & Awadallah, A. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.844 · Cited by 56
Self-citationSEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation
Clark, E., Rijhwani, S., Gehrmann, S., Maynez, J., Aharoni, R., Nikolaev, V. A., Sellam, T., Siddhant, A., Das, D., & Parikh, A. P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.584 · Cited by 60
Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
Chrysostomou, G., Zhao, Z., Williams, M., & Αλέτρας, Ν. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00695 · Cited by 45
“To mitigate the effect of prompt variability, we summarize each document using three distinct prompt templates (Table 1).”
MethodYu, H., Zhao, W. K., & Zhao, Q. (2022) · Advanced Engineering Informatics · doi:10.1016/j.aei.2022.101649 · Cited by 25
Poddar, S., Samad, A. M., Mukherjee, R., Ganguly, N., & Ghosh, S. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531745 · Cited by 36
“To this end, we tried three different quality metrics for summaries [8]: • Consistency –”
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations
Wang, L. L., Otmakhova, Y., DeYoung, J., Truong, T. H., Kuehl, B., Bransom, E., & Wallace, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.549 · Cited by 47
SNaC: Coherence Error Detection for Narrative Summarization
Goyal, T., Li, J. J., & Durrett, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.29 · Cited by 45
ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-LearningPreprint
Hasanbeig, H., Sharma, H., Betthauser, L., Frujeri, F. V., & Momennejad, I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.13701 · Cited by 32
“We use the SummEval data [13], which is based on the CNN/DailyMail dataset [26].”
MethodRe-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics
Deutsch, D., Dror, R., & Roth, D. (2022) · arXiv · doi:10.48550/arxiv.2204.10216 · Cited by 22
“Although the above definition of the system-level correlation has been used by recent meta-evaluation studies of metrics (Bhandari et al., 2020; Fabbri et al., 2021; Deutsch et al., 2021b), there is a disconnect between how the automatic metrics are evaluated and how they are used in practice.”
BackgroundMethodInfluentialReducing hallucinations of large language models via hierarchical semantic piece
Liu, Y., Yang, Q., Tang, J.-W., Guo, T., Chen, W., Pan, L., Sai, X., Gao, X., Li, Z., Liu, J., et al. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01833-9 · Cited by 26
“• SummEval [52]: For the mitigation task, we merge all summaries in the SummEval dataset according to the reference text.”
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand
Kasai, J., Sakaguchi, K., Le Bras, R., Dunagan, L., Morrison, J., Fabbri, A., Choi, Y., & Smith, N. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.259 · Cited by 17
Self-citationImproving Factual Consistency in Summarization with Compression-Based Post-Editing
Fabbri, A. R., Choubey, P. K., Vig, J., Wu, C.-S., & Xiong, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.623 · Cited by 31
Self-citationLarge Scale Multi-Lingual Multi-Modal Summarization Dataset
Verma, Y., Jangra, A., Verma, R., & Saha, S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.263 · Cited by 36
INACIA: Integrating Large Language Models in Brazilian Audit Courts: Opportunities and Challenges
Pereira, J., Assumpcao, A., Trecenti, J. A. Z., Airosa, L., Lente, C., Cléto, J., Dobins, G., Nogueira, R. F., Mitchell, L., & Lotufo, R. (2025) · Digital Government Research and Practice · doi:10.1145/3652951 · Cited by 32
“We used the SummEval Dataset [6] as a benchmark.”
Play the Shannon Game with Language Models: A Human-Free Approach to Summary Evaluation
Egan, N., Vasilyev, O., & Bohannon, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21304 · Cited by 31
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries
Tang, X., Fabbri, A. R., Li, H., Mao, Z., Adams, G., Wang, B., Çelikyılmaz, A., Mehdad, Y., & Radev, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.417 · Cited by 29
Self-citationElangovan, A., Xu, L., Ko, J., Elyasi, M., Liu, L., Bodapati, S., & Roth, D. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.03775 · Cited by 25
“SummEval (Fabbri et al., 2021) is human evaluation of summary quality using four criteria – fluency, coherence, consistency, and relevance, with Likert scores assigned from 1 to 5.”
Opportunities and Risks of LLMs for Scalable Deliberation with Polis
Small, C. T., Vendrov, I., Durmus, E., Homaei, H., Barry, E., Cornebise, J., Suzman, T., Ganguli, D., & Megill, C. (2023) · arXiv · doi:10.48550/arxiv.2306.11932 · Cited by 20
“Prior work has proposed automated metrics to detect such errors (Maynez et al., 2020; Durmus et al., 2020; Kryscinski et al., 2020; Goyal and Durrett, 2021); however, they do not always correlate well with human judgements (Fabbri et al., 2021; Durmus et al., 2022).”
BackgroundChu, S. Y., Kim, J. W., & Yi, M. Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713181 · Cited by 29
“LLM-based text evaluation has emerged to address the shortcomings of traditional text evaluation methods, such as BLUE [60] or ROUGE-L [47], which highly depend on reference text [24, 29, 30].”
InfluentialSummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization
Vig, J., Kryściński, W., Goel, K., & Rajani, N. F. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.18 · Cited by 24
“Other summaries might contain significant hallucinations, including words that are not found in the source document (Kry´sci´nski et al., 2020; Maynez et al., 2020).”
BackgroundSelf-citationDialSummEval: Revisiting Summarization Evaluation for Dialogues
Gao, M., & Wan, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.418 · Cited by 55
“Similarly, Fabbri et al. (2021b) used CNN/DailyMail dataset (Hermann et al., 2015) and the output of some models for human evaluation covering four facets of relevance, consistency, fluency, and coherence, and then re-evaluated the metrics in 14 categories.”
BackgroundMethodResultInfluentialToward Human-Like Evaluation for Natural Language Generation with Error Analysis
Lu, Q., Ding, L., Xie, L., Zhang, K., Wong, D. F., & Tao, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.324 · Cited by 36
How Far are We from Robust Long Abstractive Summarization?
Koh, H. Y., Ju, J., Zhang, H., Liu, M. Z., & Pan, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.172 · Cited by 47
Improving Factuality of Abstractive Summarization via Contrastive Reward Learning
Chern, I.-C., Wang, Z., Das, S. K., Sharma, B., Liu, P., & Neubig, G. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.6 · Cited by 26
Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model
Jia, Q., Ren, S., Liu, Y., & Zhu, K. Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.679 · Cited by 36
Zero-shot Bilingual App Reviews Mining with Large Language Models
Wei, J., Courbis, A.-L., Lambolais, T., Xu, B., Bernard, P. L., & Dray, G. (2023) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai59109.2023.00135 · Cited by 26
“2) Evaluation Metrics: As outlined in Fabbri et al. [33], human evaluators rate the generated summaries based on four dimensions: relevance (the degree to which crucial information from the source has been included), consistency (how well the summary aligns with the factual details of the source),…”
BackgroundStruc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data?
Tang, X., Zong, Y., Phang, J., Zhao, Y., Zhou, W., Cohan, A., & Gerstein, M. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-short.2 · Cited by 51
A Survey on Medical Document SummarizationPreprint
Jain, R., Jangra, A., Saha, S., & Jatowt, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.01669 · Cited by 28
“(5) Noisy Reference Summaries: Dependency on a good quality reference summary is one of the major limitations of metrics like ROUGE or BLEU [37, 70] as reference summary itself can be of below-par quality.”
BackgroundMethodInfluentialGLIDER: Grading LLM Interactions and Decisions using Explainable RankingPreprint
Deshpande, D. G., Ravi, S. S., CH-Wang, S., Mielczarek, B., Kannappan, A., & Qian, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.14140 · Cited by 23
Transformer-Based Abstractive Summarization of Legal Texts in Low-Resource Languages
Masih, S., Hassan, M., Fahad, L. G., & Hassan, B. (2025) · Electronics · doi:10.3390/electronics14122320 · Cited by 20
Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline
Kapania, S., Ballard, S. A., Kessler, A., & Vaughan, J. W. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732005 · Cited by 29
“…for evaluation, auxiliary models were alsoincreasingly usedtoevaluate the qualityofthe primarymodel’s outputsby producingratings orscores (as in LLM-as-a-judge [110]), a task that was previously typically carried out by human annotators or more conventional automated metrics (e.g., [22, 73]).”
A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
Adams, G., Zucker, J. E., & Elhadad, N. (2023) · PubMed · doi:10.48550/arxiv.2303.03948 · Cited by 38
“We cannot adapt all metrics, especially metrics (Sellam et al., 2020; Ribeiro et al., 2022) which directly learn from news summary annotation benchmarks (Wang et al., 2020; Pagnoni et al., 2021; Fabbri et al., 2021a; Laban et al., 2022).”
BackgroundInfluentialLoFT: Enhancing Faithfulness and Diversity for Table-to-Text Generation via Logic Form Control
Zhao, Y., Qi, Z., Nan, L., Flores, L. J., & Radev, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.40 · Cited by 34
Self-citationThe Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics
Leiter, C., Opitz, J., Deutsch, D., Gao, Y., Dror, R., & Eger, S. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.10 · Cited by 40
Metadata
- Authors
- Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D.
- Affiliations
- Yale University (US)
- Salesforce (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 391-409
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3045321166
- Cited by
- 587
- References
- 102
- Citations by year
- 2026: 109 · 2025: 93 · 2024: 74 · 2023: 166 · 2022: 84 · 2021: 57
- FWCI
- 33.37
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Social Sciences › Linguistics and Language › Text Readability and Simplification
- Keywords
- text summarization, summarization evaluation, automatic evaluation metrics, crowdsourced annotation, evaluation toolkit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.