SummEval: Re-evaluating Summarization Evaluation
Open accessThe DOI matches its Crossref record; 98 of the 102 references it cites were found in a registry or an index.
Abstract
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SummEval (Fabbri et al., 2021) is a news domain dataset, where the task is abstractive summarization.The outputs are systemgenerated summaries of CNN/Daily Mail articles, with human ratings provided on coherence, consistency, fluency, and relevance.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Human evaluation has long been considered the gold standard for evaluating the quality of natural language generation (NLG) systems (Belz and Reiter, 2006;Lai and Tetreault, 2018;Fabbri et al., 2021).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indirectly, these metrics also evaluate composition, since it has been shown that ROUGE and BLEU correlate with human judgments on fluency of a text [67].”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“Data in AG-GREFACT comes from 9 factual consistency evaluation datasets on CNN or XSum, including widely used ones such as SummaC (Laban et al., 2022), FRANK (Pagnoni et al., 2021), and SummEval (Fabbri et al., 2021).”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Fabbri et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al.”
BackgroundMethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“ROUGE-WE (Ng and Abrecht, 2015) extends ROUGE by using soft lexical matching based on the cosine similarity of Word2Vec (Mikolov et al., 2013) embeddings.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“However, it has recently been shown that the mentioned datasets contain human judgments for model outputs scoring on a lower scale compared to current summarization systems putting into question the true performance of those metrics in the new setting (Peyrard, 2019).”
BackgroundInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani ∗ Equal contributions from authors et al., 2017) as well as the availability of largescale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et…”
BackgroundProceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 2×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al., 2019b; Dong et al., 2019; Ziegler et al., 2019; Raffel et al., 2019; Lewis et al., 2019).”
BackgroundMethod(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 4×
“vated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al., 2019b; Dong et al., 2019; Ziegler et al., 2019; Raffel et al., 2019; Lewis et al., 2019).”
BackgroundMethodInfluentialCIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,672
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416 · cited 3×
“…model which views relevance labels of sentences in a document as binarylatent variables M4 - REFRESH (Narayan et al., 2018) propose using REINFORCE (Williams, 1992) to extract summaries, approximating the search space during training by limiting to combinations of individually high-scoring…”
MethodInfluentialBERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
From Word Embeddings To Document Distances
Kusner, M. J., Sun, Y., Kolkin, N., & Weinberger, K. Q. (2015) · PolyPublie (École Polytechnique de Montréal) · Cited by 2,700
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“A standard dataset for training summarization models is the CNN/DailyMail corpus (Hermann et al., 2015), originally a question answering task, which was repurposed for summarization by Nallapati et al.”
BackgroundMethodAbstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Vinyals, O., Fortunato, M., & Jaitly, N. (2015) · neural information processing systems · Cited by 2,043
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,635
Text Summarization with Pretrained Encoders
Liu, Y., & Lapata, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1387 · Cited by 1,553
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225 · cited 3×
“M21 - UniLM (Dong et al., 2019) introduce a model pretrained on three language modeling tasks: unidirectional, bidirectional, and sequenceto-sequence prediction.”
BackgroundMethodchrF: character n-gram F-score for automatic MT evaluation
Popović, M. (2015) · Proceedings of the Tenth Workshop on Statistical Machine Translation · doi:10.18653/v1/w15-3049 · Cited by 1,220 · cited 3×
“CHRF (Popović, 2015) calculates character-based n-gram overlap between model outputs and reference documents.”
BackgroundPEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1912.08777 · Cited by 1,206 · cited 3×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al., 2019b; Dong et al., 2019; Ziegler et al., 2019; Raffel et al., 2019; Lewis et al., 2019).”
BackgroundMethodOn Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182
Self-citationMariani, J. (2018) · HAL · Cited by 1,151
Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments
Lavie, A., & Agarwal, A. (2007) · Proceedings of the Second Workshop on Statistical Machine Translation - StatMT '07 · doi:10.3115/1626355.1626389 · Cited by 1,047 · cited 2×
“METEOR (Lavie and Agarwal, 2007) computes an alignment between candidate and reference sentences by mapping unigrams in the generated summary to 0 or 1 unigrams in the reference based on stemming, synonyms and paraphrastic matches.”
MethodComputing Krippendorff's Alpha-Reliability
Krippendorff, K. (2011) · ScholarlyCommons (University of Pennsylvania) · Cited by 809
Evaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669
Self-citationBottom-Up Abstractive Summarization
Gehrmann, S., Deng, Y., & Rush, A. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1443 · Cited by 667 · cited 1×
“M10 - Bottom-Up (Gehrmann et al., 2018) introduce a bottom-up approach whereby a content selection model restricts the copy attention distribution of a pretrained Pointer Generator Network during inference.”
BackgroundFast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
Chen, Y.-C., & Bansal, M. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1063 · Cited by 595 · cited 1×
“M9 - Fast-abs-rl (Chen and Bansal, 2018) propose a model which first extracts salient sentences with a Pointer Network and rewrites these sentences with a Pointer Generator Network.”
BackgroundNew York Times Annotated Corpus
Sandhaus, E., & Linguistic Data Consortium (2012) · TIB Data Manager · doi:10.57702/qbrk4fuh · Cited by 586
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1053 · Cited by 490 · cited 1×
“…when used outside of its original setting (Liu and Liu, 2008; Cohan and Goharian, 2016) as well as the introduction of many variations on ROUGE (Zhou et al., 2006; Ng and Abrecht, 2015; Ganesan, 2015; ShafieiBavani et al., 2018) and other text generation metrics (Peyrard, 2019; Zhao et al.,…”
BackgroundNewsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies
Grusky, M., Naaman, M., & Artzi, Y. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1065 · Cited by 479 · cited 1×
“, 2017) as well as the availability of largescale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et al., 2018; Narayan et al., 2018).”
BackgroundFine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413 · cited 3×
“M20 - GPT-2 (Ziegler et al., 2019) build off of GPT-2 (Radford et al.”
BackgroundZhang, X., Wei, F., & Zhou, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1499 · Cited by 377 · cited 2×
“…variations on ROUGE (Zhou et al., 2006; Ng and Abrecht, 2015; Ganesan, 2015; ShafieiBavani et al., 2018) and other text generation metrics (Peyrard, 2019; Zhao et al., 2019; Zhang* et al., 2020; Scialom et al., 2019; Clark et al., 2019), ROUGE has remained the default automatic evaluation metric.”
BackgroundMethodAbstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
Nallapati, R. M., Zhou, B., dos Santos, C. N., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.06023 · Cited by 371
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Wang, A., Cho, K., & Lewis, M. (2020) · arXiv · doi:10.48550/arxiv.2004.04228 · Cited by 356
Neural Document Summarization by Jointly Learning to Score and Select Sentences
Zhou, Q., Yang, N., Wei, F., Huang, S., Zhou, M., & Zhao, T. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1061 · Cited by 344 · cited 2×
“M1 - NEUSUM (Zhou et al., 2018) jointly scores and selects sentences by first building a hierarchical representation of a document and considering the partially outputted summary at each time step.”
BackgroundNeural Text Summarization: A Critical Evaluation
Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1051 · Cited by 335
Self-citationPEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. X. (2020) · International Conference on Machine Learning · Cited by 325
Durmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306
A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss
Hsu, W.-T., Lin, C.-K., Lee, M.-Y., Min, K., Tang, J., & Sun, M. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1013 · Cited by 252 · cited 1×
“M12 - Unified-ext-abs (Hsu et al., 2018) propose to use the probability output of an extractive model as sentence-level attention to modify word-level attention scores of an abstractive model, introducing an inconsistency loss to encourage consistency between these two levels of attention.”
MethodDang, H. T. (2005) · Cited by 219
Automatically Assessing Machine Summary Content Without a Gold Standard
Louis, A., & Nenkova, A. (2013) · Computational Linguistics · doi:10.1162/coli_a_00123 · Cited by 177
Overview of the TAC 2008 update summarization task
Dang, H. T., & Owczarzak, K. (2008) · Cited by 171
Neural Latent Extractive Document Summarization
Zhang, X., Lapata, M., Wei, F., & Zhou, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1088 · Cited by 163 · cited 2×
“M3 - LATENT (Zhang et al., 2018b) propose a latent variable extractive model which views relevance labels of sentences in a document as binarylatent variables M4 - REFRESH (Narayan et al., 2018) propose using REINFORCE (Williams, 1992) to extract summaries, approximating the search space during training by limiting to…”
BackgroundImproving Abstraction in Text Summarization
Kryściński, W., Paulus, R., Xiong, C., & Socher, R. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1207 · Cited by 163
Self-citationMulti-Reward Reinforced Summarization with Saliency and Entailment
Pasunuru, R., & Bansal, M. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2102 · Cited by 162 · cited 1×
“M13 - ROUGESal (Pasunuru and Bansal, 2018) propose a keyphrase-based salience reward as well as an entailment-based reward in addition to using a ROUGE-based reward in a REINFORCE setting, optimizing rewards simultaneously in alternate mini-batches.”
MethodBanditSum: Extractive Summarization as a Contextual Bandit
Dong, Y., Shen, Y., Crawford, E., van Hoof, H., & Cheung, J. C. K. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1409 · Cited by 162 · cited 1×
“M2 - BanditSum (Dong et al., 2018) treats extractive summarization as a contextual bandit problem where the document is the context and the sequence of sentences to include in the summary is the action.”
BackgroundContent Selection in Deep Learning Models of Summarization
Kedzie, C., McKeown, K., & Daumé III, H. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1208 · Cited by 152 · cited 1×
“In (Kedzie et al., 2018) the authors examined the influence of various model components on the quality of content selection.”
BackgroundNeural Extractive Text Summarization with Syntactic Compression
Xu, J., & Durrett, G. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1324 · Cited by 147
Better Summarization Evaluation with Word Embeddings for ROUGE
Ng, J.-P., & Abrecht, V. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1222 · Cited by 146
Callison-Burch, C., & Association for Computing Machinery-Digital Library (2010) · Cited by 144
Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE
Graham, Y. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1013 · Cited by 143 · cited 2×
“In a similar line of work, Graham (2015) conducted a large-scale study of the effectiveness of different ROUGE metric variants and compared the it against the BLEU metric on the DUC datasets.”
ResultSentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts
Clark, E., Celikyilmaz, A., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1264 · Cited by 135 · cited 3×
“Sentence Mover’s Similarity (SMS) (Clark et al., 2019) extends Word Mover’s Distance to view documents as a bag of sentence embeddings as well as a variation which represents documents as both a bag of sentences and a bag of words.”
BackgroundInfluentialLearning to Extract Coherent Summary via Deep Reinforcement Learning
Wu, Y., & Hu, B. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11987 · Cited by 133
Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation
Guo, H., Pasunuru, R., & Bansal, M. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1064 · Cited by 129 · cited 1×
“M14 - Multi-task (Ent + QG ) (Guo et al., 2018) propose question generation and entailment generation as auxiliary tasks in a multi-task framework along with a corresponding multi-task architecture.”
MethodAnswers Unite! Unsupervised Metrics for Reinforced Summarization Models
Scialom, T., Lamprier, S., Piwowarski, B., & Staiano, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1320 · Cited by 121
Non-expert evaluation of summarization systems is risky
Gillick, D., & Liu, Y. (2010) · Cited by 112
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization
Gao, Y., Zhao, W., & Eger, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.124 · Cited by 112
ParaEval: using paraphrases to evaluate summaries automatically
Zhou, L., Lin, C.-Y., Munteanu, D. S., & Hovy, E. (2006) · Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics - · doi:10.3115/1220835.1220892 · Cited by 106 · cited 1×
“Additionally, despite problems associated with ROUGE when used outside of its original setting (Liu and Liu, 2008; Cohan and Goharian, 2016) as well as the introduction of many variations on ROUGE (Zhou et al., 2006; Ng and Abrecht, 2015; Ganesan, 2015; ShafieiBavani et al., 2018) and other text generation metrics (Pe…”
MethodROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks
Ganesan, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.01937 · Cited by 105 · cited 2×
“…(Liu and Liu, 2008; Cohan and Goharian, 2016) as well as the introduction of many variations on ROUGE (Zhou et al., 2006; Ng and Abrecht, 2015; Ganesan, 2015; ShafieiBavani et al., 2018) and other text generation metrics (Peyrard, 2019; Zhao et al., 2019; Zhang* et al., 2020; Scialom et al.,…”
BackgroundMethodRanking Sentences for Extractive Summarization with Reinforcement Learning
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1158 · Cited by 99 · cited 3×
“M4 - REFRESH (Narayan et al., 2018) propose using REINFORCE (Williams, 1992) to extract summaries, approximating the search space dur-”
BackgroundMethodThe price of debiasing automatic metrics in natural language evalaution
Chaganty, A., Mussmann, S., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1060 · Cited by 96 · cited 2×
“In (Chaganty et al., 2018) the authors in-”
BackgroundLooking for a Few Good Metrics: Automatic Summarization Evaluation — How Many Samples Are Enough?
Lin, C.-Y. (2004) · NTCIR · Cited by 93
Correlation between ROUGE and human evaluation of extractive meeting summaries
Liu, F., & Liu, Y. (2008) · Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies Short Papers - HLT '08 · doi:10.3115/1557690.1557747 · Cited by 81 · cited 2×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al., 2019b; Dong et al., 2019; Ziegler et al., 2019; Raffel et al., 2019; Lewis et al., 2019).”
BackgroundMethodAn Entity-Driven Framework for Abstractive Summarization
Sharma, E., Huang, L., Hu, Z., & Wang, L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1323 · Cited by 78
Fill in the BLANC: Human-free quality estimation of document summaries
Vasilyev, O., Dharnidharka, V., & Bohannon, J. (2020) · Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems · doi:10.18653/v1/2020.eval4nlp-1.2 · Cited by 77
Learning to Score System Summaries for Better Content Selection Evaluation.
Peyrard, M., Botschen, T., & Gurevych, I. (2017) · Proceedings of the Workshop on New Frontiers in Summarization · doi:10.18653/v1/w17-4510 · Cited by 75 · cited 2×
“CHRF, and METEOR tended to favor extractive models, assigning highest scores to their outputs.”
BackgroundBetter Rewards Yield Better Summaries: Learning to Summarise Without References
Böhm, F., Gao, Y., Meyer, C. M., Shapira, O., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1307 · Cited by 67 · cited 1×
“M18 - NeuralTD (Böhm et al., 2019) learn a reward function from 2,500 human judgments which is used in a reinforcement learning setting.”
A Decade of Automatic Content Evaluation of News Summaries: Reassessing the State of the Art
Rankel, P. A., Conroy, J. M., Dang, H. T., & Nenkova, A. (2013) · Cited by 62
Studying Summarization Evaluation Metrics in the Appropriate Scoring Range
Peyrard, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1502 · Cited by 50
Revisiting Summarization Evaluation for Scientific Articles
Cohan, A., & Goharian, N. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3uxkgihvmbtw · Cited by 49
A Repository of Corpora for Summarization
Dernoncourt, F., Ghassemi, M. M., & Chang, W. H. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2xvghosmyh3a · Cited by 46 · cited 1×
“In relation to datasets, Dernoncourt et al. (2018) presented a detailed taxonomy of existing summarization datasets.”
BackgroundProceedings of the Human Language Technology Conference of the NAACL, Main Conference
Murray, G., Renals, S. J., Carletta, J., & Moore, J. D. (2006) · The Association for Computational Linguistics · Cited by 40
HighRES: Highlight-based Reference-less Evaluation of Summarization
Hardy, Narayan, S., & Vlachos, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1330 · Cited by 35
Closed-Book Training to Improve Summarization Encoder Memory
Jiang, Y., & Bansal, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1440 · Cited by 34 · cited 1×
“M15 - Closed book decoder (Jiang and Bansal, 2018) build upon a Pointer Generator Network by adding an additional copy-less and attention-less decoder during training time to force the encoder to be more selective in encoding salient content.”
SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics
Deutsch, D., & Roth, D. (2020) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/2020.nlposs-1.17 · Cited by 31
Learning to Extract Coherent Summary via Deep Reinforcement LearningPreprint
Wu, Y., & Hu, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.07036 · Cited by 30
A Graph-theoretic Summary Evaluation for ROUGE
ShafieiBavani, E., Ebrahimi, M., Wong, R. K., & Chen, F. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1085 · Cited by 29
STRASS: A Light and Effective Method for Extractive Summarization Based on Sentence Embeddings
Bouscarrat, L., Bonnefoy, A., Peel, T., & Pereira, C. (2019) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/p19-2034 · Cited by 25 · cited 1×
“M7 - STRASS (Bouscarrat et al., 2019) extracts a summary by selecting the sentences with the closest embeddings to the document embedding, learning a transformation to maximize the similarity between the summary and the ground truth reference.”
BackgroundProceedings of ACL-08: HLT, Short Papers
Georgila, K., Wolters, M. K., & Moore, J. D. (2008) · The Association for Computational Linguistics · Cited by 24
Neural Extractive Text Summarization with Syntactic CompressionPreprint
Xu, J., & Durrett, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00863 · Cited by 23
On the Abstractiveness of Neural Document Summarization
Zhang, F., Yao, J.-G., & Yan, R. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1089 · Cited by 18
Assessing the Effect of Inconsistent Assessors on Summarization Evaluation
Owczarzak, K., Rankel, P. A., & Dang, H. T. (2013) · Cited by 6
Research works printed but not linked: 2
- Dang, 2009, Overview of the TAC 2009 summarization track
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
Cited by
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“Unlike previous studies that predominantly focused on specific tasks such as summarization (Fabbri et al., 2021; Tang et al., 2022)or particular domains such as world knowledge (Miao et al., 2023), this benchmark assesses factuality across five domains: world knowledge, science and technology,…”
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
“SummEval (Fabbri et al., 2021) is a benchmark that compares different evaluation methods for summarization.”
BackgroundG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
“SummEval (Fabbri et al., 2021) is a benchmark that compares different evaluation methods for summarization.”
BackgroundCan Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2305.03514 · Cited by 839
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
Metadata
- Authors
- Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D.
- Affiliations
- Yale University (US)
- Salesforce (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 391-409
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3045321166
- Cited by
- 587
- References
- 102
- Citations by year
- 2026: 109 · 2025: 93 · 2024: 74 · 2023: 166 · 2022: 84 · 2021: 57
- FWCI
- 33.37
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Social Sciences › Linguistics and Language › Text Readability and Simplification
- Keywords
- text summarization, summarization evaluation, automatic evaluation metrics, crowdsourced annotation, evaluation toolkit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.