SummEval: Re-evaluating Summarization Evaluation
Open accessThe DOI matches its Crossref record; 99 of the 102 references it cites were found in a registry or an index.
Abstract
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SummEval (Fabbri et al., 2021) is a news domain dataset, where the task is abstractive summarization.The outputs are systemgenerated summaries of CNN/Daily Mail articles, with human ratings provided on coherence, consistency, fluency, and relevance.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Human evaluation has long been considered the gold standard for evaluating the quality of natural language generation (NLG) systems (Belz and Reiter, 2006;Lai and Tetreault, 2018;Fabbri et al., 2021).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indirectly, these metrics also evaluate composition, since it has been shown that ROUGE and BLEU correlate with human judgments on fluency of a text [67].”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“Data in AG-GREFACT comes from 9 factual consistency evaluation datasets on CNN or XSum, including widely used ones such as SummaC (Laban et al., 2022), FRANK (Pagnoni et al., 2021), and SummEval (Fabbri et al., 2021).”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Fabbri et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al.”
BackgroundMethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“ROUGE-WE (Ng and Abrecht, 2015) extends ROUGE by using soft lexical matching based on the cosine similarity of Word2Vec (Mikolov et al., 2013) embeddings.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“However, it has recently been shown that the mentioned datasets contain human judgments for model outputs scoring on a lower scale compared to current summarization systems putting into question the true performance of those metrics in the new setting (Peyrard, 2019).”
BackgroundInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani ∗ Equal contributions from authors et al., 2017) as well as the availability of largescale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et…”
BackgroundProceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Research works printed but not linked: 2
- Dang, 2009, Overview of the TAC 2009 summarization track
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
Cited by
Items 201–300 of 762
The Glass Ceiling of Automatic Evaluation in Natural Language Generation
Colombo, P., Peyrard, M., Noiry, N., West, R., & Piantanida, P. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.16 · Cited by 28
Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation
Zhang, Y., Su, Y., Liu, Y., Wang, X., Burgess, J., Sui, E., Wang, C., Aklilu, J. G., Lozano, A., Wei, A., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02754 · Cited by 45
SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics
Deutsch, D., & Roth, D. (2020) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/2020.nlposs-1.17 · Cited by 31
A Multi-Document Coverage Reward for RELAXed Multi-Document Summarization
Parnell, J., Unanue, I. J., & Piccardi, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.351 · Cited by 26
Reference-free Summarization Evaluation via Semantic Correlation and Compression Ratio
Liu, Y., Jia, Q. F., & Zhu, K. Q. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.153 · Cited by 29
“We evaluate the variants of the SOTA summariza-2112 tion evaluation metric (Shan.) and our proposed reference-free summarization evaluation metric (SDC*) on CNNDM. Semantic distribution correlation.”
MethodInfluentialAnswer Summarization for Technical Queries: Benchmark and New Approach
Yang, C., Xu, B., Thung, F., Shi, Y., Zhang, T., Yang, Z., Zhou, X., Shi, J., He, J., Han, D., et al. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3560421 · Cited by 19
“., experts and crowd-sourced workers) [10].”
BackgroundInfluentialInvestigating Answerability of LLMs for Long-Form Question AnsweringPreprint
Bhat, M. M., Meng, R., Ye, L., Zhou, Y., & Yavuz, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.08210 · Cited by 23
“Since LLMs are expected to generate free form answers for our setting, we take inspiration from prior works on long-form text generation metrics (Fabbri et al., 2020) and adopt them in our evaluation for coherency, consistency, accuracy, and relevance.”
BackgroundMethodInfluentialBooookScore: A systematic exploration of book-length summarization in the era of LLMs
Chang, Y., Lo, K., Goyal, T., & Iyyer, M. (2023) · arXiv · doi:10.48550/arxiv.2310.00785 · Cited by 18
“Just two years ago, automatically-generated summaries were riddled with artifacts such as grammar errors, repetition, and hallucination (Zhao et al., 2020; Fabbri et al., 2020; Goyal & Durrett, 2021).”
BackgroundSemantic Similarity Based Evaluation for Abstractive News Summarization
Fikri, F. B., Oflazer, K., & Yanikoglu, B. A. (2021) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2021.gem-1.3 · Cited by 29
“Following the work of Fabbri et al. (2021), we asked native Turkish annotators to rate each predicted summary in terms of relevance (selection of important content from the source), consistency (the factual alignment between the summary and the summarized source) and fluency (the quality of…”
MethodFaMeSumm: Investigating and Improving Faithfulness of Medical Summarization
Zhang, N., Zhang, Y., Guo, W., Mitra, P., & Zhang, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.673 · Cited by 32
Automation of Text Summarization Using Hugging Face NLP
Asmitha, M., Danda, A., Bysani, H., Singh, R. P., & Kanchan, S. (2024) · International Conference for Emerging Technology (INCET) · doi:10.1109/incet61516.2024.10593316 · Cited by 18
“The results obtained from the study by Fabbri [3] include the re-evaluation of 14 automatic evaluation metrics, benchmarking 23 recent summarization models, and combining together the largest collection of summaries generated by models trained on the dataset.”
MethodAbstractive text summarization: A comprehensive survey of techniques, systems, and challenges
Almohaimeed, N., & Azmi, A. M. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100762 · Cited by 19
Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters
Schuster, T., Chen, S., Buthpitiya, S., Fabrikant, A., & Metzler, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.28 · Cited by 17
Warm-Starting for Improving the Novelty of Abstractive Summarization
Alomari, A., Al‐Shamayleh, A. S., Idris, N. B., Sabri, A. Q. M., Alsmadi, I. M., & Omary, D. (2023) · IEEE Access · doi:10.1109/access.2023.3322226 · Cited by 18
“Sentence-BERT [39] was recently introduced as a means for facilitating trustworthy sentence-based semantic textual similarity.”
BackgroundMethod“Why is this misleading?”: Detecting News Headline Hallucinations with Explanations
Shen, J., Liu, J., Finnie, D., Rahmati, N., Bendersky, M., & Najork, M. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583375 · Cited by 32
“MNBM [38] 2,500 FRANK [43] 671 QAGS [56] 239 SummEval [19] 1,600 FEVER [55] 18,209 Vitamin-C [49] 63,054”
MethodTowards Argument-Aware Abstractive Summarization of Long Legal Opinions with Summary Reranking
Elaraby, M., Zhong, Y., & Litman, D. J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.481 · Cited by 30
Action-Item-Driven Summarization of Long Meeting Transcripts
Golia, L., & Kalita, J. K. (2023) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3639233.3639253 · Cited by 25
“However, ROUGE scores have many flaws since they focus solely on the lexical overlap between the machine-generated summaries and the human reference summaries ratherthan their semantic similarity [5].”
BackgroundEnd to End Urdu Abstractive Text Summarization With Dataset and Improvement in Evaluation Metric
Raza, H., & Shahzad, W. (2024) · IEEE Access · doi:10.1109/access.2024.3377463 · Cited by 22
ROUGE-SS: A New ROUGE Variant for the Evaluation of Text Summarization
Kumar, S., Solanki, A., & Jhanjhi, N. Z. (2024) · Recent Advances in Computer Science and Communications · doi:10.2174/0126662558304595240528111535 · Cited by 20
GupShup: Summarizing Open-Domain Code-Switched Conversations
Mehnaz, L., Mahata, D., Gosangi, R., Gunturi, U. S., Jain, R., Gupta, G., Kumar, A., Lee, I. G., Acharya, A., & Shah, R. R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.499 · Cited by 27
“Following (Fabbri et al., 2020), we had three annotators rate the summaries for the following four metrics 6 : consistency, coherence, fluency, and relevance on a Likert scale of 1 to 5.”
BackgroundMethodEfficient Few-Shot Fine-Tuning for Opinion Summarization
Bražinskas, A., Nallapati, R. M., Bansal, M., & Dreyer, M. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.113 · Cited by 30
The statistical advantage of automatic NLG metrics at the system level
Wei, J., & Jia, R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.533 · Cited by 16
“On the human side, we may focus on creating larger effect sizes or reducing noise by adopting new annotation schemes (L¨aubli et al., 2018; Shapira et al., 2019) or employing professional linguists (Fabbri et al., 2020; Toral et al., 2018).”
MethodSpurious Correlations in Reference-Free Evaluation of Text Generation
Durmus, E., Ladhak, F., & Hashimoto, T. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.102 · Cited by 16
Automated Text Generation and Summarization for Academic Writing
Benites, F., Benites, A. D., & Anson, C. M. (2023) · Springer eBooks · doi:10.1007/978-3-031-36033-6_18 · Cited by 21
Briman, M. K. H., & Yildiz, B. (2024) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213024500179 · Cited by 18
What’s New? Summarizing Contributions in Scientific Literature
Hayashi, H., Kryściński, W., McCann, B., Rajani, N. F., & Xiong, C. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.72 · Cited by 22
Self-citationTowards Better Evaluation of Instruction-Following: A Case-Study in Summarization
Skopek, O., Aralikatte, R., Gooding, S., & Cărbune, V. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.16 · Cited by 28
Abstractive Summarizers Become Emotional on News Summarization
Ahuir, V., González, J. Á., Hurtado, L.-F., & Segarra, E. (2024) · Applied Sciences · doi:10.3390/app14020713 · Cited by 17
“This study has been performed on three state-of-the-art transformer-based systems [12]: B ART [13], P EGASUS [14], and T5 [15].”
MethodCan Jev Judge Radiology Reports? Evaluating a System One Model for Clinical FactualityPreprint
Huang, J., Yang, H., Ma, X., Liang, X., Cai, K., Ma, J., Chen, S., Sun, Y., & Tan, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.27607 · Cited by 17
“In summarization, SummEval includes output length among its simple comparison features [9].”
Multi-Dimensional Evaluation of Text Summarization with In-Context Learning
Jain, S., Keshava, V., Mysore Sathyendra, S., Fernandes, P., Liu, P., Neubig, G., & Zhou, C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.537 · Cited by 15
Fast and Accurate Factual Inconsistency Detection Over Long Documents
Lattimer, B. M., Chen, P. M., Zhang, X., & Yang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.105 · Cited by 39
Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluation
Zhang, R., Ouni, J., & Eger, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00519 · Cited by 22
“We utilize the same evaluation criteria as SummEval (Fabbri et al. 2021).”
BackgroundMethodInfluentialMDEval: Evaluating and Enhancing Markdown Awareness in Large Language Models
Chen, Z., Liu, Y., Shi, L. Q., Wang, Z.-J., Chen, X., Zhao, Y., & Ren, F. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714674 · Cited by 21
“Unlike traditional tasks such as text summarization where human ratings are available [12], Markdown Awareness presents considerable challenges in evaluation without human preference datasets.”
Oliva, G., Rajbahadur, G. K., Bhatia, A., Zhang, H.-X., Chen, Y., Chen, Z., Leung, A., Lin, D.-Y., Chen, B.-Y., & Hassan, A. E. (2025) · International Conference on Automated Software Engineering · doi:10.1109/ase63991.2025.00192 · Cited by 20
“For all IRA comparisons reported in this section and throughout the study, we use Krippendorff’s α [11], which satisfies three essential criteria: (1) it supports any number of annotators, (2) it handles nominal data, and (3) it is widely trusted in empirical research [19, 20].”
NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist
Ni’mah, I., Fang, M., Menkovski, V., & Pechenizkiy, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.69 · Cited by 30
IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages
Ananya Sai, B., Dixit, T., Nagarajan, V., Kunchukuttan, A., Kumar, P., Khapra, M. M., & Dabre, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.795 · Cited by 52
Why Do You Feel This Way? Summarizing Triggers of Emotions in Social Media Posts
Zhan, H., Sosea, T., Caragea, C., & Li, J. J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.642 · Cited by 25
HydraSum: Disentangling Style Features in Text Summarization with Multi-Decoder Models
Goyal, T., Rajani, N. F., Liu, W., & Kryściński, W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.30 · Cited by 28
“Recent work (Fabbri et al., 2021) has shown that ROUGE is insufficient to evaluate summary quality and recommends human evaluation.”
Self-citationX-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization
Chaudhury, S., Swaminathan, S., Gunasekara, C., Crouse, M., Ravishankar, S., Kimura, D., Murugesan, K., Astudillo, R. F., Naseem, T., Kapanipathi, P., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.478 · Cited by 20
“Sum-mEval (Fabbri et al., 2021) also study re-evaluation of factuality scores for abstractive summarization.”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive SummarizationPreprint
Cao, S., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09209 · Cited by 21
“Alongside improving factuality, we have also witnessed growing interests in automated factu-ality evaluation, since popular word-matching-based metrics, e.g., ROUGE, correlate poorly with human-rated factual consistency levels (Gabriel et al., 2021; Fabbri et al., 2021).”
BackgroundA Survey of Automatic Hallucination Evaluation on Natural Language GenerationPreprint
Qi, S., Gui, L., He, Y., & Yuan, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.12041 · Cited by 25
“We also present Task-specific(§2.1) Summarization Maynez et al. (2020) , CoGenSumm (Falke et al., 2019) , QAGS (Wang et al., 2020) , FRANK (Pagnoni et al., 2021) , FactEval Wang et al. (2022) , RefMatters Gao et al. (2023) , Poly-tope (Huang et al., 2020) , Falsesum (Utama et al., 2022) , FaithBench (Bao et al., 2024)…”
InfluentialManakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.4 · Cited by 14
Reliability of Human Evaluation for Text Summarization: Lessons Learned and Challenges Ahead
Iskender, N., Polzehl, T., & Möller, S. (2021) · Cited by 58
Xia, Y., Xie, Z., Liu, P., Lu, K., Liu, Y., Wang, W., & Ji, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728875 · Cited by 24
“Consequently, conventional quantitative metrics like n-grams or BLEU prove inadequate[31, 52].”
Iskender, N., Polzehl, T., & Möller, S. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.16 · Cited by 24
“Especially, languages other than English can benefit from our results, since they lack easy-to-use automatic evaluation metrics in the form of simplified toolkits, and a well-executed evaluation can accelerate the research on automatic summarization (Fabbri et al., 2020).”
BackgroundMethodResultA New Approach to Overgenerating and Scoring Abstractive Summaries
Song, K., Wang, B., Feng, Z., & Liu, F. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.110 · Cited by 16
MLASK: Multimodal Summarization of Video-based News Articles
Krubiński, M., & Pecina, P. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.67 · Cited by 27
“Fabbri et al. (2021) benchmarked over 20 recent summarization models and concluded that the abstractive summaries produced by pre-trained generative languages models fine-tuned on summarization datasets (Zhang et al., 2020; Lewis et al., 2020) consistently performed best with regards to both…”
BackgroundTowards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation
Liu, Y., Fabbri, A. R., Zhao, Y., Liu, P., Joty, S., Wu, C.-S., Xiong, C., & Radev, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1018 · Cited by 41
Self-citationKartha, R. S., Agal, S., Odedra, N. D., Nanda, C. S. K., Rao, V. S., Kuthe, A. M., & Taloba, A. I. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.01505124 · Cited by 13
“Additionally, extractive and abstractive findings are derived from the summary findings Fabbri et al. [10].”
Zeyad, A. M. A., & Biradar, A. (2024) · International Conference on Advancements in Power, Communication and Intelligent Systems (APCI) · doi:10.1109/apci61480.2024.10616418 · Cited by 17
“These metrics play an essential role in model development and refinement, ensuring new models align closely with human standards of summarization quality and coherence [4], [5].”
BackgroundWang, S., Zhao, F., Bu, D., Lu, Y., Gong, M., Liu, H., Yang, Z., Zeng, X., Yuan, Z., Wan, B., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64142-2 · Cited by 23
“Frameworks like LINS, which are based on retrieval-augmented generation 14 , address this limitation by enabling real-time retrieval of online information to supplement the knowledge referenced during LLM responses.”
InfluentialGoldilocks: Consistent Crowdsourced Scalar Annotations with Relative Uncertainty
Chen, Q. Z., Weld, D. S., & Zhang, A. X. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476076 · Cited by 23
“, a news article) or even multiple documents [25].”
BackgroundImproving Automatic Summarization for Browsing Longform Spoken Dialog
Li, D., Chen, T., Zadikian, A., Tung, A., & Chilton, L. B. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581339 · Cited by 18
“6The intuition is that if a generated summary more closely follows the ordering of words in the reference, then the generated summary is more fuent [15].”
BackgroundInfluentialDecision-Focused Summarization
Hsu, C.-C., & Tan, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.10 · Cited by 18
“Evaluation of summarization is very challenging, partly because the goal of summarization is usually vague (Nenkova and Passonneau, 2004; Fabbri et al., 2021).”
BackgroundSummarizing long scientific documents through hierarchical structure extraction
Sharma, G., Sharma, D., & Sasikumar, M. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100080 · Cited by 15
Carammia, M., Iacus, S. M., & Porro, G. (2026) · Social Science Computer Review · doi:10.1177/08944393261485320 · Cited by 19
How to Select Datapoints for Efficient Human Evaluation of NLG Models?
Zouhar, V., Cui, P., & Sachan, M. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.60 · Cited by 33
Few-shot fine-tuning SOTA summarization models for medical dialogues
Navarro, D. F., Dras, M., & Berkovsky, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/2022.naacl-srw.32 · Cited by 18
Generating User-Engaging News Headlines
Cai, P., Song, K., Cho, S., Wang, H., Wang, X., Yu, H., Liu, F., & Dong, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.183 · Cited by 33
“We also experiment with a variety of automatic metrics to assess headline quality in terms of informativeness, relevance to the source article, and content accuracy (Kryscinski et al., 2020; Fabbri et al., 2021).”
MethodEvaluating and Improving Factuality in Multimodal Abstractive Summarization
Wan, D., & Bansal, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.654 · Cited by 24
MeaningBERT: assessing meaning preservation between sentences
Beauchemin, D., Saggion, H., & Khoury, R. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1223924 · Cited by 36
“The approach is built upon the SumEval (Fabbri et al., 2021) idea that uses an LLM to create a set of questions using a source sentence as a reference.”
MethodTeacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models
Liusie, A., Fathullah, Y., & Gales, M. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.81 · Cited by 22
Proceedings of the 4th New Frontiers in Summarization Workshop
(2023) · doi:10.18653/v1/2023.newsum-1 · Cited by 12
LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores
Liu, Y., Moosavi, N., & Lin, C. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.753 · Cited by 12
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
Haldar, R., & Hockenmaier, J. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1361 · Cited by 11
“However, even for humans this is far lower We saw the LLM judges struggle particularly with one of the datasets within this benchmark, Sum-mEval (Fabbri et al., 2021).”
InfluentialHalluMeasure: Fine-grained Hallucination Measurement Using Chain-of-Thought Reasoning
Akbar, S. A., Hossain, M. M., Wood, T., Chin, S.-C., Salinas, E. M., Alvarez, V., & Cornejo, E. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.837 · Cited by 44
“Our results demonstrate Hal-luMeasure’s superior performance over baseline models, with at least a 10-point F1 score improvement on TechNewsSumm and a 3-point AUC ROC increase on SummEval.”
InfluentialSpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
Zhuge, X., Shen, X., Wang, Z., Dang, F., Li, D., Hao, T., Ma, Q., Miao, X., Han, Y., & Yang, Z. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661278 · Cited by 13
“As shown in Table II: HumanEval [34] with 164 programming problems; C-Eval [35], a comprehensive Chinese suite with 13,948 questions; SummEval [36] with 100 CNN/DailyMail news articles; and SAMSum [37] with 16k messenger-like conversations and summaries.”
Localizing Factual Inconsistencies in Attributable Text Generation
Cattan, A., Roit, P., Zhang, S., Wan, D., Aharoni, R., Szpektor, I., Bansal, M., & Dagan, I. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.598 · Cited by 16
Cryptoblend: An AI-Powered Tool for Aggregation and Summarization of Cryptocurrency News
Pozzi, A., Barbierato, E., & Toti, D. (2023) · Informatics · doi:10.3390/informatics10010005 · Cited by 11
SummScore: A Comprehensive Evaluation Metric for Summary Quality Based on Cross-Encoder
Lin, W., Li, S., Zhang, C., Ji, B., Yu, J., Ma, J., & Yi, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25198-6_6 · Cited by 12
Summarization of Elicitation Conversations to Locate Requirements-Relevant Information
Spijkman, T., Bondt, X. D., Dalpiaz, F., & Brinkkemper, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-29786-1_9 · Cited by 13
“3.2), NLP summarization task metrics could be applied, e.g., coherence, consistency, percentage of text shown [13].”
MethodInfluentialHuman-in-the-loop Abstractive Dialogue Summarization
Chen, J., Dodda, M., & Yang, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.584 · Cited by 23
Łajewska, W., & Balog, K. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615132 · Cited by 14
“However, some dimensions are not reliably covered by currently available automatic metrics and require manual evaluation (e.g., coherence and relevance) [12], while others (e.g., completeness) can be evaluated automatically, provided that more fine-grained annotations are available.”
BackgroundFace4Rag: Factual Consistency Evaluation for Retrieval Augmented Generation in Chinese
Xu, Y., Cai, T., Jiang, J., & Song, X. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671656 · Cited by 23
“RAGAS[13] RefChecker[18] FRANK[34] SummEval[14] 𝑄 2 [17] DialFact[16] VitaminC[37] FACTSCORE(GPT-4) 70. fail to detect logical fallacy since some logical connections may be discarded during the decomposition.”
BackgroundMethodEvaluating the Efficacy of Summarization Evaluation across Languages
Koto, F., Lau, J. H., & Baldwin, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.71 · Cited by 11
PDSum: Prototype-driven Continuous Summarization of Evolving Multi-document Sets Stream
Yoon, S., Chan, H. P., & Han, J. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583371 · Cited by 15
WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning
Wu, W., Li, W., Xiao, X., Liu, J., Li, S., & Lyu, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.18 · Cited by 20
Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
Chehbouni, K., Haddou, M., Cheung, J. C., & Farnadi, G. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3029 · Cited by 10
Clinical Text Generation: Are We There Yet?
Hiebel, N., Ferret, O., Fort, K., & Névéol, A. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095202 · Cited by 12
“Well-established metrics have been re-evaluated several times on recent datasets and they present poor correlation with human judgments (103, 104, 105).”
Yuan, Z., Liu, K., Li, S., Peng, R., Leybourne, D. J., Musa, N., & Yang, P. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3586374 · Cited by 18
“The entities of long textual descriptions, such as pest descriptions and management practice, utilize BERTScore [31] to measure the textual similarity between extracted values and ground-truth values.”
On the Trade-off between Redundancy and Cohesiveness in Extractive Summarization
Cardenas, R., Gallé, M., & Cohen, S. B. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15191 · Cited by 12
Adams, G., Nguyen, B. H., Smith, J. A., Xia, Y., Xie, S., Ostropolets, A., Deb, B., Chen, Y.-J., Naumann, T., & Elhadad, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.587 · Cited by 18
Steen, J., & Markert, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.160 · Cited by 15
Towards Understanding the Robustness of LLM-based Evaluations under PerturbationsPreprint
Chaudhary, M., Gupta, H. K., Bhat, S., & Varma, V. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.09269 · Cited by 21
Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form GenerationPreprint
Li, Z., Chang, Y., Zhou, Y., Wu, X., Liang, Z., Sung, Y. Y., & Boyd-Graber, J. L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.15068 · Cited by 13
“Unlike short-form tasks with clear correctness signals, long-form outputs, like summaries, dialogues, or open-ended answers, lack binary ground truths and require assessing coherence, factuality, structure, and helpfulness (Chiang et al., 2024; Fabbri et al., 2021; Li et al., 2025a).”
RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs
Wu, J., Ning, L., Liu, L., Lee, H., Wu, N., Wang, C., Prakash, S., O’Banion, S., Green, B., & Xie, J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34738 · Cited by 23
Liu, Z., Kou, J., Zhang, W., Gu, C., Fang, X., Huang, Z., Hao, Y., Li, H., Lu, X., Yin, A., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189137 · Cited by 12
DNACSE: Enhancing Genomic LLMs with Contrastive Learning for DNA Barcode Identification
Wang, J., Wang, B., Zhou, S., Cao, B., Li, W., & Zheng, P. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c02747 · Cited by 13
Lee, H., Yoo, K. M., Park, J., Lee, H., & Jung, K. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.76 · Cited by 12
Slobodkin, A., Roit, P., Hirsch, E., Ernst, O., & Dagan, I. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.385 · Cited by 13
Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications
Faysse, M., Viaud, G., Hudelot, C., & Colombo, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.559 · Cited by 20
Meeting the challenge: A benchmark corpus for automated Urdu meeting summarization
Sadia, B., Adeeba, F., Shams, S., & Javed, K. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103734 · Cited by 9
Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks
Himmi, A., Irurozki, E., Noiry, N., Clémençon, S., & Colombo, P. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.688 · Cited by 15
Learning Evaluation Models From Large Language Models for Sequence Generation
Wang, C., Zhou, H., Chang, K., Liu, T., Zhang, C., Yang, M., Du, Q., Xiao, T., Zhang, Y., & Zhu, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3587460 · Cited by 12
“We computed the correlation scores of four single-aspect summarization evaluation models with human judgments on the SummEval benchmark [69], respectively.”
Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling
Jwalapuram, P., Joty, S., & Lin, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.418 · Cited by 19
SummerTime: Text Summarization Toolkit for Non-experts
Ni, A., Azerbayev, Z., Mutuma, M., Feng, T., Zhang, Y., Tao, Y., Awadallah, A. H., & Radev, D. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.37 · Cited by 13
“To evaluate the performance of each supported model on certain dataset, SummerTime integrates with SummEval (Fabbri et al., 2020) and provides the following evaluation metrics for the users to understand model performance: ROUGE (Lin, 2004) is a recall-oriented method based on overlapping n-grams,…”
MethodInfluentialSelf-citationQuestioning the Validity of Summarization Datasets and Improving Their Factual Consistency
Guo, Y., Clavel, C., Eddine, M. K., & Vazirgiannis, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.386 · Cited by 14
Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarization
Cheang, C., Chan, H. P., Wong, D. F., Liu, X., Li, Z., Sun, Y., Liu, S., & Chao, L. S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1007 · Cited by 22
“Some of these datasets (Kryscinski et al., 2020; Wang et al., 2020; Fabbri et al., 2021) annotate an overall faithfulness label for each summary.”
BackgroundMehnaz, L., Mahata, D., Gosangi, R., Gunturi, U. S., Jain, R., Gupta, G., Kumar, A., Lee, I. G., Acharya, A., & Shah, R. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08578 · Cited by 13
“Following (Fabbri et al., 2020), we considered 4 metrics: consistency , coherence , fluency , and relevance .”
MethodFaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs
Bao, F. S., Li, M., Qu, R., Luo, G., Wan, E., Tang, Y., Fan, W., Tamber, M. S., Kazi, S., Sourabh, V., et al. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2025.naacl-short.38 · Cited by 8
Metadata
- Authors
- Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D.
- Affiliations
- Yale University (US)
- Salesforce (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 391-409
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3045321166
- Cited by
- 587
- References
- 102
- Citations by year
- 2026: 109 · 2025: 93 · 2024: 74 · 2023: 166 · 2022: 84 · 2021: 57
- FWCI
- 33.37
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Social Sciences › Linguistics and Language › Text Readability and Simplification
- Keywords
- text summarization, summarization evaluation, automatic evaluation metrics, crowdsourced annotation, evaluation toolkit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.