SummEval: Re-evaluating Summarization Evaluation
Open accessThe DOI matches its Crossref record; 99 of the 102 references it cites were found in a registry or an index.
Abstract
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SummEval (Fabbri et al., 2021) is a news domain dataset, where the task is abstractive summarization.The outputs are systemgenerated summaries of CNN/Daily Mail articles, with human ratings provided on coherence, consistency, fluency, and relevance.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Human evaluation has long been considered the gold standard for evaluating the quality of natural language generation (NLG) systems (Belz and Reiter, 2006;Lai and Tetreault, 2018;Fabbri et al., 2021).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indirectly, these metrics also evaluate composition, since it has been shown that ROUGE and BLEU correlate with human judgments on fluency of a text [67].”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“Data in AG-GREFACT comes from 9 factual consistency evaluation datasets on CNN or XSum, including widely used ones such as SummaC (Laban et al., 2022), FRANK (Pagnoni et al., 2021), and SummEval (Fabbri et al., 2021).”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Fabbri et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al.”
BackgroundMethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“ROUGE-WE (Ng and Abrecht, 2015) extends ROUGE by using soft lexical matching based on the cosine similarity of Word2Vec (Mikolov et al., 2013) embeddings.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“However, it has recently been shown that the mentioned datasets contain human judgments for model outputs scoring on a lower scale compared to current summarization systems putting into question the true performance of those metrics in the new setting (Peyrard, 2019).”
BackgroundInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani ∗ Equal contributions from authors et al., 2017) as well as the availability of largescale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et…”
BackgroundProceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Research works printed but not linked: 2
- Dang, 2009, Overview of the TAC 2009 summarization track
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
Cited by
Items 401–500 of 762
Wang, S., Xu, R., Liu, Y., Zhu, C., & Zeng, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.479 · Cited by 8
“Moreover, from Table 4, we can see that our method is quite good at evaluating the factuality and achieves the best performance over all baselines on SummEval FAC, Rank19, and Q-XSUM.”
InfluentialKEMoS: A knowledge-enhanced multi-modal summarizing framework for Chinese online meetings
Qi, P., Sun, Y., Yao, M., & Tao, D. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106417 · Cited by 4
Okadome, Y., Yuguchi, A., Fukui, R., & Matsumoto, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72356-8_3 · Cited by 5
Intelligent Text Processing: A Review of Automated Summarization Methods
Sorokina, S. (2024) · Virtual Communication and Social Networks · doi:10.21603/2782-4799-2024-3-3-203-222 · Cited by 5
Unified Multi-Scenario Summarization Evaluation and Explanation
Shang, S., Yao, Z., Fu, H., Tao, C., Chen, X., Wang, F., Wang, Y., Ren, Z., & Gao, S. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3509715 · Cited by 5
“Then the trained evaluators are tested on the meta-evaluation benchmark SummEval [43] to measure the rank correlation coefficient between the evaluation model and human judgment.”
Kavitha, J., Roselinkiruba, R., Vasumathy, M., Mohan, K. R., & Rani, P. A. J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128351 · Cited by 4
TSTR: Too Short to Represent, Summarize with Details! Intro-Guided Extended Summary Generation
Sotudeh, S., & Goharian, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.25 · Cited by 7
Best-k Search Algorithm for Neural Text Generation
Xu, J., Xiong, C., Savarese, S., & Zhou, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.692 · Cited by 8
Self-citationEvaluating Patient-Oriented Echocardiogram Reports Augmented by Artificial Intelligence
Martin, J. A., Hill, T., Sarić, M., Vainrib, A., Bamira, D. G., Bernard, S., Ro, R., Zhang, H., Austrian, J. S., Aphinyanaphongs, Y., et al. (2024) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2024.05.021 · Cited by 4
Bao, X., Zhu, X., Yang, D., Lou, H., Wang, R., Wu, Y., Li, W., Xia, Y., Zeng, L., Pan, Y., et al. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-06871-y · Cited by 4
“…illusory or harmful responses to user questions in certain special circumstances, and there are no automated indicators for assessing harmfulness [31], we use a 5-point Likert scale (1 = Strongly Agree (Extremely harmful), 2 = Agree, 3 = Neutral, 4 = Disagree, 5 = Strongly Disagree) to manually…”
Kumar, S., Ghosal, T., & Ekbal, A. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00348-3 · Cited by 5
“We use the summeval tool [13] to evaluate the BERTScore.”
MethodTwo Phase Ensemble Learning Based Extractive Summarization for Short Documents
Mishra, G., Sethi, N., & Agilandeeswari, L. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-27524-1_14 · Cited by 4
DATScore: Evaluating Translation with Data Augmented Translations
Eddine, M. K., Shang, G., & Vazirgiannis, M. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.69 · Cited by 5
de Andrade, C. M. V., França, C., Belém, F. M., Jallais, G. F., Ganem, M., Texeira, G. D. S., Laender, A. H. F., & Gonçalves, M. A. (2023) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2023.232532 · Cited by 4
“Por fim, utilizamos o coeficiente alfa de Krippendorff, que mede o nível de concor-dância entre os avaliadores, sendo amplamente utilizado quando a quantidade de avalia-dores é maior que dois [Akter & Wamba 2016, Fabbri et al. 2021].”
BackgroundBetter than Random: Reliable NLG Human Evaluation with Constrained Active Sampling
Ruan, J., Pu, X., Gao, M., Wan, X., & Zhu, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29857 · Cited by 14
CASESUMM: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court OpinionsPreprint
Heddaya, M., MacMillan, K., Malani, A., Mei, H., & Tan, C. (2025) · Qeios · doi:10.32388/ej9r29 · Cited by 4
Confidence, Not Perplexity: A Better Metric for the Creative Era of LLMsPreprint
Parupudi, V. S. R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.08596 · Cited by 4
“…perplexity can operate without a ground truth, but it has a fundamental flaw: it rewards models for making safe, high-frequency word choices, which leads to dull, repetitive "neural text degeneration" (Holtzman et al., 2020) and correlates poorly with human quality judgments (Fabbri et al., 2021).”
Towards Flexible Evaluation for Generative Visual Question Answering
Ji, H., Si, Q., Lin, Z., & Wang, W. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681400 · Cited by 3
“To evaluate the inter-annotator agreement, following previous works [11, 19, 23], we apply Krippendorff’s alpha [29] and obtain a result of 0.713.”
InfluentialCourtEval: A Courtroom-Based Multi-Agent Evaluation Framework
Kumar, S., Nargund, A. A., & Sridhar, V. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1327 · Cited by 12
Guidance in Radiology Report Summarization: An Empirical Evaluation and Error Analysis
Trienes, J., Youssef, P., Schlötterer, J., & Seifert, C. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.13 · Cited by 4
Exploring the Trade-Off within Visual Information for MultiModal Sentence Summarization
Yuan, M., Cui, S., Zhang, X., Wang, S., Xu, H., & Liu, T. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657753 · Cited by 6
“Following Fabbri et al. [16], we compute the Krippendorff’s alpha [25] and achieve 0.52, 0.49, and 0.54 on consistency, fluency, and relevance, respectively, reflecting a reasonable inter-annotator agreement of the human evaluation.”
InfluentialCoUDA: Coherence Evaluation via Unified Data Augmentation
Zhu, D., Wu, W., Song, Y., Zhu, F., Cao, Z., & Li, S. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.55 · Cited by 5
Transformer-Based Sequence Modeling Short Answer Assessment Framework
Sharmila, P., Anbananthen, K. S. M., Chelliah, D., Parthasarathy, S., Balasubramaniam, B., & Lurudusamy, S. N. (2024) · HighTech and Innovation Journal · doi:10.28991/hij-2024-05-03-06 · Cited by 3
“The data was assessed using three metrics, specifically BLEU introduced by Fabbri et al. [33], which measures the similarity in terms of precision for n-grams, and ROUGE by Barbella & Tortora [34], which captures different aspects of text quality including overlap and longer contiguous sequences.”
MethodBookWorm: A Dataset for Character Description and Analysis
Papoudakis, A., Lapata, M., & Keller, F. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.258 · Cited by 14
Bae, J., Jeon, H., & Seo, J. (2025) · Visualization · doi:10.1109/vis60296.2025.00014 · Cited by 4
ESTIME: Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings
Vasilyev, O., & Bohannon, J. (2021) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2021.eval4nlp-1.10 · Cited by 9
Q&A Generation for Flashcards Within a Transformer-Based Framework
Thabet, B., Zanichelli, N., & Zanichelli, F. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-29800-4_59 · Cited by 4
Leveraging and Evaluating Automatic Code Summarization for JPA Program Comprehension
Mayer, R., Moser, M., & Geist, V. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00088 · Cited by 4
“Accordingly, concerns are raised about their applicability for code summarization [7], [8] since the metrics score the similarity to a reference, but give no hint on how well a summary matches the method’s source code [9].”
BackgroundResultEchoes from Alexandria: A Large Resource for Multilingual Book Summarization
Scirè, A., Conia, S., Ciciliano, S., & Navigli, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.54 · Cited by 8
BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics
Ma, L., Cao, S., Logan, R. L., Lu, D., Ran, S., Zhang, K., Tetreault, J., & Jaimes, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.716 · Cited by 11
Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets
Laban, P., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.135 · Cited by 5
Self-citationEvaluating Factual Consistency of Texts with Semantic Role Labeling
Fan, J., Aumiller, D., & Gertz, M. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.9 · Cited by 16
Re-Examining Summarization Evaluation across Multiple Quality Criteria
Ernst, O., Shapira, O., Dagan, I., & Levy, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.924 · Cited by 9
“Surprisingly, a closer examination of metric performances in SummEval reveals that many of the high correlations are unlikely to imply high fitness of the best performing metrics and the respective QC.”
BackgroundMethodInfluentialDon’t Add, don’t Miss: Effective Content Preserving Generation from Pre-Selected Text Spans
Slobodkin, A., Caciularu, A., Hirsch, E., & Dagan, I. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.852 · Cited by 7
Answer Relevance Score (ARS): Novel Evaluation Metric for Question Answering System
Gupta, T., & Kumar, E. (2023) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit60255.2023.10466080 · Cited by 3
“By calculating Pearson, Kendall, and Spearman correlations between reference answers and system-generated summaries using BERTScore, ROUGE-1, ROUGE-2, QAEval-EM [9], QAEval-F1, and QAEval-BERTScore metrics, they found that QAEval-LERC performs better on TAC’08 and SummEval datasets [10, 11].”
MethodStudent Answer Script Evaluation Using Large Language Model
Joseph, H. M., Aishwarya, S., Sriga, & Khumar, H. (2024) · International Conference on Electronics, Computing, Communication and Control Technology (ICECCC) · doi:10.1109/iceccc61767.2024.10593995 · Cited by 4
Optimizing the role of human evaluation in LLM-based spoken document summarization systems
Kroll, M., & Kraus, K. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-2268 · Cited by 7
“…inexpensive automatic evaluations are highly-desired, the limitations of existing automated metrics, e.g. ROUGE and BERTScore, have driven the community to pair these metrics with human evaluation tasks, which are still considered the gold standard for LLM-generated content [2, 3, 1, 4, 5, 6].”
BackgroundCommon Law Annotations: Investigating the Stability of Dialog System Output Annotations
Lee, S., DeLucia, A., Nangia, N., Ganedi, P., Guan, R., Li, R., Ngaw, B., Singhal, A., Vaidya, S., Yuan, Z., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.780 · Cited by 8
To Beam Or Not To Beam: That is a Question of Cooperation for Language GANsPreprint
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., & Staiano, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06363 · Cited by 18
“Three professional English speakers rated 300 sampled summaries and followed the same protocol from Fabbri et al. [9].”
MethodAutomatic Text Summarization Using Deep Learning MethodsPreprint
Shopnil, T. A., Das, A., & Saha, D. (2024) · doi:10.36227/techrxiv.171073069.99938630/v1 · Cited by 3
SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text Generation
Duong, S., Bronnec, F. L., Allauzen, A., Guigue, V., Lumbreras, A., Soulier, L., & Gallinari, P. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2502.13674 · Cited by 8
“However, reference-based metrics limitations are well known to correlate poorly with faithfulness (Fabbri et al., 2021; Gabriel et al., 2021).”
Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying ProbesPreprint
Maiya, S., Liu, Y., Debnath, R., & Korhonen, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.17755 · Cited by 3
“Text Quality The NEWSROOM (Grusky et al., 2018), SummEval (Fabbri et al., 2020), and HANNA (Chhun et al., 2024) datasets all concern the evaluation of text in terms of high-level concepts.”
Cui, W., Zhang, J., Li, Z., Lopez, D., Das, K., Malin, B., & Kumar, S. (2024) · arXiv (Cornell University) · doi:10.13140/rg.2.2.20131.02089 · Cited by 6
Scirè, A., Ghonim, K., & Navigli, R. (2024) · arXiv · doi:10.48550/arxiv.2403.02270 · Cited by 2
“In this section, we present statistics to highlight the difference in sample size between the long-form summarization and short-form summarization datasets, namely, SummaC (Laban et al., 2022), and SummEval (Fabbri et al., 2021).”
BackgroundDSciSum: Detailed summarization of long scientific documents
Liu, R., Mao, X.-L., & Huang, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113409 · Cited by 4
CARE: A Framework for Correcting Numerical Hallucinations in LLM-Generated Financial Texts
Kim, J., & Jung, W. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00018 · Cited by 4
“Self-Refine [8] [15], [16] and used GPT-3.”
Hedderich, M. A., Wang, A., Zhao, R., Eichin, F., Fischer, J., & Plank, B. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.985 · Cited by 6
Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probes
Maiya, S., Liu, Y., Debnath, R., & Korhonen, A. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.444 · Cited by 9
HEAL-Summ: a lightweight and ethical framework for accessible summarization of health information
Fisher, A., Srinivasan, K. K., Hillier, S. A., & Mago, V. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1619274 · Cited by 2
“These reviews served as qualitative sanity checks to validate the coherence, factuality, and tone of summaries (29).”
MIRAGE: A Metrics lIbrary for Rating hAllucinations in Generated tExt
Vendeville, B., Магдановна, Е. Л., Loor, P. D., & Kamps, J. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761644 · Cited by 2
Generating Feedback for School Students Essay with Large Language Models
Zhang, D., Hoang, T. N., Zhu, Y., Wang, R., & Crouch, P. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3055-7_25 · Cited by 3
Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors
Hu, Y., Tong, X., Bai, H., Ding, X., Bharadwaj, S. M., Cao, S., Nowak, R. D., & Zhang, J. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.2092 · Cited by 2
Exsumm-VIZ: Visual Interpretation of Attention Model In Text Summarizations
Salako, A. A., Pu, J., Jinlun, Z., Guanqun, L. T. L., & Agbley, B. L. Y. (2021) · International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) · doi:10.1109/iccwamtip53232.2021.9674053 · Cited by 2
“ROUGE [1], an automatic metric that helps evaluate summarization models, has been demonstrated to be out of sync with human judgments [2] and only provide high-level insights while failing to detect specific model flaws.”
BackgroundDecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering
Ke, P., Huang, F., Mi, F., Wang, Y., Liu, Q., Zhu, X., & Huang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.539 · Cited by 11
Measurement and Identification of Informative Reviews for Automated Summarization
Nguyen, H., Chen, H., Maganti, R., Hossain, K. S. M. T., & Ding, J. (2023) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest58265.2023.00031 · Cited by 2
“2) Are the pre-trained deep learning models generally effective for software review summarization? If they do, which model is the best? Deep learning models have been proven more effective than traditional machine learning models in many studies [21], especially the recent advances in pre-trained deep learning models…”
BackgroundUm Estudo Sobre Métricas de Avaliação para Sumarização de Acórdãos
Feltrin, G. R., Vianna, D., & da Silva, A. S. (2023) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2023.232000 · Cited by 2
Linguistic summarisation of multiple entities in RDF graphs
Zimina, E., Järvelin, K., Peltonen, J., Ranta, A., Stefanidis, K., & Nummenmaa, J. (2024) · Applied Computing and Intelligence · doi:10.3934/aci.2024001 · Cited by 2
“Following the ideas of [3, 25, 26], we put forward several assessment criteria.”
MethodiiPCS: Intent-Based In-Context Learning for Project-Specific Code Summarization
Wang, Y., Liu, X., Lu, X., & Zhou, A. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650776 · Cited by 3
“Fortunately, due to the training process using RLHF or the reward model, LLMs have a strong ability to compare and evaluate text quality like humans [39].”
MethodDeepfake-Driven Lecture Summary Generation
Chugh, T., Srikanth, H., Sundeep, A., Sukruth, K., & Preethi, P. (2024) · Asia Pacific Conference on Innovation in Technology (APCIT) · doi:10.1109/apcit62007.2024.10673451 · Cited by 2
“In an in-depth evaluation, [10], 23 models were tested for summarization tasks and consistently outperformed others on both human and automated measurements.”
BackgroundWiktor, N. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641555.3705044 · Cited by 2
Mobile application review summarization using chain of density prompting
Shrestha, S., & Mahmoud, A. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00533-5 · Cited by 4
“Evaluating the performance of summarization techniques can be a challenging task [40].”
PSentScore: Evaluating Sentiment Polarity in Dialogue Summarization
Zhou, Y., Ringeval, F., & Portet, F. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5ewqiiqn52yz · Cited by 5
“Recent works (Huang et al., 2020; Fabbri et al., 2021) have already pointed out that these metrics do not correlate equally with all kinds of human judgments.”
BackgroundUMSE: Unified Multi-scenario Summarization Evaluation
Gao, S., Yao, Z., Tao, C., Chen, X., Ren, P., Ren, Z., & Chen, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.236 · Cited by 3
Text Summarization Evaluation Using Semantic Probability Distributions
Le, A., Wu, F., Vu, L., & Le, T. T. (2023) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci62032.2023.00038 · Cited by 2
“Intrinsic evaluation methods, thanks to their advantages in terms of computational time and annotation cost, have been used by the great majority of research papers on ATS model evaluation [2] [3].”
MethodPrompt scoring system for dialogue summarization using GPT-3Preprint
Prodan, G., & Pelican, E. (2022) · doi:10.36227/techrxiv.16652392.v2 · Cited by 2
Abstractive Summarization with LLMs for Texts in Brazilian Portuguese
Camargo, H. A. P. G. D., Paiola, P. H., Garcia, G. L., & Papa, J. P. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5811 · Cited by 2
“Recent works suggest combining automatic metrics with targeted human evaluations to achieve a more reliable and comprehensive assessment [Fabbri et al ., 2021].”
HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks
Assadi, A. E., Chung, I., Solomatin, R., Muennighoff, N., & Enevoldsen, K. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2510.10062 · Cited by 5
“Human evaluation is well established in NLP, especially for generative tasks like machine translation (Graham et al., 2013), summarization (Fabbri et al., 2021), and dialogue (Gupta et al., 2019).”
Zeyad, A. M. A., & Biradar, A. (2025) · International Journal of Computer Network and Information Security · doi:10.5815/ijcnis.2025.06.07 · Cited by 2
Faithfulness metric fusion: Improving the evaluation of LLM trustworthiness across domainsPreprint
Malin, B., Kalganova, T., & Boulgouris, N. V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.05700 · Cited by 2
“Some authors have incorporated ground truth data within the LLM prompt, which can aid in its judgment, and other authors retrieve granular outputs (such as 5-point or 100-point scales) instead of a binary classification [15]–[17].”
Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters
Zhang, X., Gao, T., Jin, S., Wang, T., Ye, T., Adar, E., & Mei, Q. (2026) · Transactions on machine learning research · doi:10.48550/arxiv.2510.25860 · Cited by 2
“We use the SummEval benchmark [13], which provides human evaluations of machine-generated summaries of news articles.”
Representation, information extraction, and summarization for automatic multimedia understanding
Harrando, I. (2022) · theses.fr (ABES) · doi:10.70675/702b8ba3z3acaz4ce3za9b9zf2cb1609e328 · Cited by 2
Naser, M. Z. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-026-09938-4 · Cited by 2
Bayesian Calibration of Win Rate Estimation with LLM Evaluators
Gao, Y., Xu, G., Wang, Z., & Cohan, A. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.273 · Cited by 17
“The datasets we use in the experiments are HANNA (Chhun et al., 2022), OpenMEVA-MANS (Guan et al., 2021), SummEval (Fabbri et al., 2021), LLMBar (Zeng et al., 2024), MT-Bench (Zheng et al., 2023), and LLMEval 2 (Zhang et al., 2023), covering tasks of story generation (HANNA, OpenMEVA-MANS),…”
MethodInfluentialSupplementing e-learning systems with adaptive content generation elements
Parahonco, A., & Petic, M. (2023) · Computer Science Journal of Moldova · doi:10.56415/csjm.v31.18 · Cited by 1
Enhancing Academic Title Generation Using SciBERT and Linguistic Rules
Callegari, E., Vajdečka, P., Xhura, D., & Ingason, A. K. (2023) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2023.wiesp-1.10 · Cited by 1
“Considering that most of the evaluation mechanisms based on similarity scores are highly correlated (Fabbri et al., 2021), we decided to resort to ROUGE (Lin, 2004).”
BackgroundLiu, J., Du, W., Li, Q., Wang, X., Zhou, Z., & Liu, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448276 · Cited by 1
“In the QA-based metrics category, we include QAFactE-val [24], QuestEval [14], and FEQA [13] as our baseline models.”
MethodHan, S.-H., Kang, M., Kang, P., & Hwang, W. H. (2024) · Journal of Korean Institute of Industrial Engineers · doi:10.7232/jkiie.2024.50.2.097 · Cited by 1
Benbarka, M., & Kassimi, M. A. (2024) · International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET) · doi:10.1109/iraset60544.2024.10548450 · Cited by 1
“In general, there are four dimensions that is regarded during a human evaluation process [16], [17]: 1) Relevance: each generated summary should contain only the most important elements.”
BackgroundAnalysis and Performance of Text Summarization Tools Applied on Indian Languages
Basu, A., Chatterjee, A., Ghosh, R., Dasgupta, S., Roychowdhury, T., Dutta, P. K., Bhattacharya, P., & Tanwar, S. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-981-97-2839-8_28 · Cited by 1
EFGCLS: A Cross-Lingual Summarization Method Based on Element Fact-Relationship Generation
Huang, Y., Jiushun, M. A., Li, T., Yu, Z., Xian, Y., & Xiang, Y. (2025) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2024edp7274 · Cited by 1
“Meanwhile, sufficient experiments show that reference summaries in these standard datasets perform poorly on human evaluation dimensions, especially coherence, consistency, and relevance [27],[28].”
Enhancing text quality evaluation with integrating content security attributes
Sun, Y., & Zhao, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127234 · Cited by 1
Prompt Scoring System for Dialog Summarization
Prodan, G. P., & Pelican, E. (2025) · The European Journal on Artificial Intelligence · doi:10.1177/30504554251321829 · Cited by 1
“These criteria have been used in other studies as well, e.g. (Fabbri, 2021b) Table 6.”
Enhancing Marathi Abstractive Text Summarization with IndicBART
Mete, A. M., & Dhore, M. L. (2025) · International Conference on Inventive Computation Technologies (ICICT) · doi:10.1109/icict64420.2025.11004889 · Cited by 1
“Length of reference summary 4) BLEU Score: The BLEU metric serves as a common method for evaluating automated text which Papineni et al. presented in 2002 primarily for machine translation evaluation [25].”
Metin Özetleme ve Benzerlik Analizi Üzerine Prototip Bir Çalışma: Google Scholar Örneği
Şahin, O., & Yayla, R. (2025) · Osmaniye Korkut Ata Üniversitesi Fen Bilimleri Enstitüsü Dergisi · doi:10.47495/okufbed.1570542 · Cited by 1
Nethravathy, V., & Mahesh, T. R. (2025) · Intelligent Decision Technologies · doi:10.1177/18724981251346852 · Cited by 1
Precision in conciseness: Exploring large language models for enhanced document summarization
Surabhi, A. S., & Martha, S. (2025) · AIP conference proceedings · doi:10.1063/5.0279955 · Cited by 1
Enhancing LLM-as-a-Judge through Active-Sampling-based Prompt Optimization
Cheng, Z., Zheng, E., Kuang, J., & Tso, G. J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-industry.67 · Cited by 8
Automated Code Comments Generation Using Large Language Models: Empirical Evaluation of T5 and BART
Ghale, D. P., & Dabbagh, M. (2025) · IEEE Access · doi:10.1109/access.2025.3597601 · Cited by 5
“ROUGE-2, evaluates the overlap of bigrams (two consecutive words) between the generated and reference comments and ROUGE-L, assesses the longest common subsequence between the generated and reference comments, capturing word order and allowing for gaps [42], [43].”
From Words to Wisdom: LLMs Summarizing Instructional Content
Chomątek, Ł., Słabosz, W., & Poniszewska-Marańda, A. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728697 · Cited by 1
Mind the Link: Discourse Link-Aware Hallucination Detection in Summarization
Lee, D., Jung, H., & Choi, Y. S. (2025) · Applied Sciences · doi:10.3390/app151910506 · Cited by 2
Mayilvaghanan, K., Gupta, S., & Kumar, A. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-industry.91 · Cited by 1
“Existing metrics (Fabbri et al., 2021; Gao and Wan, 2022) effectively capture general qualities like factual correctness, relevance, and coherence, often relying on human judgments via Likert-scale annotations.”
PMDS: progressive multi-document summarization with iterative summary integration
Yu, S., Yang, C., Qin, Y., Huang, R., Chen, Y., Lin, C., & Gao, W. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2025.104501 · Cited by 1
DisT5: A Text-to-Text transformer model for disaster events
Garg, P. K., Gupta, S., Abbas, S. A., Chakraborty, R., & Dandapat, S. K. (2026) · Online Social Networks and Media · doi:10.1016/j.osnem.2026.100347 · Cited by 1
From Detection to Diagnosis: Advancing Hallucination Analysis with Automated Data Synthesis
Liu, Y., Yang, Q., Guo, T., Qu, F., Liu, J., & Wen, Y. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i38.40495 · Cited by 1
“For the summarization task, we use the Sum-mEval (Fabbri et al. 2020) for benchmarking.”
Marques, M., Fernandes, A. L., Pacheco, A. F., Rebouças, R., Cantante, I., Isidro, J., Cunha, L. F., Jorge, A. O. S., Guimarāes, N., Nunes, S., et al. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792945 · Cited by 1
“The models were evaluated using several metrics known to be reliable quality indicators for summarization tasks [9].”
Nguyen, H., Zhang, H., Zhang, Y., Ding, J., & Chen, H. (2026) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest70988.2026.00042 · Cited by 1
“First, we design a QA evaluation module assessing coverage and factual consistency [3], [11] through diverse question types [21], producing structured feedback that identifies unanswered questions (coverage gaps) and inconsistent fact triplets (erroneous claims with ground-truth corrections).”
InfluentialYe, T., Liu, Y., Qin, X., Li, H., & Wang, X. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134003 · Cited by 2
Alahmadi, M. D. (2026) · IEEE Access · doi:10.1109/access.2026.3696617 · Cited by 1
“To complement these metrics, we designed a human evaluation study following best practices recommended in the literatures [22] and [29].”
Improving Kazakh Abstractive Summarization with LLMs via Domain Adaptation and Self-Correction
Zhabayev, T., Tukeyev, U., Amirova, D., & Rakhmanberdi, N. (2026) · Information · doi:10.3390/info17070639 · Cited by 1
Narrative Consistency in Large Language Model-Generated Stories: A Survey
Park, K., Yu, S., Jang, J., Kwon, H., Lim, B., & Kim, Y. (2026) · IEEE Access · doi:10.1109/access.2026.3710595 · Cited by 1
“SummEval [91], FRANK [92], and TRUE [22] are meta-evaluation benchmarks that measure how well a given metric correlates with human factuality judgments, and they are therefore reference points for metric selection rather than metrics in their own right.”
Metadata
- Authors
- Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D.
- Affiliations
- Yale University (US)
- Salesforce (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 391-409
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3045321166
- Cited by
- 587
- References
- 102
- Citations by year
- 2026: 109 · 2025: 93 · 2024: 74 · 2023: 166 · 2022: 84 · 2021: 57
- FWCI
- 33.37
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Social Sciences › Linguistics and Language › Text Readability and Simplification
- Keywords
- text summarization, summarization evaluation, automatic evaluation metrics, crowdsourced annotation, evaluation toolkit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.