SummEval: Re-evaluating Summarization Evaluation
Open accessThe DOI matches its Crossref record; 99 of the 102 references it cites were found in a registry or an index.
Abstract
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SummEval (Fabbri et al., 2021) is a news domain dataset, where the task is abstractive summarization.The outputs are systemgenerated summaries of CNN/Daily Mail articles, with human ratings provided on coherence, consistency, fluency, and relevance.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Human evaluation has long been considered the gold standard for evaluating the quality of natural language generation (NLG) systems (Belz and Reiter, 2006;Lai and Tetreault, 2018;Fabbri et al., 2021).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indirectly, these metrics also evaluate composition, since it has been shown that ROUGE and BLEU correlate with human judgments on fluency of a text [67].”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“Data in AG-GREFACT comes from 9 factual consistency evaluation datasets on CNN or XSum, including widely used ones such as SummaC (Laban et al., 2022), FRANK (Pagnoni et al., 2021), and SummEval (Fabbri et al., 2021).”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Fabbri et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al.”
BackgroundMethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“ROUGE-WE (Ng and Abrecht, 2015) extends ROUGE by using soft lexical matching based on the cosine similarity of Word2Vec (Mikolov et al., 2013) embeddings.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“However, it has recently been shown that the mentioned datasets contain human judgments for model outputs scoring on a lower scale compared to current summarization systems putting into question the true performance of those metrics in the new setting (Peyrard, 2019).”
BackgroundInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani ∗ Equal contributions from authors et al., 2017) as well as the availability of largescale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et…”
BackgroundProceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Research works printed but not linked: 2
- Dang, 2009, Overview of the TAC 2009 summarization track
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
Cited by
Items 501–600 of 762
Heterogeneous Low-Power Agentic Architecture for Edge Distributed Home Surveillance
Zappia, L. L., Lattuada, M., Bianchessi, M. A., & Pau, D. P. (2026) · Electronics · doi:10.3390/electronics15143175 · Cited by 1
SPAR: Step-wise Path Dispatching and Asymmetric Re-routing for Efficient MoE Inference
Zhang, Q. A., Li, X. F., Kong, J., Liu, X. Y., Ji, B., Li, S., Ma, J., & Yu, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3450-9_32 · Cited by 1
Pakull, T. M. G., Bender, N., Benson, S., Fleischhauer, A., Alsara, M., Gromke, T., Hilser, T., Kaminski, K., Pogorzelski, M., Prasuhn, N., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/99136 · Cited by 1
“Established frameworks such as the Physician Documentation Quality Instrument-9 [31] or SummEval [32] have been proven useful, yet both assign scores to quality criteria rather than pinpointing specific errors, a level of detail that more closely reflects the correction work clinicians actually…”
MARS: Multi-Agent Rescheduling Framework with Large Language Models for Human-Centric Manufacturing
Park, J., Kim, K. S., Faturrahman, Z., & Kim, H.-J. (2026) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2026.07.015 · Cited by 1
Boughida, A., Kouahla, M. N., Benoughidene, A., Kouadria, A., & Kurulay, M. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08750-9 · Cited by 1
“Evaluations were structured around four dimensions adapted from clinical documentation guidelines (inspired from [2, 34]): • Coherence : Logical flow and narrative structure (1 = disjointed and 5 = excellent progression).”
The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review
Yang, E., Ko, S., & Woo, H. (2026) · Journal of Medical Internet Research · doi:10.2196/98184 · Cited by 2
“Nevertheless, human evaluation serves as an important complementary approach to benchmark-based evaluation because it can assess the context and qualitative characteristics of responses that are difficult to capture using quantitative evaluation metrics alone [126,131,132].”
de Andrade, C. M. V., Belém, F. M., Franca, C., Carvalho, M., Ganem, M., Texeira, G., Jallais, G. F., Laender, A. H. F., & Gonçalves, M. A. (2025) · Journal of Information and Data Management · doi:10.5753/jidm.2025.4298 · Cited by 1
Baradziej, S. (2026) · Frontiers in Language Sciences · doi:10.3389/flang.2026.1953328 · Cited by 1
Does Summary Evaluation Survive Translation to Other Languages?
Braun, S., Vasilyev, O., Iskender, N., & Bohannon, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.173 · Cited by 1
Evaluation of Extractive and Abstract Methods in Text Summarization
Lenka, R. K. B., Coombs, T., Assi, S. A., Jayabalan, M., Mustafina, J., Liatsis, P., Hamid, A. A., Al-Sudani, S., Ismail, N. L., & Al‐Jumeily, D. (2023) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-99-0741-0_38 · Cited by 1
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types
Murugesan, K., Swaminathan, S., Dan, S., Chaudhury, S., Gunasekara, C., Crouse, M., Mahajan, D., Abdelaziz, I., Fokoue, A., Kapanipathi, P., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.274 · Cited by 7
Revisiting Sentence Union Generation as a Testbed for Text Consolidation
Hirsch, E., Pyatkin, V., Wolhandler, R., Caciularu, A., Shefer, A., & Dagan, I. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.440 · Cited by 1
A Study on Annotation Interfaces for Summary Comparison
Gooding, S., Werner, L., & Cărbune, V. (2023) · Linguistic Annotation Workshop · doi:10.18653/v1/2023.law-1.18 · Cited by 1
“For instance, it has been found that crowd workers should not be used to evaluate summary quality because of a non-correlation with experts (Gillick and Liu, 2010; Fabbri et al., 2021).”
BackgroundUniversal Evasion Attacks on Summarization Scoring
Mu, W., & Lim, K. H. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.9 · Cited by 5
Learning to Generate Overlap Summaries through Noisy Synthetic Data
Bansal, N., Akter, M., & Santu, S. K. K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.807 · Cited by 4
“Following (Fabbri et al., 2021; West et al., 2021), we further involved human judges to evaluate the quality of generated synthetic samples.”
BackgroundMethodAssessing Inter-metric Correlation for Multi-document Summarization Evaluation
Ridenour, M., Agrawal, A., & Olabisi, O. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.40 · Cited by 2
“With each dataset we collect system summaries for a set of 100 randomly selected samples from the test set, following recent work on measuring correlations between metrics (Bhandari et al., 2020; Fabbri et al., 2021b; Gao and Wan, 2022).”
BackgroundMethodInfluentialHiding in Plain Sight: Insights into Abstractive Text Summarization
Srivastava, V. P., Bhat, S., & Pedanekar, N. (2023) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2023.insights-1.8 · Cited by 2
Multi-View Source Ablation for Faithful Summarization
Cao, S., Ma, L., Lu, D., Logan, R. L., Tetreault, J., & Jaimes, A. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.151 · Cited by 1
“, 2022), a benchmark consisting of documentsummary pairs and their binary faithfulness labels annotated by most recent work (Kryscinski et al., 2020; Maynez et al., 2020; Huang et al., 2020; Fabbri et al., 2021; Pagnoni et al., 2021; Cao and Wang, 2021; Goyal and Durrett, 2021; Cao et al., 2022).”
BackgroundThreats of Adversarial Attacks on Deep Learning
Regmi, S., & Regmi, S. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.55716 · Cited by 1
Linguistic Compression in Single-Sentence Human-Written Summaries
Yin, F., & van Schijndel, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.532 · Cited by 1
Larionov, D., Viskov, V., Kokush, G., Panchenko, A., & Eger, S. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.19 · Cited by 1
“For the construction of the example pool for summarization, we use SummEval (Fabbri et al., 2021) dataset.”
MethodDutta, T., & Liu, X. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650810 · Cited by 3
“However, they focus on mostly identifying if the generated text is hallucinated (Ribeiro et al. [2022], Fabbri et al. [2021a]).”
BackgroundHuang, J., Wei, Y., Zhang, L., & Chen, W. (2024) · International Symposium on Educational Technology · doi:10.1109/iset61814.2024.00022 · Cited by 1
“Recent research indicates that machine metrics often show no strong correlation with the quality of the generated text [34].”
BackgroundSarawanangkoor, N., Punyabukkana, P., & Suchato, A. (2024) · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP) · doi:10.1109/isai-nlp64410.2024.10799455 · Cited by 1
“We tested our metric on the SummEval dataset [10], achieving competitive or superior performance compared to state-of-the-art metrics.”
EADSum: Element-Aware Distillation for Enhancing Low-Resource Abstractive Summarization
Lu, J., Yan, B.-C., Wang, Y., Lo, T.-H., Wang, H., Pai, L.-T., & Chen, B. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10848820 · Cited by 1
“[19].”
Automatic Text Summarization for Low-Resource Settings and Languages: A Survey
Mitri, A. M., Saha, G. K., Maji, A. K., & Lyngdoh, S. A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-8476-9_5 · Cited by 1
A Rate-Distortion Framework for Summarization
Arda, E., & Yener, A. (2025) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit63088.2025.11195348 · Cited by 3
“The recent success of Large Language Models (LLMs) across diverse tasks has expanded the realm of summarization as well, enabling the generation of remarkably high-quality summaries [2], [3].”
Beckage, N., Kumar, S. H., Sahay, S., & Manuvinakurike, R. (2021) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2021.newsum-1.11 · Cited by 1
“See et al. (2017); Chen and Bansal (2018); Zhao et al. (2019); Liu (2019); Zhang et al. (2020); Feng et al. (2020); Zhu et al. (2020); Fabbri et al. (2021b)).”
BackgroundMethodOpen4Business(O4B): An Open Access Dataset for Summarizing Business DocumentsPreprint
Singh, A., & Balasubramanian, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07636 · Cited by 2
“Since the focus of this work is solely on abstractive summarization, SOTA encoder-decoder trans-former based models [15] are used for benchmarking O4B.”
MethodPrompt scoring system for dialogue summarization using GPT-3Preprint
Prodan, G., & Pelican, E. (2021) · doi:10.36227/techrxiv.16652392.v1 · Cited by 1
Abstractive Summarization System
Kumar, A., & Gupta, M. K. (2022) · Journal of Electronics and Informatics · doi:10.36548/jei.2021.4.006 · Cited by 1
Prompt scoring system for dialogue summarization using GPT-3Preprint
Prodan, G., & Pelican, E. (2021) · doi:10.36227/techrxiv.16652392 · Cited by 1
Similarity between Units of Natural Language: The Transition from Coarse to Fine EstimationPreprint
Mu, W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.14275 · Cited by 1
Building generalizable models for discourse phenomena evaluation and machine translation
Jwalapuram, P. (2022) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/165027 · Cited by 1
The Extractive-Abstractive Axis: Measuring Content "Borrowing" in Generative Language ModelsPreprint
Teneva, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.11779 · Cited by 1
“…tasks are benchmarked using human anno-tators who rate the generated summaries along dimensions such as summary relevance, fluency, coherence (Fabbri et al., 2020) or the level of factual alignment between the sum-mary and the underlying source text being summarized (e.g., answer…”
BackgroundCorrection of Errors in Preference Ratings from Automated Metrics for Text Generation
Deriu, J., von Däniken, P., Tuggener, D., & Cieliebak, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.404 · Cited by 9
Block, J. E., Honeycutt, D. R., Benda, B., Rheault, B., & Ragan, E. D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.04616 · Cited by 1
“However, in the NLP domain, the focus has been on both automated scoring functions (e.g., BLEU, ROUGE, etc.) and human evaluations to determine the agreement between a summary and the individual text document it was generated from [22].”
BackgroundSeeing Through the Fog: A Cost-Effectiveness Analysis of Hallucination Detection SystemsPreprint
Thomas, A., Rosen, S., & Vettrivel, V. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.05270 · Cited by 2
Evaluating large language models for quality control in research ethics review
Maghsoudlou, P., Fennelly-Barnwell, J., Abraham, A., Kapacee, Z., Allen, C., Totenhofer, A., Mamtora, S., Keane, P. A., Denniston, A. K., Liu, X., et al. (2026) · Scientific Reports · doi:10.21203/rs.3.rs-6417944/v1 · Cited by 2
Abstractive Summarization for Urdu Video Description Generation
Faheem, A., Ullah, F., Ayub, M. S., & Karim, A. (2025) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3762992 · Cited by 1
“ROUGE is unable to fairly evaluate the quality of the generated summary as an abstractive summary potentially contains new words that may not be present in the source document [22, 59].”
Using AI to Summarize US Presidential Campaign TV Advertisement Videos, 1952–2012
Breuer, A., Dietrich, B. J., Crespin, M. H., Butler, M., Pryse, J. A., & Imai, K. (2025) · Scientific Data · doi:10.1038/s41597-025-05558-9 · Cited by 4
“We then asked political science graduate students to answer four questions about each summary, corresponding to the four summary evaluation metrics identified in the literature: coherence, consistency, fluency, and relevance [55, 396].”
Benzoni, C., Langhals, M., Boeker, M., Modersohn, L., & Maros, M. E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.15419 · Cited by 1
“Furthermore, automatic evaluation in such seq2seq tasks as summarisation is known to be problematic [63] [64], [37], and correlation with human judgement remains the most reliable way of assessing the output quality [65].”
Aloraini, E., Shaban, K., Hamdi, A., & Elmahjub, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6582144 · Cited by 1
Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM JudgesPreprint
Tapwal, R., Kumar, A., & Maple, C. (2026) · arXiv.org · doi:10.48550/arxiv.2605.23970 · Cited by 1
“In practice, the answer is largely no [2].”
Metric Match: A Subset Selection Approach to Evaluating LLM Judge ReliabilityPreprint
Unell, A., Dullerud, N., Boneh, N., Jagadeesan, M., Hashimoto, T., Shah, N. H., & Koyejo, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.15029 · Cited by 1
“We provide empirical evaluations on 4 diverse real-world datasets: MSLR [49], HANNA [12], MedVAL [1], and SummEval [19].”
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM JudgesPreprint
Chen, X., Mu, J., Xuan, M., & Shao, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.12097 · Cited by 1
Inter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended TextPreprint
Li, H., & Chaturvedi, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.23783 · Cited by 1
“Therefore, they are generally evaluated along multiple dimensions (Fabbri et al., 2020).”
InfluentialScoring the Wrong Question: Readout Failures in Constrained-Option EvaluationPreprint
Guo, J., Zhang, K., Xin, S., Jingxin, Y., Sun, Y., & Yang, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.33179 · Cited by 1
Hallucination Detection-Guided Preference Optimization for Clinical Summarization
Seethakantha, S. K., Thai, D. N., Gudi, V. P., Tiwari, S., Matar, R., Mitra, A., Zhao, W., McCallum, A., & Salloum, W. (2026) · arXiv · doi:10.48550/arxiv.2605.28910
“Training-time methods include domain pretraining, continual pretraining, supervised fine-tuning, parameter-efficient adapters (Veen et al., 2024; Zaretsky et al., 2024), and specialized loss functions or reinforcement learning objectives (Fabbri et al., 2021; Bao et al., 2025; Asgari et al., 2025).”
All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs?
Kim, S., Janssen, M. A., Wang, J., Min-Venditti, A., Karanjia, N., & Anderies, J. M. (2026) · arXiv · doi:10.48550/arxiv.2604.17247
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Liu, Y., James, J., Wang, Y., Zhao, K., Xiao, C., & Lin, C. (2026) · arXiv · doi:10.48550/arxiv.2610.07109
“We evaluate JudgeMoE on direct-score and pairwise benchmarks spanning summarization (SummEval (Fabbri et al., 2021 and HelpSteer2 (Wang et al., 2024b)), reasoning/explanation quality (ROSCOE-eSNLI (Cam-buru et al., 2018; Golovneva et al., 2023)), translation quality estimation (WMT MQM (Freitag et…”
The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
Marioriyad, A., Rohban, M. H., & Baghshah, M. S. (2025) · arXiv · doi:10.48550/arxiv.2509.26072
“Large language models (LLMs) are increasingly used as judges to evaluate the outputs of other systems across diverse open-ended tasks, including summarization [4], dialogue [8], and creative writing [6].”
Estimation of Summary-to-Text Inconsistency by Mismatched EmbeddingsPreprint
Vasilyev, O., & Bohannon, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05156
“We used SummEval dataset 3 (Fabbri et al., 2020) for comparing ESTIME with a few well known or promising evaluation measures.”
BackgroundMethodInfluentialImproving Reader Motivation with Machine Learning
Bohn, T. (2021) · Scholarship@Western (Western University)
Ghosal, T., Singh, M., Nedoluzhko, A., & Bojar, O. (2021) · ACM SIGIR Forum · doi:10.1145/3527546.3527561
“We refer to the SummEval 19 [Fabbri et al., 2021] and SacreROUGE 20 [Deutsch and Roth, 2020] packages here which caters to the aforesaid requirement.”
MethodCombining local and global approaches to ascertain semantic similarity
Gouhar, S., Misra, A., Rathore, R., Shaik, M. A., & Dasgupta, S. (2022) · IEEE India Council International Subsections Conference (INDISCON) · doi:10.1109/indiscon54605.2022.9862898
“…there are different domains where researchers have produced plausible results such as information retrieval [7], [8], machine comprehension [9], [10], semantic concept detection [11], [12], machine translation [13], [14], sentiment analysis [15], [16], text summarisation [17], [18] and many more.”
BackgroundRank-Aware Gain-Based Evaluation of Extractive Summarization
Akter, M. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557821
“To compute the human correlation of the new redundancy-aware Sem-nCG metric, we utilized data from [4].”
MethodImage Captioning with External Knowledge
Nikiforova, S. (2023) · Utrecht University Repository (Utrecht University) · doi:10.33540/1709
Uncovering Hidden Consequences of Pre-training Objectives in Sequence-to-Sequence Models
Kew, T., & Sennrich, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.438
“…of seq2seq PLMs, covering a wide array of tasks including abstractive summarisation (Blekanov et al., 2022; Zhu et al., 2021; Tang et al., 2022; Fabbri et al., 2021), question answering (Luo et al., 2022), graph-to-text generation (Ribeiro et al., 2021), dialogue modelling (Shin et al., 2022)…”
BackgroundThe Fourth Workshop on Insights from Negative Results in NLP
Tafreshi, S., Akula, A., Sedoc, J., Валентинович, Д. А., Rogers, A., & Rumshisky, A. (2023) · doi:10.18653/v1/2023.insights-1 · Cited by 1
SUMARIZAÇÃO DE ARTIGOS CIENTÍFICOS EM PORTUGUÊS NO DOMÍNIO DA SAÚDE
NASCIMENTO, D. N. C. R. D. (2023) · doi:10.17771/pucrio.acad.64511
Multilingual abstractive summarization of podcasts with Longformers
Tanaka, E. K. (2022) · doi:10.47749/t/unicamp.2022.1259169 · Cited by 2
Evaluating the Summarization Comprehension of Pre-Trained Language Models
Chernyshev, D., & Dobrov, B. V. (2023) · Lobachevskii Journal of Mathematics · doi:10.1134/s1995080223080115
“To address the issue of the incompleteness of evaluation protocol Fabbri et al. [5] re-evaluated available evaluation metrics and benchmarked 23 state-of-the-art summarization models.”
BackgroundMethodBao, F. S., Tu, R., Luo, G., Yang, Y., Li, H., Qiu, M., He, Y., & Chen, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.87 · Cited by 2
“SummEval covers four aspects: CONsistency, RELevance, COHer-ence, and FLUency.”
MethodInfluentialMulti Document Summarization Evaluation in the Presence of Damaging Content
Manevich, A., Carmel, D., Cohen, N., Kravi, E., & Shapira, O. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1 · Cited by 1
“The various metrics capture a summary’s quality-aspects including relevance, coherence, faithfulness and more (Fabbri et al., 2021).”
EntSUMv2: Dataset, Models and Evaluation for More Abstractive Entity-Centric Summarization
Mehra, D., Xie, L., Hofmann-Coyle, E., Kulkarni, M., & Preoțiuc-Pietro, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.337 · Cited by 5
Natural Language Generation with Reinforcement Learning
Scialom, T. (2022) · HAL · Cited by 1
Ehnert, P., & Schröter, J. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1200949
Fushman, D. D. (2024) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-55865-8_9
The Likelihood Gain of a Language Model as a Metric for Text Summarization
Levin, D., & Kipnis, A. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619426
“We use the following datasets: • SummEval [26].”
Learning to generate text with auxiliary tasks
Hùng, P. Q., Nguyen, M.-T., Inoue, S., Tran-Tien, M., & Phan, X.-H. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112570
A Dataset for Evaluating LLM-based Evaluation Functions for Research Question Extraction TaskPreprint
Fujisaki, Y., Takagi, S., Asoh, H., & Kumagai, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.06883
“Research on automatic evaluation of document summarization has verified the correlation between automatic evaluation functions and human evaluation (Fabbri et al., 2020).”
BackgroundMethodA Logical Model for Recognizing Sentences with Punctuation using Python API
Aparna, C., & Rajchandar, K. (2024) · International Conference on IoT Based Control Networks and Intelligent Systems (ICICNIS) · doi:10.1109/icicnis64247.2024.10823228
“Other notable works include a Bengali-to-text model with 71.7% accuracy [4] and improvements in summarization methods [5], [6], [7].”
InfluentialImproved Text Summarization with RL-Enabled PEGASUS and Siamese Network Evaluation
Dhyuti, N. S., Mercy Faustina, J., & Sivaselvan, B. (2024) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon61640.2024.10902830
“Existing evaluation metrics such as ROUGE, BLEU, and CHRF, developed in conferences like DUC and TAC, rely on word frequency counts between generated and reference outputs [3].”
Leveraging Deep Learning and NLP for Automated Text Summarization in Healthcare
Rani, R. U., & Swarupa, M. L. (2025) · E3S Web of Conferences · doi:10.1051/e3sconf/202561903005
Advancements in Quantum Imaging for Early Detection of Cancer
Kazem, A. J., Fahd, L. B. A., Jassim, H. T., Marid, H. T., & Abbas, M. R. (2025) · Bright Sky Publications eBooks · doi:10.62906/bs.book.373
Herserant, T., & Guigue, V. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8186-0_11
Methods to Evaluate and Improve the Quality of Text Summarization
Nguyen, H. (2025) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc2443390 · Cited by 1
Can We Employ LLM to Meta-Evaluate LLM-Based Evaluators? A Preliminary Study
Wang, H., & Yu, L. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_14
Wassista, D., & Jain, A. (2025) · International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV) · doi:10.1109/icicv64824.2025.11085882
“Fabbri et al. [26] proposed SummEval, a comprehensive benchmark that combines multiple metrics along with expert and crowd-sourced human judgments to assess coherence, consistency, relevance, and fluency.”
He, Y., Li, G., Liu, F., & Yin, S. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-93251-9_21
A Multi-Document Summarization Method for Customer Feedback Based on Large Language Models
Bao, S., Guan, X., Cao, B., & Yang, C. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-93251-9_15
Singh, A., Ghosh, S. K., Singh, K., & Lepri, B. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3587696
“These metrics are selected based on recent studies evaluating LLM generations [26], [27], [28], and they have been shown to correlate well with human judgments of quality and informativeness.”
Harnessing AI for Business Insight: Key Considerations for Deploying LLMs in Summarization Pipelines
Miller, A. P., Hosanagar, K., & Krishnan, R. (2025) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780198940272.013.0045
When graph anomaly breaks the coherence: A multi-evidence approach with language models
Cheng, X., Lu, J., Xiao, C., Yang, M., Zhong, M., & Zhou, F. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129557
Fine-Tuning BART for Croatian Abstractive Text Summarization Using a Novel News Dataset
Davidović, V., & Martinčić-Ipšić, S. (2025) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro65660.2025.11131703
“Automatic summarization is considered as a transformation from a longer input text into its shorter version with the goal of creating a fluent, coherent, consistent, and relevant summary of the input text [8].”
A Study on Generative Text Summarization Methods Based on Improved Pointer Generation Networks
Liu, Z., Tan, X., Liu, J., & Yu, S. (2025) · International Conference on Communication Networks and Smart Systems Engineering (ICCNSE) · doi:10.1109/iccnse66404.2025.11144293
Lee, Y., Noh, Y., Ryu, J., An, S., Kweon, J., & Lee, S. (2025) · 국토계획 · doi:10.17208/jkpa.2025.08.60.4.148
Advancements in Automatic Text Summarization using Natural Language Processing
Irfan, R., Tehseen, R., Mustaqeem, A., Saeed, R., Aamer, U., & Hassan, J. (2025) · International Journal of Innovations in Science and Technology · doi:10.33411/ijist/20257211451160
Enhancing Automatic Text Summarization: Extractive, Abstractive, and Hybrid Approaches
Thakar, H. A., Pradhan, V., & Thakar, D. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-4536-7_28
Entity hallucination index for text summarization models
Katwe, P., Balabantaray, R. C., & Vittala, K. P. (2025) · Applied Computing and Informatics · doi:10.1108/aci-09-2024-0371 · Cited by 2
Data Profile Generation Framework for Data Utilization
Matsumoto, I., Matsui, T., Fujita, Y., Suwa, H., & Yasumoto, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4575-9_6
An Automated Length-Aware Quality Metric for SummarizationPreprint
Foland, A. D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.07653 · Cited by 2
“The process of comparing a summary to reference summaries (and not the parent paragraph) using the ROUGE metric is described as "standardized" for summarization evaluation in [Fabbri et al., 2021].”
Lin, H., Gao, W. G., Ma, J., Deng, Y., Luo, Z., Wang, B., Yang, R., & Chua, T. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3637552 · Cited by 4
“Additionally,theheterogeneousnatureoftextnecessitatesthatdistinct attributessuchasinformativeness,fluency,soundness,etc.[61], beevaluated.”
Text Tree Edit Distance: A Language Model-Based Metric for Text Hierarchies
Sobolevsky, F., & Vorontsov, K. V. (2025) · International Scientific-Technical Conference on Actual Problems of Electronic Instrument Engineering (APEIE) · doi:10.1109/apeie66761.2025.11289395
“This approach, however, does not account for the interrelation between structure of text trees and their textual contents, and statistical metrics like ROUGE may not account for text semantics [5].”
From Speech to Summary: A Comprehensive Survey of Speech Summarization
Retkowski, F., Züfle, M., Sudmann, A., Pfau, D., Niehues, J., & Waibel, A. (2025) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000180972 · Cited by 3
Galanis, D. (2023) · DSpace - NTUA (National Technical University of Athens) · doi:10.26240/heal.ntua.24737 · Cited by 1
Metadata
- Authors
- Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D.
- Affiliations
- Yale University (US)
- Salesforce (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 391-409
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3045321166
- Cited by
- 587
- References
- 102
- Citations by year
- 2026: 109 · 2025: 93 · 2024: 74 · 2023: 166 · 2022: 84 · 2021: 57
- FWCI
- 33.37
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Social Sciences › Linguistics and Language › Text Readability and Simplification
- Keywords
- text summarization, summarization evaluation, automatic evaluation metrics, crowdsourced annotation, evaluation toolkit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.