SummEval: Re-evaluating Summarization Evaluation
Open accessThe DOI matches its Crossref record; 99 of the 102 references it cites were found in a registry or an index.
Abstract
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SummEval (Fabbri et al., 2021) is a news domain dataset, where the task is abstractive summarization.The outputs are systemgenerated summaries of CNN/Daily Mail articles, with human ratings provided on coherence, consistency, fluency, and relevance.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Human evaluation has long been considered the gold standard for evaluating the quality of natural language generation (NLG) systems (Belz and Reiter, 2006;Lai and Tetreault, 2018;Fabbri et al., 2021).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indirectly, these metrics also evaluate composition, since it has been shown that ROUGE and BLEU correlate with human judgments on fluency of a text [67].”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“Data in AG-GREFACT comes from 9 factual consistency evaluation datasets on CNN or XSum, including widely used ones such as SummaC (Laban et al., 2022), FRANK (Pagnoni et al., 2021), and SummEval (Fabbri et al., 2021).”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Fabbri et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al.”
BackgroundMethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“ROUGE-WE (Ng and Abrecht, 2015) extends ROUGE by using soft lexical matching based on the cosine similarity of Word2Vec (Mikolov et al., 2013) embeddings.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“However, it has recently been shown that the mentioned datasets contain human judgments for model outputs scoring on a lower scale compared to current summarization systems putting into question the true performance of those metrics in the new setting (Peyrard, 2019).”
BackgroundInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani ∗ Equal contributions from authors et al., 2017) as well as the availability of largescale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et…”
BackgroundProceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Research works printed but not linked: 2
- Dang, 2009, Overview of the TAC 2009 summarization track
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
Cited by
Items 601–700 of 762
Code Summarization and Program Synthesis with Large Language Models
Lam, K. (2024) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 1
Krohn, A. (2023) · KTH Publication Database DiVA (KTH Royal Institute of Technology)
Liu, Y. (2024) · DigitalGeorgetown (Georgetown University Library) · Cited by 1
Ji, T. (2022) · Dublin City University Open Access Institutional Repository (Dublin City University)
Böke, S. (2025) · Lund University Publications Student Papers (Lund University)
Leveraging large language models for automated chart summarization
Bazin, R. (2025) · eScholarship@McGill (McGill) · doi:10.82308/30829
Enhancing Faithfulness in Abstractive Summarization via Span-Level Fine-TuningPreprint
Huang, S., Yan, Q., Wang, S., & Lane, I. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.09915
“We use GPT-4o as the underlying model and report its consistency score, which measures the factual alignment between the summary and the source document (Fabbri et al., 2021).”
Étude critique du corpus CNN/DailyMail pour le résumé automatique
Bachey, F., Rodrigues, C., & Bossard, A. (2025) · HAL
MAD-HD: Multi-agent Debate-Driven Ungrounded Hallucination Detection
Gao, Z., Bao, F., & Li, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3343-5_39
Bianchini, D., & Garda, M. (2025) · Digital Government Research and Practice · doi:10.1145/3785143
“For each session 𝑆 1,2,3 , the domain experts were asked to express their degree of agreement or disagreement with seven claims (inspired by [18]) for each crime type, based on the output produced by the LLM. Specifically, they were required to assign a score from 1 to 5 using a Likert Scale [23],…”
Hybrid Generative AI Framework for Multilingual Text Summarization in Indian Languages
Singh, M. K., Maurya, A., & Kumar, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-10756-5_9
Afzal, A., Mercier, A., & Matthes, F. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-0887-7_27
Probabilistic Guarantees for Reducing Contextual Hallucinations in LLMsPreprint
Rautenberg, N., & Schippkus, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.00641 · Cited by 1
“Summarization studies further demonstrate that even conditioned generation systems often produce unsupported claims (Fabbri et al. 2021; Kryscinski et al. 2020; Maynez et al. 2020), and retrieval-augmented models continue to introduce content not present in the retrieved passages (Adlakha et al.…”
MediSum: A NLP Framework for Automatic Summarization of Medical Reports
Shrivastava, T., Patel, V., THAKUR, Y. S., & Butani, V. (2026) · International Journal of Advancement and Innovation in Technology and Research · doi:10.63844/ijaitr.v3.i1.2026.198-200
Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP SystemPreprint
Bang, G., Kim, D., & Min, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.05267 · Cited by 1
“We employ criteria grounded in automatic summarization evaluation to assess established quality dimensions of generated summaries [9].”
A Comparative Analysis of Chatgtp-4/4.5 and Human-written Summaries in Linguistic Research
Stošić, A., Milojković, M., & Branković, A. (2025) · International Conference on e-Learning (eLearning) · doi:10.46793/elearning2025.205s
Evaluating Toxicity Understanding of LLM Agents
Mothilal, R. K., Ahmed, S. I., & Guha, S. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00121
“…concerns around achieving high level of consensus among human annotators [43] or annotation quality gap between crowdworkers and experts [44], [45], as discussed below, the human assessment setup followed by almost all prior works for toxicity-related inquiries largely involve a reductionist…”
Elatiky, A. F., Hamad, A. M., Khaled, H., & Fayez, M. (2026) · Algorithms · doi:10.3390/a19020096
“Even higher-order n -gram variants struggle to reflect coherence [16].”
Wu, P. Y., Chou, H.-W., & Chen, K.-Y. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3659424
“These approaches generally fall into three categories: (1) Multi-annotator consensus methods [10], which aggregate multiple human references to approximate a gold standard; (2) Quality filtering pipelines [48], which employ automatic metrics or human verification to prune unfaithful or low-quality…”
Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Models
Striglli, E. (2026) · DSpace - NTUA (National Technical University of Athens) · doi:10.26240/heal.ntua.30996
DIAL-SUMMER: A Structured Evaluation Framework of Hierarchical Errors in Dialogue SummariesPreprint
Ramnath, S., Chitsazan, N., Zhou, M., Lee, C.-H., Zhang, S.-X., Rawls, S., Sahu, S., Cho, S., Ren, X., Winata, G. I., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.08149
“SummEval (Fabbri et al., 2020) identified core relevant properties that a summary has to satisfy (coherence, consistency, fluency, relevance) and released corresponding human annotations on a subset of CNN/DailyMail summaries (Nallapati et al., 2016); these human annotations could be used to…”
Wang, J., Liang, Y., Sun, Y., & Gong, B. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3777962
CAST: Achieving Stable LLM-based Text Analysis for Data AnalyticsPreprint
Xie, J., Li, Z., He, W., Ding, R., Han, S., & Zhang, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.15861
“Most prior work evaluates generation quality or factual consistency, rather than the stability of outputs (Fabbri et al., 2021; Liu et al., 2023).”
Topic Models Meet LLMs: Generating Insightful Questions from User Feedback
Zhang, D., Wang, Y., Nayak, R., & Bashar, M. A. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-6888-8_14
The Perils of Naive Truncation: A Context Ablation Study for Dialogue Summarization on DialogSum
Rafi, S., Boddu, L. V. S. R. K., Kolla, D. V., Pothuguntla, C., & Myla, R. B. (2025) · International Conference on Artificial Intelligence and Signal Processing (AISP) · doi:10.1109/aisp68263.2025.11396283
Perumal, S. P., Prabhu, P., & Mujtaba, T. (2025) · IEEE Conference on Information and Communication Technology (CICT) · doi:10.1109/cict67193.2025.11399068
“Final summarization and QA step of WIReTT pipeline is pre-trained on SummEval benchmark dataset [23] and finetuned with the output of translation module.”
A text consistency evaluation method combining unified modeling and mixture-of-expert networks
Zhang, L., Xiong, W., Yang, R., & Li, M. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131809
Semantic structure fusion graph for abstractive dialogue summarization
Wang, F., Zhu, Z., Lu, Q., Gong, S., Pei, H., Fu, Z., & Zhao, D. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133176
AI-Powered Automatic Summarization of Short Documents: A Transformer-Based Approach
Mohamed Irshath, M., Robert David, J., Rajesh Kumar, S., & Balaji, D. C. (2026) · International Conference on Computational Intelligence in Data Science (ICCIDS) · doi:10.1109/iccids69108.2026.11407829
“The gains seen here corroborate recent assertions that training-time objective design might lessen hallucination, which has been recognized as a persistent failure pattern of abstractive systems in previous studies [15], [16].”
InfluentialAn Iterative Multi-Agent Analysis for Automated Evaluation in NLG Tasks
Alhawasi, H., Shan, R., & Youssef, A. S. (2025) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi67134.2025.00012
“Second, the Summarization task summarizes input texts, which, along with their respective ground truth summaries, are provided in the SummEval [18] dataset.”
Борисов, А. А., Burtsev, T., Kosov, P., Bobrovskaya, T. V., Vasilev, Y. A., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Pamova, A. P., & Arzamasov, K. M. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1768005 · Cited by 1
“These metrics provide good scalability and consistency; however, they primarily assess the coincidence of n-grams at the surface level, without considering the deep semantic interactions of terms and without evaluating the semantic correctness of the transmitted information (Fabbri et al., 2021).”
A causality-aware and hallucination-mitigation framework for faithful news summarization
Yang, C., Yu, S., Chen, Y., Huang, R., Qin, Y., & Lin, C. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132143
LLM-guided data distillation for explainable recommender system
Basak, A., & Patra, B. K. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08407-7
Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document StreamsPreprint
Lee, Y., Lim, Y., Jung, W., Choi, W., & Yoon, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.19250
A Sentence-Level Risk Estimator for Identifying Hallucinations in Generative AI
Gupta, V., Chakravarti, L., Akhtar, M. M., Maheshwari, P., Garg, P., & Tiwari, D. (2026) · International Conference on AI-Driven Smart Systems and Ubiquitous Computing (ICAUC) · doi:10.1109/icauc68182.2026.11441054 · Cited by 10
“SummEval [12] evaluated 14 metrics across 23 models, demonstrating how diverse metrics including QA-based, NLI-based, and embedding-based approaches correlate differently with human judgments, motivating the need for combining multiple signals.”
Ethical Considerations of Large Language Models: Challenges and Best Practices
(2026) · Frontiers research topics · doi:10.3389/978-2-8325-7702-8
Apprendre à représenter et à générer du texte en utilisant des mesures d'information
Colombo, P. (2021) · theses.fr (ABES) · doi:10.70675/3ef6e602zd6bfz4736zbe24z9f76afee3067 · Cited by 1
Approches neuronales pour le résumé abstractif de transcriptions de parole
Tardy, P. (2021) · theses.fr (ABES) · doi:10.70675/f561056fz76c9z482fz8a91z214c60b57183
Characterization and detection of semantic textual outliers
Pantin, J. (2023) · theses.fr (ABES) · doi:10.70675/3abc7c1bz47b8z44b9z9a09z23c263b7a2af
Analysis and control of online interactions through neural natural language processing
Laugier, L. (2022) · theses.fr (ABES) · doi:10.70675/4743aa15z209az4addz8f29z95150aa230b8 · Cited by 2
S'informer sur les médias sociaux via l'élaboration d'information synthétique
Dusart, A. (2022) · theses.fr (ABES) · doi:10.70675/5770eeadz1145z476bz840czb60749893c5a
Contribution to natural language generation : systems and evaluation
Eddine, M. K. (2022) · theses.fr (ABES) · doi:10.70675/d1be4d48z831bz40cdzb61az9296693d2f3a
Meta-Evaluation Methodology and Benchmark for Automatic Story Generation
Chhun, C. (2024) · theses.fr (ABES) · doi:10.70675/ec296eb8z87a1z4fdbzbbbdz12bb27449b8f · Cited by 2
Large Language Models for Causal Inference in Eutrophication Control of the as Conchas Reservoir
Soilán, M., Araújo, M., Balado, J., & Pazo, M. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-16995-2_8
Enhancing Requirements Elicitation through App Stores Mining
Wei, J. (2024) · theses.fr (ABES) · doi:10.70675/846c68aezfb1ez4f32z8c81z0a2d2b71d77c
Data-Based Natural Language Generation : Evaluation and Explainability
Faille, J. (2023) · theses.fr (ABES) · doi:10.70675/3e73dd43z7aa6z4405za5cez27c78b408a3c
Zhou, Y. (2024) · theses.fr (ABES) · doi:10.70675/8ec4dbe0z863dz45b6z8150zbdca53aadd38 · Cited by 1
Contributions to evaluating and improving the faithfulness for text generation
Zhang, K. (2025) · theses.fr (ABES) · doi:10.70675/28453a20z78f2z4fa2za6edz700065e69e97
Création éthique de données textuelles artificielles : application au domaine biomédical
Hiebel, N. (2025) · theses.fr (ABES) · doi:10.70675/ed1998c5z91c4z41b2z94edz9316aa820a03
QuotAnaSum: Interactive News Summary Generation through Quotation AnalysisPreprint
Han, S., Li, G., Wang, S., Cao, P., Bai, Y., & Liu, C. H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6554165
What Stays and What Goes: Auditing the Impact of LLM Summarization on News Partisanship
Savgira, P., Kreiss, E., & Hosseinmardi, H. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799057 · Cited by 1
EchoScriptor: Automatic Lifelogging Narratives via Activity-Based Audio–Language Model
Li, K. Y., Zhou, X., Zheng, H., Shin, K. G., & Sample, A. P. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791528 · Cited by 1
“…for participant sample sizes in Human-Computer Interaction research [10], aligns with summarization benchmarks that collect eight judgments per summary [25], and supports crossed participant-by-item analysis using linear mixed-effects models, which are recommended for this type of design [9, 92].”
Kim, Y., Kim, M., Kim, S., Cho, S., & Kim, J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790971
RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing AgentsPreprint
Rosati, R., Colucci, E., Bolognini, M., Mancini, A., & Sernani, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.11655
Nellipudi, S. K. K., & Patel, N. (2026) · International Journal of Intelligent Systems and Applications in Engineering · doi:10.5281/zenodo.19644529
Николаевич, Г. В. (2026) · Программные системы и вычислительные методы · doi:10.7256/2454-0714.2026.1.78734
Liu, W., & ZHANG, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-9131850/v1
A Unified Multi-Reference Framework for Training and Evaluation in Abstractive Summarization
Rao, A. B., Aithal, S. G., & Singh, S. (2026) · IEEE Access · doi:10.1109/access.2026.3686976
“While SEAHORSE [25] offered fine-grained annotations across six dimensions in multiple languages, SummEval [26] found substantial differences between automatic metrics and human judgments.”
Badsi, H. B. A., Louhichi, S., Khaldi, T., Torkhani, J., Indiana, J., & Arfaoui, N. (2025) · International Conference on Algorithms, Computing and Systems (ICACS) · doi:10.1145/3789418.3789432
“As summarization systems increasingly integrate with real-world applications — from news aggregation to medical and legal informatics — the field continues to evolve toward generating context-aware, bias-controlled, and user-adaptive summaries [13, 40].”
LLM-ReSum: A Framework for LLM Reflective Summarization Through Self-Evaluation
Nguyen, H., Zhang, H., Zhang, Y., Chen, H., & Ding, J. (2026) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice70481.2026.00012 · Cited by 1
“Five Amazon Mechanical Turk workers and three experts rated summaries on a 5-point Likert scale across coherence, consistency, fluency, and relevance [5].”
InfluentialLi, X., Mao, Q., Sun, J., Han, Z., Shang, W., & Liu, J. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2026.3678533
“These three aspects are frequently used in the summarization community and NLG evaluation [55], [56].”
FREEsum: A Conceptual Framework for Evaluating Text Summarization Approaches
Alves, L. V., & Rocha, C. (2026) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.5753/sbsi.2026.248662
“Although there exist many libraries and tools for generating and evaluating summaries—such as SacreROUGE [Deutsch and Roth 2020], SummEval [Fabbri et al. 2021], SummerTime [Ni et al. 2021], and FineSurE [Song et al. 2024], which consolidate metrics, models, and evaluation routines and facilitate experimentation—there…”
InfluentialNEWSLYTIC: An AI-Powered News Analysis Engine
Thiyagu, T. M., Mohana Priya, M., & Lakshmi, M. (2026) · IEEE International Conference on Emerging Computing and Intelligent Technologies (ICoECIT) · doi:10.1109/icoecit68303.2026.11496850
“While current research has made significant progress in model development and evaluation, most studies focus on general news summarization or technical benchmarks [4], [5], [10], [15].”
InfluentialRaiyan, M. S., & Ferdous, N. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491391
“The summary should condense the source by including only its essential information [25].”
Can LLMs Replace Human Raters? Evaluating EFL Learners’ English Summaries
Kato, M. (2026) · International Education Studies · doi:10.5539/ies.v19n3p31
“However, these methods may fail to capture the specific aspects of summarization that human raters prioritize, such as clarity, accuracy, and comprehensiveness (Fabbri et al., 2021; Nguyen et al., 2024).”
Bhavya Sree, S., Viju, L. M., Nandhini, K., Jayapriya, J., & Vinay, M. (2026) · International Conference on Innovative Trends in Information Technology (ICITIIT) · doi:10.1109/icitiit68860.2026.11499622
“To overcome these gaps, researchers have explored combining BERTScore with factual metrics like QAGS, SummaC, and QuestEval (Fabbri et al., 2021) [9] , yet Nair et al. (2024) [8] observed that comparative evaluation rarely considers hybrid-model workflows rather than isolated model outputs.”
Identifying Algorithmic and Domain-Specific Bias in Parliamentary Debate Summarisation
Cunningham, E., Cross, J., & Greene, D. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-19108-3_28
TEBScore: Trustworthy and Explainable AI May Be Within Reach?
Siddiqui, S. M., & Madisetti, V. K. (2026) · IEEE Access · doi:10.1109/access.2026.3693072
“Setup: We begin by randomly sampling 100 observations from the SummEval dataset [26].”
Factuality of summaries from generative models in a controlled semantic context
Akani, E. (2024) · theses.fr (ABES) · doi:10.70675/163735f0z980ez40fczbf39ze3eeea7b4de1 · Cited by 1
Structure Retention in Embedding Spaces as a Predictor of Benchmark PerformancePreprint
Myntti, A., Kanerva, J., Laippala, V., & Ginter, F. (2026) · arXiv.org · doi:10.48550/arxiv.2605.22202
“For example, ARCChallenge contains question–answer pairs, SummEval document–summary pairs; we refer to these partitions as queries and targets, respectively.”
InfluentialDoan, T.-T., Mai, T.-P., Duong, Q.-H., Can, D.-C., Vuong, T.-H.-Y., & Tran, M.-V. (2026) · VNU Journal of Science Computer Science and Communication Engineering · doi:10.25073/2588-1086/vnucsce.5797
“Reference-based metrics such as ROUGE and BERTScore measure lexical and semantic overlap against a single gold reference, but do not directly capture human-perceived quality aspects such as factual faithfulness or informativeness [29].”
Towards Automating Articles Screening Processes Using Chain-of-Thought Large Language Models
Arpini, C., Cesarini, M., & Lotano, E. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3795601
“As already mentioned in [24] [13] [6], automatic evaluation of short summaries is usually computed via n-gram overlap metrics which struggle with abstractive summarization (e.g., synonyms, different sentence structures, rhetorical figures).”
A classification framework of large language model in the post-transformer era
Athiya, R., & Gupta, A. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133034
Synonym Knowledge Graph Enhanced Language Model for Inconsistent Hallucination Detection
Yang, J., Zheng, H., Wang, H., & Kim, H. (2026) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3819585
“Furthermore, the SummEval dataset [13] comprises summarizer outputs from seven extractive and 16 abstractive models.”
The Medium is the Prompt: Prompts as Design Science Artifacts
Schulte, K., Herath, S., Møller, F., & Legner, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-28316-0_18
Exploring migration research with large language model: methods and insights
Iacus, S. M., Qi, H., & Han, J. (2026) · Royal Society Open Science · doi:10.1098/rsos.251577 · Cited by 1
“Evidence indicates declining annotation quality in crowd‑sourced data [17] and low agreement with experts [18].”
Guo, R., Chang, Z., & Fu, L. (2026) · Applied Sciences · doi:10.3390/app16125780
Design of an Automated Summarization Model Using Convolutional Graph Based Network
Dineshnath, G., & Saraswathi, S. (2026) · SN Computer Science · doi:10.1007/s42979-026-05149-4
Tran, T. T. P., Vũ, T. H., Nguyen, V. V., & Nguyen, P. T. (2026) · IEEE Access · doi:10.1109/access.2026.3701952
“However, meta-evaluation studies show that many automatic metrics correlate only moderately with human judgments and may under-capture dimensions such as factual consistency, thereby motivating stronger evaluation protocols and multi-dimensional scoring [1].”
InfluentialDetect, Remask, Repair: Diffusion Editing for Faithful Summarization of Evolving Contexts
Zou, H., Horvitz, Z., Karthick, C., Yu, Z., & Mckeown, K. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.1876
Puneeth, R., & Somasekar, J. (2026) · Journal of Innovation and Technology · doi:10.61453/joit.v2026_0207
Evas: a unified deep learning framework for multi-dimensional evaluation of text summaries
G, K. K. T., Shreyas, J., Nema, V. R., Radhakrishna, S., Kashyap, K., & Koolagudi, S. G. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02827-w
When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews
Kumar, S., Kamdar, Y., Hossain, A., Kumari, B., Saikh, T., & Ekbal, A. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1908
“…peer reviews and evidence-level, graded annotations makes large-scale collection costly, and similar dataset scales are standard for tasks requiring detailed expert judgment, as seen in fine-grained evaluation benchmarks such as Sum-mEval (Fabbri et al., 2020) and Qasper (Dasigi et al., 2021).”
Contextual Diversity Measure (CDM) for Controllable Story Generation in Large Language Models
Susilo, R., Suominen, H., & Haslum, P. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-srw.98
“However, these metrics primarily measure surface-level similarity (Deutsch and Roth, 2021) rather than contextual diversity or semantic content (Mathur et al., 2020; Fabbri et al., 2021).”
Temporal and linguistic enrichment for abstractive text summarization using transformer models
Abdalgader, K., Ahmed, F. Y. H., Wassie, B., Kerala, K., Heramo, N., & Gudeta, R. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01710-5
“Among the two dominant approaches—extractive and abstractive—abstractive summarization is more challenging but also more powerful, as it generates novel sentences rather than merely selecting fragments from the source [1] [2] [3] [4].”
Lv, S., Li, Y., Chen, Y., Lin, F., & Cheng, G. (2026) · International Conference on Signal Image Processing and Communication (ICSIPC) · doi:10.1109/icsipc69751.2026.11583935
“To validate the effectiveness of the SRMEF in open-generation evaluation, as well as the alignment between the evaluator model and human preferences, this paper selects SummEval [13], a widely recognized meta-evaluation benchmark in the field of text summarization.”
Cole, C., Deschryvere, M. K. S., Ehsan, T., & Hajikhani, A. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2480-7_20
“Research indicates that larger model sizes generally produce improved performance in summarization evaluation, with stronger correlation to human judgments [6, 15].”
Hatoum, C., Combes, C., Fresse, V., Gravier, C., & Orzalesi, M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-31141-2_13
PromoCraft+: a human-in-the-loop framework for auditing AI-generated insurance promotions
Saxena, A. (2026) · Marketing Intelligence & Planning · doi:10.1108/mip-03-2025-0273
Ayalew, A. M., Seppänen, T., & Oussalah, M. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820416
Guo, X., Xie, Z., & Chen, P. H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809915
“As large language models (LLMs) have demonstrated strong performance across various applications [4, 10, 12, 14, 39], the reliability of their responses has become a concern, especially in many high-stakes scenarios [15, 17, 27].”
AutoChecklist: Automated Checklist Refinement for LLM Judges
Uniyal, M., Singh, M. P., Verbruggen, G., Le, V., & Gulwani, S. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805263
“The SummEval dataset [5] consists of news articles and summaries.”
Agnostic Multi-Source Retrieval-Augmented Generation for Documents and Database Question Answering
Adi, K. D. S., & Siregar, I. M. (2026) · Jurnal Ragam Pengabdian · doi:10.62710/qw0ytn73
“This multi-aspect approach is consistent with the SummEval methodology (Fabbri et al. 2021).”
Evaluating Summaries of Social Media Posts: Balancing Fairness for Readers and Authors
Chhikara, G., Ghosh, K., Ghosh, S., & Chakraborty, A. (2026) · ACM Transactions on the Web · doi:10.1145/3833377 · Cited by 1
Beyond accuracy: completeness and relevance metrics for evaluating the quality of long answers
Cortes, E. G., Vieira, R., & Barone, D. A. C. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-026-09936-6
Farozi, M., Stiawan, D., Abdiansah, & Budiarto, R. (2026) · International journal of intelligent engineering and systems · doi:10.22266/ijies2026.0831.05
AURA: Automated Analysis and Reporting of Therapeutically Applied Table-top Role-Playing Games
Melhárt, D., Dockhorn, A., Drachen, A., & Andersen, M. E. (2026) · International Conference on the Foundations of Digital Games (FDG) · doi:10.1145/3815598.3815645
Hallucination Is Not One Thing: A Two-Axis Taxonomy for Structured Diagnosis in Generative AI
Khan, M., Billion-Polak, P., & Khoshgoftaar, T. M. (2026) · IEEE Access · doi:10.1109/access.2026.3718830
“…[16] Intrinsic-Factuality → Internal fact alignment verifiers integrated into truth-preserving summarization models or editorial auditing pipelines [36] Extrinsic-Faithfulness → Prompt constraint adherence analyzers, including Chain-of-Thought (CoT) Authorized licensed use limited to the terms of…”
Spatial Visual Analytics for Multi-Document Summary VerificationPreprint
Xu, J., Liu, W., Liu, Y., Krokos, E., Whitley, K. N., Wang, X., Faust, R., & North, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.28853
“In this section, we illustrate how SVS supports the relevance task of assessing summary coverage over a source collection [15].”
Metadata
- Authors
- Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D.
- Affiliations
- Yale University (US)
- Salesforce (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 391-409
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3045321166
- Cited by
- 587
- References
- 102
- Citations by year
- 2026: 109 · 2025: 93 · 2024: 74 · 2023: 166 · 2022: 84 · 2021: 57
- FWCI
- 33.37
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Social Sciences › Linguistics and Language › Text Readability and Simplification
- Keywords
- text summarization, summarization evaluation, automatic evaluation metrics, crowdsourced annotation, evaluation toolkit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.