SummEval: Re-evaluating Summarization Evaluation
Open accessThe DOI matches its Crossref record; 99 of the 102 references it cites were found in a registry or an index.
Abstract
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SummEval (Fabbri et al., 2021) is a news domain dataset, where the task is abstractive summarization.The outputs are systemgenerated summaries of CNN/Daily Mail articles, with human ratings provided on coherence, consistency, fluency, and relevance.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Human evaluation has long been considered the gold standard for evaluating the quality of natural language generation (NLG) systems (Belz and Reiter, 2006;Lai and Tetreault, 2018;Fabbri et al., 2021).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Fabbri et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indirectly, these metrics also evaluate composition, since it has been shown that ROUGE and BLEU correlate with human judgments on fluency of a text [67].”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“Data in AG-GREFACT comes from 9 factual consistency evaluation datasets on CNN or XSum, including widely used ones such as SummaC (Laban et al., 2022), FRANK (Pagnoni et al., 2021), and SummEval (Fabbri et al., 2021).”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Fabbri et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Recent advances in pretrained language models, such as BERT (Devlin et al., 2019), have motivated a corresponding shift to pretraining methods in summarization (Liu and Lapata, 2019; Zhang et al.”
BackgroundMethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“ROUGE-WE (Ng and Abrecht, 2015) extends ROUGE by using soft lexical matching based on the cosine similarity of Word2Vec (Mikolov et al., 2013) embeddings.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“However, it has recently been shown that the mentioned datasets contain human judgments for model outputs scoring on a lower scale compared to current summarization systems putting into question the true performance of those metrics in the new setting (Peyrard, 2019).”
BackgroundInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani ∗ Equal contributions from authors et al., 2017) as well as the availability of largescale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et…”
BackgroundProceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“∗Equal contributions from authors The field has benefited from advances in neural network architectures (Sutskever et al., 2014; Bahdanau et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) as well as the availability of large-scale datasets (Sandhaus, 2008; Hermann et al.”
BackgroundAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Research works printed but not linked: 2
- Dang, 2009, Overview of the TAC 2009 summarization track
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
Cited by
Items 701–762 of 762
Atwine, M. T., Kintu, T. M., Lujumba, I., Ibrahim, M., Muwonge, K., Muwonge, L., Nakiire, L., Ntono, V., Nsubuga, M., Galiwango, R. M., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-10357473/v1
Grading with LLMs: A Case Study in Chile
Yavas, C. E., Mead, N. R., Chen, L., Kadlec, C., & Ordenes, P. (2026) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc69745.2026.11631330
Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streams
Lee, Y., Lim, Y., Jung, W., Choi, W., & Yoon, S. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817994
Mitigating Scoring Bias in LLM-as-a-Judge via Random Number GenerationPreprint
Asato, Y., Shirai, K., & Kertkeidkachorn, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.05726
“SummEval [4] is a benchmark that comprises an original article in CNN and DailyMail [9], its corresponding summary, and a score on a five-point Likert scale (1–5) for coherence, consistency, fluency, and relevance.”
CMA-ES Decoding Optimization for MedGemma: A Flat Landscape on Chest X-Ray Reports
Chen-Wang, M., Abreu-Cardenas, M., & Ramírez, S. C. (2026) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3795101.3814726
“This concept is related to self-consistency in LLM reasoning [22] and has been used to assess generation reliability in text summarization [6].”
Jabur, A. J., Dakhil, A. A. A., Mohammed, I. S., & Ajel, B. N. (2026) · Iraqi Journal for Computers and Informatics · doi:10.25195/ijci.v52i2.865
“In the corpus, the most standardised multi-system human evaluation is in SummEval [23 ROUGE is retained as the primary comparative metric because it is the only measure reported consistently across all reviewed systems (see Table 4) and does not systematically favour any particular approach.”
InfluentialInformation Satisfaction: A Reader-Centered Axis for Summarization Evaluation
Cachola, I. A., Walden, W. G., Kriz, R., & Dredze, M. H. (2026) · ACM Conference on Human-AI Complementarity and Alignment · doi:10.1145/3834580.3838749
Task-Specific QLoRA Adaptation in RAG-Based Information Extraction from Research Articles
Dikilitaş, Y., Sayar, A., & Göz, F. (2026) · Applied Sciences · doi:10.3390/app16168294
EduAssist: Evaluating a Locally Deployed Large Language Model for Educational Document Summarization
Afzal, M., & Tahreem, S. (2026) · International Journal of Engineering Research and Science & Technology · doi:10.62643/ijerst.2026.v22.n3.4484
Song, M., Kim, C., Eo, S., & Park, C. (2026) · arXiv · doi:10.48550/arxiv.2608.30373
“Gemma3-27B also shows a better RMSE under Symmetric MAD on Korean Essay (0.512 vs. 0.528), although Single Judge retains an advantage on Spearman correlation and on Sum-mEval.”
InfluentialTowards a Reliable and Practical Eval PipelinePreprint
Nguyen, E. T., & Ghose, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.00805
“Dataset and LLMs : The SummEval [Fabbri et al., 2021] dataset (MIT License) is used for experiments.”
Baumgärtel, F., Bono, E., Fillinger, L., Galou, L., Kęska, K., Walter, S. M., Andorfer, P., Kratochwill, K., Perco, P., & Ley, M. (2026) · iScience · doi:10.1016/j.isci.2026.117438
Kong, D. (2026) · International Conference on Cyber Security, Artificial Intelligence and Digital Economy (CSAIDE) · doi:10.1145/3819836.3819961
“In recent years, the evaluation of machine translation and generative artificial intelligence systems has progressively shifted toward hybrid models that combine automatic metrics with human review, aiming to enhance semantic coherence and pragmatic adequacy in generated texts [9].”
Багатовимірна оцінка використання великих мовних моделей у задачах узагальнення текстів
(2026) · Ìnformatika, kulʹtura, tehnìka. · doi:10.15276/ict.03.2026.10
Stakeholder-Adaptive Safety Report Generation Using RAG and GSN
Kato, H., Hayashi, M., Tsuchiya, T., Yoda, M., & Matsuno, Y. (2026) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re68928.2026.00047
“Generation quality is assessed with an LLM-as-a-Judge protocol [18], using criteria from SummEval [19], RAGAs [20], and APPLS [21].”
The Core Levers of LLM Evaluation: Sets, Templates, and Raters
Robsky, A., Lavitas, L., & Wang, Y. (2026) · Springer eBooks · doi:10.1007/978-3-032-26749-8_3
Kamińska, A. M. (2026) · Zagadnienia Informacji Naukowej - Studia Informacyjne · doi:10.36702/zin2026.01.02
Bino, D., & Shrivastava, M. (2026) · MEC International Conference on Big Data and Smart City (ICBDSC) · doi:10.1109/icadaai64945.2026.11679237
“SummEval [19] establishes that lexical and semantic evaluation metrics identify different model rankings, a finding our oracle flip-rate of 71.4% corroborates at the dataset level.”
InfluentialCan We Trust the Judges? Validation of Factuality Evaluation Methods via Answer PerturbationPreprint
Gharsallah, S., Robaldo, A., Tokareva, M., Pinheiro, G. G., Guendouz, I., Troncy, R., Papotti, P., & Michiardi, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.15561
“Benchmarks such as SummEval [Fabbri et al., 2021], FRANK [Pagnoni et al., 2021], and TruthfulQA [Lin et al., 2022] provide expert-annotated judgments of consistency and hallucination, forming the basis for validating automatic metrics.”
Kharisma, I. L., Pasya, H. G., & Sujjada, A. (2026) · Jurnal Teknik Informatika (Jutif) · doi:10.52436/1.jutif.2026.7.4.5562
Towards reliable LLM evaluators: Discovering and internalizing explicit scoring logic
Meng, J., Mao, J., Ma, M., & Li, G. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.105154
Медианные значения критериев качества генерации обзоров при сравнении с альтернативными подходами
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.171.79.4
Кузнецов, И. (2026) · Международный научно-исследовательский журнал · doi:10.60797/irj.2026.171.79
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.171.79.1
Основные параметры конфигурации системы при проведении эксперимента
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.171.79.2
Медианные значения критериев качества генерации обзоров для рассматриваемых моделей и подходов
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.171.79.3
Multi-document text summarization techniques applications, challenges and future directions
Khan, R., Khan, S. S., Khan, S., Khan, M., & Danyal, M. M. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134341
Shahabi, A. S., Kangavari, M. R., & Rahmani, A. M. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.105152
Serikov, A., Biloshchytskyi, A., Mukhatayev, A., & Zheldybayeva, B. (2026) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2026.16.9.2697
“It identifies consistency (faithfulness), coherence, fluency, and relevance for assessing generated text quality [29].”
Development of Trusted AI for Effective Digital Government Document Summarization
Sampath, D., Nivisha, S., Kharnika, G. R., & Prasanna, M. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9129-8.ch012
Budget-Constrained Preference-Aligned Prompt Optimization for Controllable Summarization
Li, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-5693-8_21
Coherence-Aware Distributional Evaluation of Open-Ended Text GenerationPreprint
Liu, J., Zhu, J., Jiang, W., Liu, H., & Wen, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.34240
“We adapt the released SummEval (Fabbri et al., 2021) coherence rubric from G-Eval (Liu et al., 2023) to standalone passages.”
Entity and Aspect-Aware Summarization of Indonesian News with Trade-off Analysis
Clairine, G., Widjaja, G., Lucky, H., Tedjasulaksana, J. J., & Pradana, R. C. (2026) · International Electronics Symposium (IES) · doi:10.1109/ies71033.2026.11702588
Nayeem, M. T., Chakraborty, S., & Rafiei, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.36218
Vera, L., Craveiro, O., Malheiro, R., Dias, M., & Correia, R. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100305
PENGEMBANGAN SISTEM INFORMASI PERINGKASAN OTOMATIS DOKUMEN KHUTBAH JUM’AT BERBASIS INDOBERT
Rahmiasari, S. A., Supriyono, & Azis, O. Q. (2026) · Prosiding Seminar Nasional Teknologi dan Sistem Informasi · doi:10.33005/sitasi.v6i1.3050
Figure-Level Substitutability in Scientific Document Summarization
Bodnár, A., Gulyás, L., & Kárász, Z. (2026) · Intelligent Robotics FAIR · doi:10.1145/3831600.3831605
“Prior summarization research shows that different automatic metrics capture different aspects of output quality and may behave inconsistent with one another [2].”
Ristianto, R. P., Arisal, A., Kurniasari, D., & Khotimah, P. H. (2026) · International Conference on Computer, Control, Informatics and its Applications · doi:10.1109/ic3ina72033.2026.11714656
“ROUGE [7] measures n-gram and longest-common-subsequence overlap against reference summaries and remains the field standard, but correlates imperfectly with human judgment, particularly when system output diverges stylistically from the reference [15].”
Inside summary evaluation: Understanding human and GPT judgments to improve LLM-based evaluation
Sadeghi, Z., Milios, E., & Rudzicz, F. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200738
Fan, H., Yang, Y., Luo, S., & Lin, L. (2026) · Scientific Reports · doi:10.1038/s41598-026-74183-2
Jiang, B., Rim, K., Lynch, K., & Pustejovsky, J. D. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846037
“Overlap correlates only weakly with human judgments of quality [13], and it misjudges content in both directions, rewarding a summary that reuses the reference wording while negating a central fact and penalizing a faithful paraphrase.”
Huang, L., Ren, H., Cao, X., & Zhang, Y. (2026) · Scientometrics · doi:10.1007/s11192-026-05826-4
Ouverney, T., & Pereira, V. (2026) · Anais do Symposium on Knowledge Discovery, Mining and Learning (KDMiLe) · doi:10.5753/kdmile.2026.27518
“The experiments use the SummEval benchmark [Fabbri et al. 2021], which contains 100 news articles, 16 summarization systems, and 1600 document-summary pairs.”
InfluentialA Comparison of Commonly Used Automatic Evaluation Metrics for Open-Ended Tasks in Portuguese
Faé, E., Balreira, D. G., & Moreira, V. P. (2026) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2026.26578
“Recent studies still show medium to low correlations between automated metrics and human judgments [Deutsch et al. 2021, Fabbri et al. 2021, Deutsch et al. 2022, Liu et al. 2023, Junqueira and Moreira 2026].”
InfluentialMESO: Memory-Efficient Speculative Offloading for Large Language Model Inference
Zhang, Q., Li, X., Ji, B., Liu, X., Yu, J., Peng, L., & Xu, H. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729215
“HumanEval [27] Coding 164 problems Summeval [28] Summarization 100 articles Samsum [29] Summarization ˜16 k dialogues length.”
Granularity is the Hidden Bias: Revisiting the Factual Consistency Evaluation for Summarization
Ma, W., Gao, J., Nie, Z., Yang, Z., Tong, C., Guo, C., & Jiang, L. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728799
“SummEval [12] provides human annotations for summaries generated by both extractive and abstractive systems and is commonly used to assess alignment between automatic metrics and human factuality judgments.”
GardenBench: A lightweight, daily evaluation of LLM capabilitiesPreprint
Cummins-Mburu, B., Mejia, M. G., Heath, O., Kreyzerman, S., Murray, E., Surendiran, A., & Alexander, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.23866
AlMazrouii, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10939304/v1
AlignLLM: addressing short answer limitations in LLM-as-a-judge systems.
Abeyratne, R., Wiratunga, N., Martin, K., & Nkisi-Orji, I. (2026) · Open Access Institutional Repository at Robert Gordon University (Robert Gordon University)
On Multilingual Text Summarization: A Review of Model Architectures, Techniques, and Research Gaps
Muia, M., Oirere, A. M., & Wanjau, S. K. (2026) · International Journal of Data Informatics and Intelligent Computing · doi:10.59461/ijdiic.v5i4.311
“…explicitly acknowledged that ROUGE scores exhibit reduced sensitivity to lexical and morphological variation across languages, a limitation reported in [35], whereby n-gram overlap-based scoring may inadequately capture summary quality in morphologically rich or agglutinative target languages.”
Трёхфакторная диагностическая оценка качества машинных рефератов научных текстов на русском языке
Николаевич, Г. В. (n.d.) · CyberLeninK (CyberLeninka)
A Hausa abstractive summarization dataset with baseline evaluation
Salisu, B. A., Bichi, A. A., ADAMU, A., & Abdulrahman, S. M. (2026) · Journal of Statistical Sciences and Computational Intelligence · doi:10.64497/jssci.339
“To assess the quality of the reference summaries in HAUSUMM, a human evaluation was conducted using multidimensional summarization evaluation criteria adapted from previous summarization research [25].”
Rahmiasari, S. A., Supriyono, S., & Azis, O. Q. (2026) · Discover Informatics · doi:10.1007/s44564-026-00032-z
“…on established automated metrics: ROUGE [12] measures n-gram overlap and remains the standard benchmark metric, BERTScore [13] provides semantic similarity assessment beyond surface-level overlap, and SummEval [41] offers a multi-dimensional evaluation framework validated against human judgments.”
Consistency and Scoring Behavior of LLM-Based Evaluators for English and Spanish Text Summarization
Kuş, A., & Campo, M. J. T. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11728178
“However, previous studies have shown that these metrics often fail to fully capture semantic similarity and factual correctness, particularly in abstractive summarization tasks [1], [9].”
Concept-Based Tabulation and Summarization for Automated Note Making from Biology Textbooks
Nithyakalyani, A. (2026) · Journal of Intelligent Decision Making and Information Science · doi:10.17762/jidmis.v3.5633
Predicting Human-Gain Curves under Local Proxy Optimization from Repeated RatingsPreprint
Choi, M., Nam, K., & Lee, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.41114
Pribisalić, M., & Martinčić-Ipšić, S. (2026) · FinTech · doi:10.3390/fintech5040101
A Contrastive Framework for Enhanced Automatic Attribution Evaluation Through Error Generation
Djeddal, H., Pinel-Sauvagnat, K., Soulier, L., Katrenko, S., & Tamine, L. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3840754
LLM as a Judge on Summary Evaluation
Hernández-Del-Toro, T., Salado-Mejía, M., & Saurio, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-44176-8_32
VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge AggregationPreprint
Shin, C., & Sala, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.00491
Avula, S. R., Bhat, S. S., Haarika, A., & Pattanashetti, S. (2026) · doi:10.1109/nmitcon70171.2026.11754495
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · arXiv · doi:10.48550/arxiv.2301.13298 · Cited by 3
“Our paper is limited to faithfulness evaluation , but summaries are typically evaluated for salience, fluency, coherence as well (Fabbri et al., 2021).”
BackgroundMethodInfluential
Metadata
- Authors
- Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D.
- Affiliations
- Yale University (US)
- Salesforce (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 391-409
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3045321166
- Cited by
- 587
- References
- 102
- Citations by year
- 2026: 109 · 2025: 93 · 2024: 74 · 2023: 166 · 2022: 84 · 2021: 57
- FWCI
- 33.37
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Social Sciences › Linguistics and Language › Text Readability and Simplification
- Keywords
- text summarization, summarization evaluation, automatic evaluation metrics, crowdsourced annotation, evaluation toolkit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.