The limits of automatic summarisation according to ROUGE
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
This paper discusses some central caveats of summarisation, incurred in the use of the ROUGE metric for evaluation, with respect to optimal solutions.The task is NPhard, of which we give the first proof.Still, as we show empirically for three central benchmark datasets for the task, greedy algorithms empirically seem to perform optimally according to the metric.Additionally, overall quality assurance is problematic: there is no natural upper bound on the quality of summarisation systems, and even humans are excluded from performing optimal summarisation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1979) · Cited by 43,841
(2004) · Cited by 9,718
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656 · cited 1×
“There have been recent advances in supervised summarisation mainly with respect to supervised learning using neural networks (for example (Rush et al., 2015; Chopra et al., 2016)).”
BackgroundAutomatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 1×
“We know that 100% perfect scores are impossible, so what is a perfect score according to ROUGE? How do we know when no improvement is possible? Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; L…”
BackgroundAbstractive Sentence Summarization with Attentive Recurrent Neural Networks
Chopra, S., Auli, M., & Rush, A. M. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1012 · Cited by 881 · cited 3×
“, 2015; Chopra et al., 2016)). However, due to data size requirements, these systems are constrained to title generation systems and therefore not in the scope of this work. Hong et al. (2014) survey the state-of-the-art using the central DUC 2004 dataset.”
BackgroundEvaluating Content Selection in Summarization: The Pyramid Method
Nenkova, A., & Passonneau, R. (2004) · Columbia University · doi:10.7916/d80r9xvd · Cited by 527 · cited 2×
“The metric returns an average of ROUGE scores over multiple reference summaries in order to avoid bias (Nenkova and Passonneau, 2004).”
BackgroundA scalable global model for summarization
Gillick, D., & Favre, B. (2009) · Proceedings of the Workshop on Integer Linear Programming for Natural Langauge Processing - ILP '09 · doi:10.3115/1611638.1611640 · Cited by 280 · cited 1×
“Of these, ICSISum (Gillick and Favre, 2009) is the only global summariser using an exact algorithm; it obtains the best ROUGE-2 score without supervision.”
MethodRe-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE
Graham, Y. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1013 · Cited by 143 · cited 1×
“Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; Lin and Hovy, 2003; Liu and Liu, 2008; Graham, 2015).”
BackgroundA Repository of State of the Art and Competitive Baseline Summaries for Generic News Summarization
Hong, K., Conroy, J. M., Favre, B., Kulesza, A., Lin, H. Z., & Nenkova, A. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4aa4e4yajcec · Cited by 122 · cited 2×
“…summarisation, recent research over conventional benchmark datasets has suggested the following: according to the most widely accepted automatic evaluation metric, ROUGE, there has been no substantial improvement in performance on central datasets in the field in the last decade (Hong et al., 2014).”
BackgroundMethodInfluentialCorrelation between ROUGE and human evaluation of extractive meeting summaries
Liu, F., & Liu, Y. (2008) · Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies Short Papers - HLT '08 · doi:10.3115/1557690.1557747 · Cited by 81 · cited 1×
“Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; Lin and Hovy, 2003; Liu and Liu, 2008; Graham, 2015).”
Background
Cited by
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207
“Human Evaluation In addition to automatic evaluation using ROUGE which can be misleading when used as the only means to assess the informativeness of summaries (Schluter, 2017), we also evaluated system output by eliciting human judgments in two ways.”
MethodOn Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185
“ness of summaries (Schluter, 2017).”
MethodLearning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
“Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE [39], have received criticism for poor correlation with human judgments [55, 45, 6, 33].”
BackgroundRanking Sentences for Extractive Summarization with Reinforcement Learning
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1158 · Cited by 688
“In addition to ROUGE which can be misleading when used as the only means to assess the informativeness of summaries (Schluter, 2017), we also evaluated system output by eliciting human judgments in two ways.”
MethodTextual keyword extraction and summarization: State-of-the-art
Nasar, Z., Jaffry, S. W., & Malik, M. K. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.102088 · Cited by 113
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92
“Automated metrics like BLEU and ROUGE [99] often do not correlate with human judgment [103, 164, 166, 177], making them unreliable for evaluation and training.”
The challenging task of summary evaluation: an overview
Lloret, E., Plaza, L., & Aker, A. (2017) · Language Resources and Evaluation · doi:10.1007/s10579-017-9399-2 · Cited by 98
“In the same line, Schluter (2017) shows that, according to ROUGE, there has been no substantial improvement in performance of summarization systems in the last decade, stressing the fact that perfect scores of ROUGE are impossible to reach even for humans.”
BackgroundImproving Abstractive Dialogue Summarization with Graph Structures and Topic Words
Zhao, L., Xu, W., & Guo, J. H. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.39 · Cited by 71
“Since the ROUGE score often fails to quantify the machine generated summaries (Schluter, 2017), we focus on evaluating the relevance and readability of each summary.”
BackgroundA Comprehensive Survey of Abstractive Text Summarization Based on Deep Learning
Zhang, M., Zhou, G., Yu, W., Huang, N., & Liu, W. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/7132226 · Cited by 102
“e main limitation of the ROUGE metric is that it is coherence-insensitive [118].”
BackgroundROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics
Zhang, M., Li, C., Wan, M., Zhang, X., & Zhao, Q. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121364 · Cited by 77
Learning towards Abstractive Timeline Summarization
Chen, X., Chan, Z., Gao, S., Yu, M.-H., Zhao, D., & Yan, R. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/686 · Cited by 49
“[Schluter, 2017] notes that only using the ROUGE metric to evaluate summarization quality can be misleading.”
BackgroundHTSS: A novel hybrid text summarisation and simplification architecture
Zaman, F., Shardlow, M., Hassan, S., Aljohani, N. R., & Nawaz, R. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102351 · Cited by 64
Searle, T., Ibrahim, Z. M., Teo, J., & Dobson, R. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104358 · Cited by 83
“ROUGE has been criticised in the literature as summarisation quality can score highly whilst perform poorly during manual evaluation(Schluter, 2017).”
BackgroundMethodInfluentialBridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 133
“However, these metrics often fail to correlate with human-perceived text quality, particularly when models are overly optimized for these metrics (Schluter, 2017; Mathur et al., 2020; Gehrmann et al., 2022; Paulus et al., 2017; Amrhein and Sennrich, 2022).”
West, P., Holtzman, A., Buys, J., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1389 · Cited by 72
“This is both to combat known issues with ROUGE metrics (Schluter, 2017) and to experiment beyond limited supervised domains.”
BackgroundIterative Document Representation Learning Towards Summarization with Polishing
Chen, X., Gao, S., Tao, C., Song, Y., Zhao, D., & Yan, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1442 · Cited by 45
“Schluter (2017) noted that only using the Rouge metric to evaluate summarization quality can be misleading.”
BackgroundAutomating Question Generation From Educational Text
Bhowmick, A. K., Jagmohan, A., Vempaty, A., Dey, P., Hall, L., Hartman, J., Kokku, R., & Maheshwari, H. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47994-6_38 · Cited by 46
HighRES: Highlight-based Reference-less Evaluation of Summarization
Hardy, Narayan, S., & Vlachos, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1330 · Cited by 35
“measure performance in summarization (Schluter, 2017), also known for other tasks in which the goal is to generate natural language (Novikova et al.”
BackgroundMethodIndian Legal Text Summarization: A Text Normalization-based Approach
Ghosh, S., Dutta, M., & Das, T. (2022) · IEEE India Conference · doi:10.1109/indicon56171.2022.10039891 · Cited by 44
“The tool is evaluated using RecallOriented Understudy for Gisting Evaluation (ROUGE) and human scoring.”
MethodInfluentialA Survey of Text Summarization Approaches Based on Deep Learning
Hou, S., Huang, X., Fei, C., Zhang, S., Li, Y., Sun, Q., & Wang, C. (2021) · Journal of Computer Science and Technology · doi:10.1007/s11390-020-0207-x · Cited by 36
“One of the limitations of the ROUGE metric is coherence-insensitive ([122]).”
BackgroundGao, S., Chen, X., Li, P., Chan, Z., Zhao, D., & Yan, R. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1388 · Cited by 30
“(Schluter, 2017) notes that only using the ROUGE metric to evaluate summarization quality can be misleading.”
MethodImproving Latent Alignment in Text Summarization by Generalizing the Pointer Generator
Shen, X., Zhao, Y., Su, H., & Klakow, D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1390 · Cited by 35
“use the non-anonymized version from See et al. (2017)).”
BackgroundA comprehensive review of the video-to-text problem
Perez-Martin, J., Bustos, B., Guimarães, S. J. F., Sipiran, I., Pérez, J. E. P., & Said, G. C. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10104-1 · Cited by 43
“In this sense, BLEU and ROUGE are not optimal (Reiter 2018; Schluter 2017) and have been shown to correlate with humans weakly (Vedantam et al.”
BackgroundA Data-driven Latent Semantic Analysis for Automatic Text Summarization using LDA Topic Modelling
Onah, D. F. O., Pang, E. L. L., & El‐Haj, M. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020259 · Cited by 30
“ROUGE has measures that allow for the evaluation of the accuracy of system summary as compared to a human created summary known as the model summary [32], [33].”
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way
Wang, A., Pang, R. Y., Chen, A., Phang, J., & Bowman, S. R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.75 · Cited by 96
What is this Article about? Extreme Summarization with Topic-aware Convolutional Neural Networks
Narayan, S., Cohen, S. B., & Lapata, M. (2019) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11315 · Cited by 19
Target-aware Abstractive Related Work Generation with Contrastive Learning
Chen, X., Alamro, H., Li, M., Gao, S., Yan, R., Gao, X., & Zhang, X. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532065 · Cited by 44
“Schluter [32] notes that only using the ROUGEmetric to evaluate generation quality can be misleading.”
Fact-based Content Weighting for Evaluating Abstractive Summarisation
Xu, X., Dušek, O., Li, J., Rieser, V., & Konstas, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.455 · Cited by 18
“Evaluating abstractive summarisation remains an open challenge (Schluter, 2017; Kryściński et al., 2019): First, decoders are amenable to pathogeniessuch as hallucination and/or omission of important information, which are hard to capture using existing evaluation metrics (Cao et al., 2018;…”
MethodEstimating redundancy in clinical text
Searle, T., Ibrahim, Z. M., Teo, J., & Dobson, R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103938 · Cited by 32
“Interestingly, recall and precision are equivalent for ROUGE and BERTScore.”
BackgroundInfluentialOctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“…task via code execution and measure pass@ k (Chen et al., 2021) instead of evaluating the explanation itself using heuristic-based metrics like BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) which have major limitations (Reiter, 2018; Schluter, 2017; Eghbali & Pradel, 2022; Zhou et al., 2023b).”
MethodBidirectional LSTM Networks for Abstractive Text Summarization
Kovačević, A., & Кеčo, D. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-90055-7_21 · Cited by 17
Hierarchical Sliding Inference Generator for Question-driven Abstractive Answer Summarization
Li, B., Yang, P., Zhao, H., Zhang, P., & Liu, Z. (2022) · ACM Transactions on Information Systems · doi:10.1145/3511891 · Cited by 19
“The specific calculation method is as follows [49]:”
MethodDialSummEval: Revisiting Summarization Evaluation for Dialogues
Gao, M., & Wan, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.418 · Cited by 55
Clustering-based topic modeling for biomedical documents extractive text summarization
AbdelAziz, N. M., Ali, A. A., Naguib, S. M., & Fayed, L. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06640-6 · Cited by 24
Li, M., Chen, X., Yang, M., Gao, S., Zhao, D., & Yan, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17565 · Cited by 17
Learning by Semantic Similarity Makes Abstractive Summarization BetterPreprint
Yoon, W., Yeo, Y. S., Jeong, M., Yi, B.-J., & Kang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07767 · Cited by 27
“rics are known to be have high correlation with human judgement, they have limitations and may mislead the actual quality of a generated text (Schluter, 2017; Maynez et al., 2020; Gehrmann et al., 2021).”
BackgroundTowards Clinical Encounter Summarization: Learning to Compose Discharge Summaries from Prior NotesPreprint
Shing, H.-C., Shivade, C., Pourdamghani, N., Nan, F., Resnik, P., Oard, D. W., & Bhatia, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13498 · Cited by 48
Facet-Aware Evaluation for Extractive Summarization
Mao, Y., Liu, L., Zhu, Q., Ren, X., & Han, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.445 · Cited by 24
“However, recent studies (Paulus et al., 2017; Schluter, 2017; Kryscinski et al., 2019) reveal the limitations of ROUGE and find that in many cases, it fails to reach consensus with human judgment.”
BackgroundInfluentialA qualitative analysis of themes in instant messaging communication of software developers
Silva, C. C., Galster, M., & Gilson, F. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111397 · Cited by 16
Topic-sensitive neural headline generation
Yana, A., Wang, Z., Xu, L., Liu, Z., & Sun, M. (2020) · Science China Information Sciences · doi:10.1007/s11432-019-2657-8 · Cited by 12
“The question-answering (QA) based method was adopted to evaluate the ability of the system to capture salient information for a long time [23,24,26–28].”
MethodEngineering Document Summarization: A Bidirectional Language Model-Based Approach
Qiu, Y., & Jin, Y. (2022) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4054203 · Cited by 15
“Researchers doubted that ROUGE could be mis-leading when it is the only method used for evaluation [64].”
Jointly Extracting and Compressing Documents with Summary State Representations
Mendes, A., Narayan, S., Miranda, S., Marinho, Z., Martins, A. F. T., & Cohen, S. B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1397 · Cited by 60
“In addition to ROUGE, which can be misleading when used as the only means to assess summaries (Schluter, 2017), we also conducted a question-answering based human evaluation to assess the informativeness of our summaries in their ability to preserve key information from the document (Narayan et al.”
MethodTowards Summarization for Social Media - Results of the TL;DR Challenge
Syed, S., Völske, M., Lipka, N., Stein, B., Schütze, H., & Potthast, M. (2019) · International Conference on Natural Language Generation · doi:10.18653/v1/w19-8666 · Cited by 11
“Furthermore, a key issue of ROUGE is that it does not provide any upper bounds for the quality of a summarization system (Schluter, 2017), thus warranting an extensive manual evaluation of the systems.”
BackgroundA Topic-aware Summarization Framework with Different Modal Side Information
Chen, X., Li, M., Gao, S., Cheng, X., Yang, Q., Zhang, Q., Gao, X., & Zhang, X. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591630 · Cited by 16
“Schluter [38] noted that only using the ROUGEmetric to evaluate generation quality can be misleading.”
MethodSteen, J., & Markert, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.160 · Cited by 10
WBI at MEDIQA 2021: Summarizing Consumer Health Questions with Generative Transformers
Sänger, M., Weber, L., & Leser, U. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.9 · Cited by 12
“handling of synonyms, abbreviations or enumerations, must be taken into account in the result interpretation (Schluter, 2017; Kané et al., 2019).”
BackgroundSclar, M., West, P., Kumar, S., Tsvetkov, Y., & Choi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.655 · Cited by 25
Sezgın, E., Sirrianni, J., & Kranz, K. (2023) · medRxiv · doi:10.1101/2023.12.06.23299573 · Cited by 10
SCAR: Sentence Compression using Autoencoders for Reconstruction
Malireddy, C., Maniar, T., & Shrivastava, M. (2020) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2020.acl-srw.13 · Cited by 15
“ROUGE only measures the content overlap and does not account for coherence.”
MethodInfluentialGenerating EDU Extracts for Plan-Guided Summary Re-Ranking
Adams, G., Fabbri, A. R., Ladhak, F., Elhadad, N., & McKeown, K. R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.151 · Cited by 18
Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative FlowPreprint
Gabriel, S., Bosselut, A., Holtzman, A., Lo, K., Çelikyılmaz, A., & Choi, Y. (2019) · arXiv (Cornell University) · Cited by 9
A fine-tuned multimodal large model for power defect image-text question-answering
Wang, Q., Zhang, J., Du, J., Zhang, K., Li, R., Zhao, F., Zou, L., & Xie, C. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03539-w · Cited by 10
“Common metrics include BLEU [44], ROUGE [45], and BERTScore [46], which are usually based on the similarity between the test set and the real answers.”
DaNewsroom: A Large-scale Danish Summarisation Dataset
Varab, D., & Schluter, N. (2020) · IT University Of Copenhagen (IT University of Copenhagen) · Cited by 7
Self-citationPhani, S., Abdul, A., Mudigonda, K. S. P., & Vemula, D. R. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.04.031 · Cited by 9
Quality Evaluation of Summarization Models for Patent Documents
Ding, J., Chen, H., Kolapudi, S. T., Pobbathi, L., & Nguyen, H. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00033 · Cited by 9
“However, studies also showed that BLEU, ROUGE, and BERTScore are not sufficient for the evaluation of summaries [32].”
Advances in Preference-based Reinforcement Learning: A Review
Abdelkareem, Y., Shehata, S., & Karray, F. O. (2022) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53654.2022.9945333 · Cited by 24
“However, the rewards used in those methods are mainly based on ROUGE scores which do not directly correlate with the quality of the summaries as judged by humans [49].”
MethodFrom task to evaluation: an automatic text summarization review
Lu, L., Liu, Y., Xu, W., Li, H., & Sun, G. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10582-5 · Cited by 10
“Undeniably, ROUGE is a convenient and fast evaluation method, it still has a significant disadvantage cause misleading (Schluter 2017) and be pointed out cannot accurately estimate the performance when used alone (Ribeiro et al. 2020).”
MethodThil, L.-A., Popa, M. C., & Spanakis, G. (2024) · ACM Symposium on Applied Computing · doi:10.1145/3605098.3635903 · Cited by 20
“• They do not account for the temporal order and structural dependencies in action sequences [26].”
BackgroundSuri, K., Saha, S., & Singh, A. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.50 · Cited by 3
“because ROUGE Score always favours more extended conversations over shorter ones (Schluter, 2017).”
BackgroundWRF: Weighted Rouge-F1 Metric for Entity Recognition
Weber, L., Ramalingam, K. J., Beyer, M., & Zimmermann, A. P. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.1 · Cited by 4
SATS: simplification aware text summarization of scientific documents
Zaman, F., Kamiran, F., Shardlow, M., Hassan, S., Karim, A., & Aljohani, N. R. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1375419 · Cited by 8
“For the evaluation of summarization systems, the ROUGE metric (Lin and Hovy, 2003) is used widely; however, due to the limitations of the existing ROUGE metric, it must be addressed carefully (Schluter, 2017).”
MethodDolomites : Domain-Specific Long-Form Methodical Tasks
Malaviya, C., Agrawal, P., Ganchev, K., Srinivasan, P., Huot, F., Berant, J., Yatskar, M., Das, D., Lapata, M., & Alberti, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00727 · Cited by 12
“We release our dataset at https: //dolomites-benchmark.github.io .”
HonestBait: Forward References for Attractive but Faithful Headline Generation
Chen, C. Y., Wu, D., & Ku, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.296 · Cited by 7
Symbolic and Statistical Learning Approaches to Speech Summarization: A Scoping Review
Rezazadegan, D., Berkovsky, S., Quiroz, J. C., Kocaballi, A. B., Wang, Y., Laranjo, L., & Coiera, E. W. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101305 · Cited by 19
“Furthermore, obtaining perfect ROUGE scores may be impossible even for humans (Schluter, 2017).”
BackgroundEvaluation of Summarization Systems across Gender, Age, and Race
Jørgensen, A., & Søgaard, A. (2021) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2021.newsum-1.6 · Cited by 5
SFERANET: Automatic Generation of Football Highlights
Scotti, V., Sbattella, L., & Tedesco, R. (2019) · International Conference on Computer Science, Engineering and Information Technology (CSEIT) · doi:10.5121/csit.2019.91309 · Cited by 4
Repurposing Decoder-Transformer Language Models for Abstractive SummarizationPreprint
de Oliveira, L., & Rodrigo, A. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00325 · Cited by 3
“Many (Schluter (2017) and Lloret et al. (2018), among others) have pointed out deficiencies in ROUGE as an evaluation metric, so we attempt to understand our models deficiencies manually.”
BackgroundIndian Legal Text Summarization: A Text Normalisation-based ApproachPreprint
Ghosh, S., Dutta, M., & Das, T. (2022) · doi:10.36227/techrxiv.19944665.v3 · Cited by 4
Indian Legal Text Summarization: A Text Normalisation-based ApproachPreprint
Ghosh, S., Dutta, M., & Das, T. (2022) · doi:10.36227/techrxiv.19944665 · Cited by 3
CASESUMM: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court OpinionsPreprint
Heddaya, M., MacMillan, K., Malani, A., Mei, H., & Tan, C. (2025) · Qeios · doi:10.32388/ej9r29 · Cited by 4
Miyaji, R., Moulin, R., Monção, S., & Machado, L. (2025) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2025.37829 · Cited by 5
“While more sensitive to content coverage, ROUGE shares similar limitations in failing to account for synonymy or contextual meaning [Schluter 2017].”
Grounding large language models in clinical diagnostics
Chen, X., Zhou, H., Yi, H., You, M., Liu, W., Li, W., Qin, Z., Li, H., Zhang, X., Guo, Y., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-70274-w · Cited by 18
“Additionally, BLEU and ROUGE assign equal weight to all content regardless of clinical signi fi cance 48,50 .”
Generate Descriptions of Medical Dialogues Through Two-Layers Transformer-Based Summarization
Abanoub, G. E., Fawzy, A. M., Waly, R. R., & Gomaa, W. H. (2023) · Intelligent Methods, Systems, and Applications (IMSA) · doi:10.1109/imsa58542.2023.10217636 · Cited by 3
“Rouge-Lsum: [11] A Rouge-L extension that computes the Rouge-L score for each sentence in the produced summary and its matching sentences in the reference summaries.”
BackgroundMediaHG: Rethinking Eye-catchy Features in Social Media Headline Generation
Zhang, B., & Yang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.352 · Cited by 4
Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications
Hamdhana, D., Kaneko, H., Victorino, J. N., & Inoue, S. (2024) · Healthcare · doi:10.3390/healthcare12030367 · Cited by 4
“ROUGE [18] Measures the overlap of n-grams and the longest matching sequence between a generated summary and reference texts.”
BackgroundFlexible and Adaptable Summarization via Expertise Separation
Chen, X., Li, M., Gao, S., Cheng, X., Zhu, Q., Yan, R., Gao, X., & Zhang, X. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657789 · Cited by 7
“We also conducted a human evaluation of our model to balance the potential bias of automated metrics in assessing summarization quality [36].”
MethodZhang, Y., Lai, Y., Wang, Z., & Zhou, D. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128369 · Cited by 3
esT5s: A Spanish Model for Text Summarization
Vogel-Fernandez, A., Calleja, P., & Rico, M. (2022) · Studies on the semantic web · doi:10.3233/ssw220020 · Cited by 2
A Summarization Dataset of Slovak News Articles
Šuppa, M., & Adamec, J. (2020) · Language Resources and Evaluation · Cited by 2
HighRES: Highlight-based Reference-less Evaluation of SummarizationPreprint
Hardy, Narayan, S., & Vlachos, A. (2019) · arXiv (Cornell University) · doi:10.17863/cam.40621 · Cited by 2
Automatic Summarization of Legal Text
Luijtgaarden, N. V. D. (2019) · Utrecht University Repository (Utrecht University) · Cited by 2
Spinning Sequence-to-Sequence Models with Meta-BackdoorsPreprint
Bagdasaryan, E., & Shmatikov, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10443 · Cited by 4
“These metrics are imperfect and maximizing the ROUGE score for extractive summarization (which extracts important fragments from the text) is an NP-hard problem [47].”
BackgroundIndian Legal Text Summarization: A Text Normalization-based ApproachPreprint
Ghosh, S., Dutta, M., & Das, T. (2022) · doi:10.36227/techrxiv.19944665.v2 · Cited by 2
Prompt Scoring System for Dialog Summarization
Prodan, G. P., & Pelican, E. (2025) · The European Journal on Artificial Intelligence · doi:10.1177/30504554251321829 · Cited by 1
“This limitation is well documented in the literature (Nguyen et al., 2024; Schluter, 2017).”
The Cocoruta Hub: Open and Curated Corpora, Datasets and Language Models on Brazilian Ocean Law
Santo, F. O. D. E., Peres, S. M., Gonçalves, B., Ortega, F. J. M., Matos, V. B., de Lima, A. P., Brandão, A. A. F., & Cozman, F. G. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5791 · Cited by 1
“The user then selects the set of metrics that must be computed, such as ROUGE [ Lin, 2004; Schluter, 2017], BLEU [Papineni et al ., 2002; Chen and Cherry, 2014], BERTSCORE [Zhang et al ., 2020] or MOVERSCORE [Zhao et al ., 2019].”
Rethinking ROUGE Scores for Video Game Review Summarization
Lee, Y.-H., & Jung, Y. (2022) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2022.20.3.35 · Cited by 1
“Several studies indicated the limitations of ROUGE and analyzed its shortcomings [3][4].”
BackgroundTowards Explainable Summary of Crowdsourced Reviews Through Text Mining
Moody, A., Hu, C., Zhan, H., Spurling, M., & Sheng, V. S. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-08971-8_44 · Cited by 1
Universal Evasion Attacks on Summarization Scoring
Mu, W., & Lim, K. H. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.9 · Cited by 5
An Anchor-Based Automatic Evaluation Metric for Document Summarization
Wang, K., Liu, T., Chang, B., & Sui, Z. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.500 · Cited by 1
“Challenges remain in the design of an ideal evaluation metric and the off-the-shelf metrics have their own drawbacks (Schluter, 2017; Kryscinski et al., 2019).”
BackgroundI don't understand! Evaluation methods for natural language explanations
Clinciu, M., Eshghi, A., & Hastie, H. (2021) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 1
Learning to represent, categorise and rank in community question answering
Bogdanova, D. (2018) · Dublin City University Open Access Institutional Repository (Dublin City University) · Cited by 1
Attention-based Approaches for Text Analytics in Social Media and Automatic Summarization
Barba, J. Á. G. (2021) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/172245 · Cited by 4
Indian Legal Text Summarization: A Text Normalisation-based ApproachPreprint
Ghosh, S., Dutta, M., & Das, T. (2022) · doi:10.36227/techrxiv.19944665.v4 · Cited by 1
Iterative Document Representation Learning Towards Summarization with PolishingPreprint
Chen, X., Gao, S., Tao, C., Song, Y., Zhao, D., & Yan, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10324 · Cited by 2
“Schluter (2017) noted that only using the Rouge metric to evaluate summarization quality can be misleading.”
BackgroundTündik, M. Á., Kaszás, V., & Szaszák, G. (2019) · Periodica Polytechnica Electrical Engineering and Computer Science · doi:10.3311/ppee.14052 · Cited by 1
Gao, S., Chen, X., Li, P., Chan, Z., Zhao, D., & Yan, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.08837 · Cited by 4
“(Schluter, 2017) notes that only using the ROUGE metric to evaluate summarization quality can be misleading.”
MethodCut the CARP: Fishing for zero-shot story evaluationPreprint
Matiana, S., Smith, Jr., Teehan, R., Castricato, L., Biderman, S., Gao, L., & Frazier, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03111 · Cited by 9
“Unfortunately, methods for automatically evaluating stories, such as ROUGE (Lin, 2004), are extremely limited in their ability to accurately assess models and are easily Goodharted (Belz & Reiter, 2006; Cohan & Goharian, 2016; Schluter, 2017; Schluter & Alonso, 2016; Eyal et al., 2019).”
MethodIndian Legal Text Summarization: A Text Normalization-based ApproachPreprint
Ghosh, S., Dutta, M., & Das, T. (2022) · doi:10.36227/techrxiv.19944665.v1
ExplainMeetSum: A Dataset for Explainable Meeting Summarization Aligned with Human Intent
Kim, H. J., Cho, M., & Na, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.731 · Cited by 7
“Given the known limitations of using ROUGE due to its simplified n-gram matching style (Schluter, 2017), some studies have focused on evaluation in the setting of extractive summarization (Ma et al.”
Method
Metadata
- Authors
- Schluter, N.
- Affiliations
- University of Copenhagen (DK)
- Corresponding author
- Schluter, Natalie
- ORCID
- Schluter, Natalie
- Venue
- Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers
- Published
- 2017
- Pages
- 41-45
- OpenAlex
- W2741672218
- Cited by
- 108
- References
- 12
- Citations by year
- 2026: 7 · 2025: 8 · 2024: 8 · 2023: 16 · 2022: 21 · 2021: 13 · 2020: 12 · 2019: 16 · 2018: 6 · 2017: 1
- FWCI
- 4.37
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion, cc-by-nc-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- text summarization, ROUGE, NP-hardness, greedy algorithm, benchmark dataset, quality assurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.