The limits of automatic summarisation according to ROUGE
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
This paper discusses some central caveats of summarisation, incurred in the use of the ROUGE metric for evaluation, with respect to optimal solutions.The task is NPhard, of which we give the first proof.Still, as we show empirically for three central benchmark datasets for the task, greedy algorithms empirically seem to perform optimally according to the metric.Additionally, overall quality assurance is problematic: there is no natural upper bound on the quality of summarisation systems, and even humans are excluded from performing optimal summarisation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1979) · Cited by 43,841
(2004) · Cited by 9,718
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656 · cited 1×
“There have been recent advances in supervised summarisation mainly with respect to supervised learning using neural networks (for example (Rush et al., 2015; Chopra et al., 2016)).”
BackgroundAutomatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 1×
“We know that 100% perfect scores are impossible, so what is a perfect score according to ROUGE? How do we know when no improvement is possible? Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; L…”
BackgroundAbstractive Sentence Summarization with Attentive Recurrent Neural Networks
Chopra, S., Auli, M., & Rush, A. M. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1012 · Cited by 881 · cited 3×
“, 2015; Chopra et al., 2016)). However, due to data size requirements, these systems are constrained to title generation systems and therefore not in the scope of this work. Hong et al. (2014) survey the state-of-the-art using the central DUC 2004 dataset.”
BackgroundEvaluating Content Selection in Summarization: The Pyramid Method
Nenkova, A., & Passonneau, R. (2004) · Columbia University · doi:10.7916/d80r9xvd · Cited by 527 · cited 2×
“The metric returns an average of ROUGE scores over multiple reference summaries in order to avoid bias (Nenkova and Passonneau, 2004).”
BackgroundA scalable global model for summarization
Gillick, D., & Favre, B. (2009) · Proceedings of the Workshop on Integer Linear Programming for Natural Langauge Processing - ILP '09 · doi:10.3115/1611638.1611640 · Cited by 280 · cited 1×
“Of these, ICSISum (Gillick and Favre, 2009) is the only global summariser using an exact algorithm; it obtains the best ROUGE-2 score without supervision.”
MethodRe-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE
Graham, Y. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1013 · Cited by 143 · cited 1×
“Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; Lin and Hovy, 2003; Liu and Liu, 2008; Graham, 2015).”
BackgroundA Repository of State of the Art and Competitive Baseline Summaries for Generic News Summarization
Hong, K., Conroy, J. M., Favre, B., Kulesza, A., Lin, H. Z., & Nenkova, A. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4aa4e4yajcec · Cited by 122 · cited 2×
“…summarisation, recent research over conventional benchmark datasets has suggested the following: according to the most widely accepted automatic evaluation metric, ROUGE, there has been no substantial improvement in performance on central datasets in the field in the last decade (Hong et al., 2014).”
BackgroundMethodInfluentialCorrelation between ROUGE and human evaluation of extractive meeting summaries
Liu, F., & Liu, Y. (2008) · Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies Short Papers - HLT '08 · doi:10.3115/1557690.1557747 · Cited by 81 · cited 1×
“Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; Lin and Hovy, 2003; Liu and Liu, 2008; Graham, 2015).”
Background
Cited by
Items 101–111 of 111
Sumarizando múltiplos websites para a geração do Wikipédia PT-BR automaticamente.
Oliveira, A. S. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.3.2023.tde-01032024-090345
Ehnert, P., & Schröter, J. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1200949
“…limitations of the ROUGE metric in evaluating abstractive approaches, as reflected in the variance of ROUGE scores in our results (see Table 3) and confirmed by previous research as humans tend to formulate abstractive summaries that are mostly not equivalent in terms of word choice (Schluter, 2017).”
ResultWheat Disease Monitoring Based on Multimodal Large Models
Liao, T., Li, R., Xie, C., Qi, R., Hu, T., Zhang, J., & Liu, M. Y. (2025) · International Conference on Advanced Algorithms and Control Engineering (ICAACE) · doi:10.1109/icaace65325.2025.11020040
“In terms of the evaluation method of the model,this paper adopts the BLEU[20] and ROUGE[21] methods to assess the model.”
Methods to Evaluate and Improve the Quality of Text Summarization
Nguyen, H. (2025) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc2443390 · Cited by 1
A survey on ordering of text at different granular levels
Tiwari, P., & Chowdary, C. R. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-025-02628-7
“Natalie Schluter [127] claimed its limitation be an NP-hard problem for global optimization.”
A Novel Approach For Document Summarization
K, S. K., Nair, A. B., Suresh, A. T., Baby, J., & Kumar, K. M. A. (2026) · International Conference on Image Processing and Capsule Networks (ICIPCN) · doi:10.1109/icipcn67432.2026.11438801
“ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is one of the popular metrics [11] used to evaluate the quality of machine generated summary with the human generated summary.”
Résumé automatique multi-document dynamique
Mnasri, M. (2018) · theses.fr (ABES) · doi:10.70675/2eaaad3az0ec3z400ez9138z733931802bf8
“Cependant, (Schluter, 2017) a montré que la méthode gloutonne est aussi performante que la méthode exacte globale.”
BackgroundНиколаевич, Г. В. (2026) · Программные системы и вычислительные методы · doi:10.7256/2454-0714.2026.1.78734
Patent Summarization Framework based on a Hybrid Integration of LoRA Fine-Tuning and RAG
Jang, J.-H., & Kim, G.-W. (2026) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2026.24.7.299
B, A. E., & Rajesh, M. V. (2026) · International Conference on Innovations in Engineering and Technology (ICIET) · doi:10.1109/iciet70550.2026.11712594
“Its limitations for abstractive systems are well documented [17]: it penalises valid paraphrases and fails to capture semantic equivalence.”
Pattnaik, S., & Pinsky, E. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-33123-6_6
Metadata
- Authors
- Schluter, N.
- Affiliations
- University of Copenhagen (DK)
- Corresponding author
- Schluter, Natalie
- ORCID
- Schluter, Natalie
- Venue
- Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers
- Published
- 2017
- Pages
- 41-45
- OpenAlex
- W2741672218
- Cited by
- 108
- References
- 12
- Citations by year
- 2026: 7 · 2025: 8 · 2024: 8 · 2023: 16 · 2022: 21 · 2021: 13 · 2020: 12 · 2019: 16 · 2018: 6 · 2017: 1
- FWCI
- 4.37
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion, cc-by-nc-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- text summarization, ROUGE, NP-hardness, greedy algorithm, benchmark dataset, quality assurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.