The limits of automatic summarisation according to ROUGE
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
This paper discusses some central caveats of summarisation, incurred in the use of the ROUGE metric for evaluation, with respect to optimal solutions.The task is NPhard, of which we give the first proof.Still, as we show empirically for three central benchmark datasets for the task, greedy algorithms empirically seem to perform optimally according to the metric.Additionally, overall quality assurance is problematic: there is no natural upper bound on the quality of summarisation systems, and even humans are excluded from performing optimal summarisation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1979) · Cited by 43,841
(2004) · Cited by 9,718
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656 · cited 1×
“There have been recent advances in supervised summarisation mainly with respect to supervised learning using neural networks (for example (Rush et al., 2015; Chopra et al., 2016)).”
BackgroundAutomatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578 · cited 1×
“We know that 100% perfect scores are impossible, so what is a perfect score according to ROUGE? How do we know when no improvement is possible? Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; L…”
BackgroundAbstractive Sentence Summarization with Attentive Recurrent Neural Networks
Chopra, S., Auli, M., & Rush, A. M. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1012 · Cited by 881 · cited 3×
“, 2015; Chopra et al., 2016)). However, due to data size requirements, these systems are constrained to title generation systems and therefore not in the scope of this work. Hong et al. (2014) survey the state-of-the-art using the central DUC 2004 dataset.”
BackgroundEvaluating Content Selection in Summarization: The Pyramid Method
Nenkova, A., & Passonneau, R. (2004) · Columbia University · doi:10.7916/d80r9xvd · Cited by 527 · cited 2×
“The metric returns an average of ROUGE scores over multiple reference summaries in order to avoid bias (Nenkova and Passonneau, 2004).”
BackgroundA scalable global model for summarization
Gillick, D., & Favre, B. (2009) · Proceedings of the Workshop on Integer Linear Programming for Natural Langauge Processing - ILP '09 · doi:10.3115/1611638.1611640 · Cited by 280 · cited 1×
“Of these, ICSISum (Gillick and Favre, 2009) is the only global summariser using an exact algorithm; it obtains the best ROUGE-2 score without supervision.”
MethodRe-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE
Graham, Y. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1013 · Cited by 143 · cited 1×
“Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; Lin and Hovy, 2003; Liu and Liu, 2008; Graham, 2015).”
BackgroundA Repository of State of the Art and Competitive Baseline Summaries for Generic News Summarization
Hong, K., Conroy, J. M., Favre, B., Kulesza, A., Lin, H. Z., & Nenkova, A. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4aa4e4yajcec · Cited by 122 · cited 2×
“…summarisation, recent research over conventional benchmark datasets has suggested the following: according to the most widely accepted automatic evaluation metric, ROUGE, there has been no substantial improvement in performance on central datasets in the field in the last decade (Hong et al., 2014).”
BackgroundMethodInfluentialCorrelation between ROUGE and human evaluation of extractive meeting summaries
Liu, F., & Liu, Y. (2008) · Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies Short Papers - HLT '08 · doi:10.3115/1557690.1557747 · Cited by 81 · cited 1×
“Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; Lin and Hovy, 2003; Liu and Liu, 2008; Graham, 2015).”
BackgroundExploring events and distributed representations of text in multi-document summarization
Marujo, L., Wang, L., Ribeiro, R., Gershman, A. V., Carbonell, J., de Matos, D. M., & Neto, J. P. D. S. (2015) · Knowledge-Based Systems · doi:10.1016/j.knosys.2015.11.005 · Cited by 42 · cited 1×
“So, for example, even humans are doomed to perform sub-optimally (Cf. Marujo et al. (2016)).”
BackgroundUnsupervised extractive summarization via coverage maximization with syntactic and semantic concepts
Schluter, N., & Søgaard, A. (2015) · doi:10.3115/v1/p15-2138 · Cited by 23 · cited 2×
“We use the same summary budget of 335 used by Schluter and Søgaard (2015).”
MethodSelf-citation
Cited by
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207
“Human Evaluation In addition to automatic evaluation using ROUGE which can be misleading when used as the only means to assess the informativeness of summaries (Schluter, 2017), we also evaluated system output by eliciting human judgments in two ways.”
MethodOn Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185
“ness of summaries (Schluter, 2017).”
MethodLearning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
“Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE [39], have received criticism for poor correlation with human judgments [55, 45, 6, 33].”
BackgroundRanking Sentences for Extractive Summarization with Reinforcement Learning
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1158 · Cited by 688
“In addition to ROUGE which can be misleading when used as the only means to assess the informativeness of summaries (Schluter, 2017), we also evaluated system output by eliciting human judgments in two ways.”
Method
Metadata
- Authors
- Schluter, N.
- Affiliations
- University of Copenhagen (DK)
- Corresponding author
- Schluter, Natalie
- ORCID
- Schluter, Natalie
- Venue
- Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers
- Published
- 2017
- Pages
- 41-45
- OpenAlex
- W2741672218
- Cited by
- 108
- References
- 12
- Citations by year
- 2026: 7 · 2025: 8 · 2024: 8 · 2023: 16 · 2022: 21 · 2021: 13 · 2020: 12 · 2019: 16 · 2018: 6 · 2017: 1
- FWCI
- 4.37
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion, cc-by-nc-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- text summarization, ROUGE, NP-hardness, greedy algorithm, benchmark dataset, quality assurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.