The DOI matches its DataCite record.
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics
Open accessAbstract
How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice and propose changes to rectify this disconnect. First, we calculate the system score for an automatic metric using the full test set instead of the subset of summaries judged by humans, which is currently standard practice. We demonstrate how this small change leads to more precise estimates of system-level correlations. Second, we propose to calculate correlations only on pairs of systems that are separated by small differences in automatic scores which are commonly observed in practice. This allows us to demonstrate that our best estimate of the correlation of ROUGE to human judgments is near 0 in realistic scenarios. The results from the analyses point to the need to collect more high-quality human judgments and to improve automatic metrics when differences in system scores are small.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most existing benchmarks rely on metrics like ROUGE for long-document QA and summarization, which are often noisy and unreliable.
“Unreliable metrics : For commonly used long-document QA and summarization tasks, most existing benchmarks still rely on metrics like ROUGE (Lin, 2004), which are often noisy and unreliable (Goyal et al., 2023; Deutsch et al., 2022; Chang et al., 2024).”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Deutsch et al., 2022
Existing long-context benchmarks largely rely on n-gram overlap metrics like ROUGE, which correlate poorly with human judgment for tasks with long outputs such as summarization.
“Existing long-context benchmarks (Zhang et al., 2024b; Shaham et al., 2023) largely rely on n-gram overlap metrics like ROUGE (Lin, 2004), which have been shown to correlate poorly with human judgment for tasks with long outputs, such as summarization (Goyal et al., 2023; Deutsch et al., 2022; Krishna et al., 2023).”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Deutsch et al., 2022
Citations
Cited by
Sorted by citations
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2
“Existing long-context benchmarks (Zhang et al., 2024; Shaham et al., 2023) largely rely on n-gram overlap metrics like ROUGE (Lin, 2004) that have been shown to correlate poorly with human judgment for tasks with long outputs, such as summarization (Goyal et al., 2023; Deutsch et al., 2022).”
Background
Metadata
- Authors
- Deutsch, D., Dror, R., & Roth, D.
- Affiliations
- University of Pennsylvania (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 6038-6052
- arXiv
- 2204.10216
- OpenAlex
- W4224223051
- Cited by
- 22
- References
- 0
- Citations by year
- 2026: 2 · 2024: 4 · 2023: 14 · 2022: 2
- FWCI
- 2.76
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- U.S. Department of Defense
- Office of the Director of National Intelligence
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
- Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity (2019-19051600006)
- NIH Office of the Director
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- automatic summarization, summarization evaluation metrics, ROUGE, measurement reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
