The Pyramid Method: Incorporating human content selection variation in summarization evaluation
Abstract
Human variation in content selection in summarization has given rise to some fundamental research questions: How can one incorporate the observed variation in suitable evaluation measures? How can such measures reflect the fact that summaries conveying different content can be equally good and informative? In this article, we address these very questions by proposing a method for analysis of multiple human abstracts into semantic content units. Such analysis allows us not only to quantify human variation in content selection, but also to assign empirical importance weight to different content units. It serves as the basis for an evaluation method, the Pyramid Method, that incorporates the observed variation and is predictive of different equally informative summaries. We discuss the reliability of content unit annotation, the properties of Pyramid scores, and their correlation with other evaluation methods.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The pyramid annotation scheme, one of the highest standards of human evaluation on summarization, requires multiple reference summaries.
“In fact, one of the highest standards of human evaluation on summarization, the pyramid41 annotation scheme, requires multiple reference summaries.”
From Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation (2023) · cites this paper as [41]
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,305 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448 · see citing works
Measures of the Amount of Ecologic Association Between Species
Dice, L. R. (1945) · Ecology · doi:10.2307/1932409 · Cited by 12,153 · see citing works
(2004) · Cited by 9,718 · see citing works
Introduction to Statistical Analysis.
Latshaw, V. V., Dixon, W. J., & Massey, F. J. (1957) · American Mathematical Monthly · doi:10.2307/2309989 · Cited by 5,072 · see citing works
Krippendorff, K. (1980) · Journal of International Crisis and Risk Communication Research · Cited by 1,207 · see citing works
Proceedings of the International Conference on Language Resources and Evaluation (LREC)
Wentland, W., Knopp, J., Silberer, C., & Härtung, M. (2008) · Cited by 662 · see citing works
The Theory and Practice of Discourse Parsing and Summarization
Marcu, D. (2000) · The MIT Press eBooks · doi:10.7551/mitpress/6754.001.0001 · Cited by 626 · see citing works
Evaluating Content Selection in Summarization: The Pyramid Method
Nenkova, A., & Passonneau, R. (2004) · Columbia University · doi:10.7916/d80r9xvd · Cited by 527 · see citing works
Self-citationCentroid-based summarization of multiple documents
Radev, D., Jing, H., & Budzikowska, M. (2000) · doi:10.3115/1117575.1117578 · Cited by 515 · see citing works
Automatic text structuring and summarization
Salton, G., Singhal, A., Mitra, M., & Buckley, C. (1997) · Information Processing & Management · doi:10.1016/s0306-4573(96)00062-3 · Cited by 497 · see citing works
The Kappa Statistic: A Second Look
Eugenio, B. D., & Glass, M. (2004) · Computational Linguistics · doi:10.1162/089120104773633402 · Cited by 489 · see citing works
Goldstein, J., Kantrowitz, M., Mittal, V. O., & Carbonell, J. (1999) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/312624.312665 · Cited by 462 · see citing works
Sentence level discourse parsing using syntactic and lexical information
Soricut, R., & Marcu, D. (2003) · doi:10.3115/1073445.1073475 · Cited by 412 · see citing works
Recall of prose as a function of the structural importance of the linguistic units
Johnson, R. E. (1970) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(70)80003-2 · Cited by 384 · see citing works
Summarization evaluation methods: Experiments and analysis
McKeown, K. R., Jing, H., Barzilay, R., & Elhadad, M. (1998) · Columbia Academic Commons (Columbia University) · doi:10.7916/d8tb1g77 · Cited by 179 · see citing works
Self-citationSummarization Evaluation: An Overview.
Mani, I. (2001) · Cited by 156 · see citing works
Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation
Passonneau, R. J. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4nuo6thi27ax · Cited by 142 · see citing works
Self-citationColumbia Multi-Document Summarization: Approach and Evaluation
McKeown, K. R., Hatzivassiloglou, V., Barzilay, R., et al. (2001) · doi:10.7916/d82v2qhf · Cited by 126 · see citing works
Self-citationA comparison of rankings produced by summarization evaluation measures
Donaway, R. L., Drummey, K. W., & Mather, L. A. (2000) · doi:10.3115/1117575.1117583 · Cited by 121 · see citing works
Rath, G., Resnick, A., & Savage, T. R. (1961) · American Documentation · doi:10.1002/asi.5090120210 · Cited by 117 · see citing works
Introduction to Statistical Analysis
Konasani, V. R., & Kadre, S. (2015) · Apress eBooks · doi:10.1007/978-1-4842-0043-8_5 · Cited by 109 · see citing works
Examining the consensus between human summaries
van Halteren, H., & Teufel, S. (2003) · doi:10.3115/1119467.1119475 · Cited by 92 · see citing works
Efficient Text Summarization Costs and Benefits
Garner, R. (1982) · The Journal of Educational Research · doi:10.1080/00220671.1982.10885394 · Cited by 68 · see citing works
Applying the Pyramid Method in DUC 2005
McKeown, K. R., Passonneau, R. J., Nenkova, A., et al. (2005) · doi:10.7916/d8tx3pvd · Cited by 68 · see citing works
Self-citationEvaluating Information Content by Factoid Analysis : Human annotation and stability
Teufel, S., & van Halteren, H. (2004) · Cited by 59 · see citing works
Summarization evaluation using relative utility
Radev, D., & Tam, D. (2003) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/956863.956960 · Cited by 40 · see citing works
Discourse-Based Summarization in DUC-2001
Marcu, D. (2001) · Cited by 21 · see citing works
Single-document and multi-document summary evaluation using Relative Utility
Radev, D., Tam, D., & Erkan, G. (2007) · Cited by 12 · see citing works
Proceedings of the 2000 NAACL-ANLP Workshop on Automatic Summarization
Hahn, U., Lin, C.-Y., Mani, I., et al. (2000) · Cited by 12 · see citing works
Evaluating Summaries and Answers: Two Sides of the Same Coin?
Lin, J., & Demner‐Fushman, D. (2005) · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Unifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161 · see citing works
Hunter, J., Freer, Y., Gatt, A., Reiter, E., Sripada, S., & Sykes, C. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.09.002 · Cited by 127 · see citing works
Yim, W.-W., Fu, Y., Ben Abacha, A., Snider, N., Lin, T., & Yetisgen, M. (2023) · Scientific Data · doi:10.1038/s41597-023-02487-3 · Cited by 64 · see citing works
Scirè, A., Ghonim, K., & Navigli, R. (2024) · arXiv · doi:10.48550/arxiv.2403.02270 · Cited by 2 · see citing works
- Authors
- Nenkova, A., Passonneau, R., & McKeown, K.
- Affiliations
- Columbia University (US)
- Venue
- ACM Transactions on Speech and Language Processing
- Published
- 2007-05
- Volume
- 4
- Issue
- 2
- Pages
- 4
- ISSN
- 1550-4875 (ISSN-L)
- 1550-4883
- OpenAlex
- W2123946480
- Cited by
- 307
- References
- 33
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 5 · 2023: 5 · 2022: 11 · 2021: 9 · 2020: 14 · 2019: 18 · 2018: 24 · 2017: 12 · 2016: 22 · 2015: 13 · 2014: 21 · 2013: 23 · 2012: 24
- FWCI
- 27.23
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Speech and Language Processing (publishedVersion)
- Funders
- National Institute of Standards and Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- text summarization, summary evaluation, pyramid method, content selection, human diversity, evaluation metrics, informativeness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
