Evaluating Content Selection in Summarization: The Pyramid Method
Open accessAbstract
We present an empirically grounded method for evaluating content selection in summarization. It incorporates the idea that no single best model summary for a collection of documents exists. Our method quantifies the relative importance of facts to be conveyed. We argue that it is reliable, predictive and diagnostic, thus improves considerably over the shortcomings of the human evaluation method currently used in the Document Understanding Conference.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Having multiple gold summaries for each document enables performing more in-depth analysis and thorough evaluation.
“In addition, having multiple gold summaries for each document enables performing more in-depth analysis and thorough evaluation (Nenkova and Passonneau, 2004).”
From TLDR: Extreme Summarization of Scientific Documents (2020) · cites this paper as Nenkova and Passonneau, 2004
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776 · see citing works
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407 · see citing works
The Pyramid Method: Incorporating human content selection variation in summarization evaluation
Nenkova, A., Passonneau, R., & McKeown, K. (2007) · ACM Transactions on Speech and Language Processing · doi:10.1145/1233912.1233913 · Cited by 307 · see citing works
Self-citationSupporting program comprehension with source code summarization
Haiduc, S., Aponte, J., & Marcus, A. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2 · doi:10.1145/1810295.1810335 · Cited by 291 · see citing works
Is chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235 · see citing works
TLDR: Extreme Summarization of Scientific Documents
Cachola, I., Lo, K., Cohan, A., & Weld, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.428 · Cited by 169 · see citing works
“We analyze the information content of TLDRs using an approach motivated by the nugget-based summarization evaluation framework of Nenkova and Passonneau (2004). In a similar manner, we asked two computer science researchers to read through a collection of TLDRs to both define a comprehensive set of categories of types…”
BackgroundMethodAssessing The Factual Accuracy of Generated Text
Goodrich, B., Rao, V., Liu, P. J., & Saleh, M. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330955 · Cited by 162 · see citing works
Re-evaluating Evaluation in Text Summarization
Bhandari, M., Gour, P. N., Ashfaq, A., Liu, P., & Neubig, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.751 · Cited by 148 · see citing works
The limits of automatic summarisation according to ROUGE
Schluter, N. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2007 · Cited by 108 · see citing works
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems
Thomson, C., & Reiter, E. (2020) · Proceedings of the 13th International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.22 · Cited by 38 · see citing works
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14251 · Cited by 14 · see citing works
“Computing FA CT S CORE involves (1) breaking a generation into a series of atomic facts—short statements that each contain one piece of information (Nenkova and Passonneau, 2004; Shapira et al., 2019; Zhang and Bansal, 2021; Liu et al., 2022), and (2) assigning a binary label to each atomic fact,…”
BackgroundMethodThe Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., Soboroff, I., Dang, H. T., & Lin, J. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730090 · Cited by 14 · see citing works
Wanner, M., Azzopardi, L., Thomas, P., Dan, S., Van Durme, B., & Craswell, N. (2025) · arXiv · doi:10.48550/arxiv.2510.07083
“Nugget based evaluation metrics have been used for not only assessing answer quality, but also document relevance, summarization (Nenkova and Passonneau, 2004), RAG (Pradeep et al., 2025), and conversational search (Abbasiantaeb et al., 2025).”
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · arXiv · doi:10.48550/arxiv.2301.13298 · Cited by 3
“A notable effort in this space is the Pyramid method (Nenkova and Passonneau, 2004), along with work improving Pyramid efficiency (Shapira et al.”
BackgroundMethodInfluential
- Authors
- Nenkova, A., & Passonneau, R.
- Venue
- Columbia University
- Publisher
- Columbia University
- Published
- 2004
- Pages
- 145-152
- OpenAlex
- W2102065370
- Cited by
- 527
- References
- 1
- Citations by year
- 2025: 2 · 2024: 6 · 2023: 22 · 2022: 25 · 2021: 34 · 2020: 36 · 2019: 28 · 2018: 24 · 2017: 22 · 2016: 32 · 2015: 29 · 2014: 18 · 2013: 23 · 2012: 22
- FWCI
- 33.48
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Columbia Academic Commons (Columbia University)
- North American Chapter of the Association for Computational Linguistics
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- text summarization, content selection, pyramid method, summarization evaluation, multi-document summarization, DUC
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
