Will Pyramids Built of Nuggets Topple Over?
Open accessAbstract
The present methodology for evaluating complex questions at TREC analyzes answers in terms of facts called "nuggets". The official F-score metric represents the harmonic mean between recall and precision at the nugget level. There is an implicit assumption that some facts are more important than others, which is implemented in a binary split between "vital" and "okay" nuggets. This distinction holds important implications for the TREC scoring model---essentially, systems only receive credit for retrieving vital nuggets---and is a source of evaluation instability. The upshot is that for many questions in the TREC testsets, the median score across all submitted runs is zero. In this work, we introduce a scoring model based on judgments from multiple assessors that captures a more refined notion of nugget importance. We demonstrate on TREC 2003, 2004, and 2005 data that our "nugget pyramids" address many shortcomings of the present methodology, while introducing only minimal additional overhead on the evaluation flow.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Lin and Demner-Fushman (2006) showed how importance can be extended to grades of importance by combing ratings from multiple assessors.
“Lin and Demner-Fushman (2006) showed how importance can be extended to grades of importance by combing ratings from multiple assessors (i.e., nugget importance is based on annotator popularity).”
From All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations (2025) · cites this paper as Lin and Demner-Fushman, 2006
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 10
- Enrique Amigó, Julio Gonzalo, Victor Peinado, Anselmo Peñas, and Felisa Verdejo. 2004. An empirical study of information synthesis task. In Proceedings of the 42nd Annual Meeting of the Association for Computa- tional Linguistics (ACL 2004).
- Hoa Dang. 2005. Overview of DUC 2005. In Proceed- ings of the 2005 Document Understanding Conference (DUC 2005) at NLT/EMNLP 2005.
- Wesley Hildebrandt, Boris Katz, and Jimmy Lin. 2004. Answering definition questions with multiple knowl- edge sources. In Proceedings of the 2004 Human Lan- guage Technology Conference and the North American Chapter of the Association for Computational Linguis- tics Annual Meeting (HLT/NAACL 2004).
- Jimmy Lin and Dina Demner-Fushman. 2005a. Auto- matically evaluating answers to definition questions. In Proceedings of the 2005 Human Language Technol- ogy Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005).
- Jimmy Lin and Dina Demner-Fushman. 2005b. Evalu- ating summaries and answers: Two sides of the same coin? In Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization.
- Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Lan- guage Technology Conference and the North American Chapter of the Association for Computational Linguis- tics Annual Meeting (HLT/NAACL 2003).
- Ani Nenkova and Rebecca Passonneau. 2004. Evalu- ating content selection in summarization: The pyra- mid method. In Proceedings of the 2004 Human Lan- guage Technology Conference and the North American Chapter of the Association for Computational Linguis- tics Annual Meeting (HLT/NAACL 2004).
- Ellen M. Voorhees. 1998. Variations in relevance judg- ments and the measurement of retrieval effectiveness. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1998).
- Ellen M. Voorhees. 2003. Overview of the TREC 2003 question answering track. In Proceedings of the Twelfth Text REtrieval Conference (TREC 2003).
- Ellen M. Voorhees. 2005. Using question series to eval- uate question answering system effectiveness. In Pro- ceedings of the 2005 Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005).
Cited by
Sorted by citations
The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., Soboroff, I., Dang, H. T., & Lin, J. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730090 · Cited by 14 · see citing works
“Further re-finements for question answering evaluation were introduced in later work [7, 23, 24].”
Self-citationWanner, M., Azzopardi, L., Thomas, P., Dan, S., Van Durme, B., & Craswell, N. (2025) · arXiv · doi:10.48550/arxiv.2510.07083
“Lin and Demner-Fushman (2006) showed how importance can be extended to grades of importance by combing ratings from multiple assessors (i.e., nugget importance is based on anno-tator popularity).”
The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., & Lin, J. (2025) · arXiv · doi:10.48550/arxiv.2504.15068 · Cited by 3 · see citing works
Self-citation
- Authors
- Lin, J., & Demner-Fushman, D.
- Affiliations
- University of Maryland, College Park (US)
- ORCID
- Lin, Jimmy
- Venue
- Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics -
- Published
- 2006
- Pages
- 383-390
- OpenAlex
- W2021420765
- Cited by
- 77
- References
- 0
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 2 · 2022: 2 · 2021: 1 · 2020: 3 · 2019: 1 · 2018: 1 · 2015: 2 · 2014: 3 · 2013: 1 · 2012: 6
- FWCI
- 9.21
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion)
- Funders
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- TREC, question answering evaluation, F-score, scoring system
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
