CoChat
Sign Up
Cited by 14Topic ModelingInformation Retrieval and Search BehaviorNatural Language Processing Techniques

The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models

Open access
Pradeep, Ronak, Thakur, Nandan, Upadhyay, Shivani, Campos, Daniel, Craswell, Nick, Soboroff, Ian, Dang, Hoa Trang, Lin, Jimmy·2025·Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval
DOI 10.1145/3726302.3730090
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have significantly enhanced the capabilities of information access systems, especially with retrieval-augmented generation (RAG). Nevertheless, the evaluation of RAG systems remains a barrier to continued progress, a challenge we tackle in this work by proposing an automatic evaluation framework that is validated against human annotations. We believe that the nugget evaluation methodology provides a solid foundation for evaluating RAG systems. This approach, originally developed for the TREC Question Answering (QA) Track in 2003, evaluates systems based on atomic facts that should be present in good answers. Our efforts focus on ''refactoring'' this methodology, where we describe the AutoNuggetizer framework that specifically applies LLMs to both automatically create nuggets and automatically assign nuggets to system answers. In the context of the TREC 2024 RAG Track, we calibrate a fully automatic approach against strategies where nuggets are created manually or semi-manually by human assessors and then assigned manually to system answers. Based on results from a community-wide evaluation, we observe strong agreement at the run level between scores derived from fully automatic nugget evaluation and human-based variants. The agreement is stronger when individual framework components such as nugget assignment are automated independently. This suggests that our evaluation framework provides tradeoffs between effort and quality that can be used to guide the development of future RAG systems. However, further research is necessary to refine our approach, particularly in establishing robust per-topic agreement to diagnose system failures effectively.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces AutoNuggetizer, an LLM-based framework that automates nugget creation and nugget assignment for evaluating retrieval-augmented generation (RAG) answers, validated in the TREC 2024 RAG Track against manual and semi-manual NIST assessor variants. Fully automatic nugget evaluation correlates strongly with human-based variants at the run level, but per-topic agreement remains weaker.

Key findings

  • Fully automatic AutoNuggets / AutoAssign showed strong run-level Kendall’s τ agreement with AutoNuggets+Edits / ManualAssign: 0.887 for Vital Strict and 0.901 for All Strict. Against fully manual ManualNuggets / ManualAssign, run-level τ dropped to 0.727 for Vital Strict and 0.758 for All Strict; on a 20-topic subset of AutoNuggets+Edits, τ was 0.826 and 0.838.
  • Automating only nugget assignment improved agreement over full automation: for Vital Strict, per-topic average Kendall’s τ increased from 0.490 under AutoNuggets / AutoAssign to 0.662 under AutoNuggets+Edits / AutoAssign, and all topics/runs τ improved from 0.328 to 0.599.
  • LLM assessors appeared stricter than NIST assessors in nugget assignment. Confusion matrices showed substantial agreement on Support and No Support—47.6% No Support and 23.8% Support for AutoNuggets+Edits, and 54.0% No Support and 22.1% Support for ManualNuggets—with Partial Support a source of disagreement.
  • Using LLMs to provide draft nuggets in nugget creation did not noticeably increase alignment with human nugget assignment.

Methods

  • TREC 2024 RAG Track, analyzing AG and RAG task answers over 301 queries and the MS MARCO V2.1 segment collection of 113 million passages. NIST received 93 RAG runs from 20 groups and 53 AG runs from 11 groups; analyses used 31 RAG runs from 18 groups and 14 AG runs from 9 groups, over 56 fully judged topics (36 AutoNuggets+Edits / ManualAssign and 20 ManualNuggets / ManualAssign).
  • AutoNuggetizer used GPT-4o via Azure: iterative nugget creation from documents judged at least related, importance labeling as vital or okay, selection of the first 20 nuggets, and listwise automatic assignment as support, partial_support, or not_support with at most 10 nuggets per prompt. Conditions included AutoNuggets+Edits / ManualAssign, ManualNuggets / ManualAssign, AutoNuggets / AutoAssign, and semi-manual creation with AutoAssign.
  • Scoring used All Strict and Vital Strict recall over nuggets, with run scores as the mean query score. Meta-evaluation used Kendall’s τ at the run level, averaged per-topic correlations, and correlations across all topic/run combinations.

Limitations

  • The paper states that per-topic correlations are quite low, suggesting the framework is inadequate for fine-grained debugging of individual answers, and that further research is needed for robust per-topic agreement to diagnose system failures.
  • The paper states that findings are specific to the current implementation; different LLMs or prompts may behave differently, and the analysis compares only one assessor and one LLM implementation, leaving inter-annotator and LLM-human variation unknown.
  • The paper states that it focuses exclusively on recall of information nuggets and does not cover citation support or fluency. It also notes that the corpus is likely already in LLM pretraining data despite using fresh queries; additionally, the text makes evident that manual conditions were limited to 36 and 20 topics, while the fully automatic condition covered all 301 topics.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3726302.3730090

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 19 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The resulting nuggets from different creation approaches are quite similar semantically.

    “There are minor differences, but overall the resulting nuggets are quite similar.”
    • Also stated in: § Search · plain
      “Despite some differences, resulting nuggets from the different approaches are quite similar semantically.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Manual nugget creation took roughly 2.5 hours per topic.

    “This took roughly 2.5 hours per topic, based on our measurements.”
    Quote in the paperNot a substantial claim
  • Findingown data

    LLM assessors are more strict than NIST assessors in nugget assignment.

    “For RQ3, our analyses suggest that LLM assessors appear to be more strict than NIST assessors in nugget assignment.”
    Quote in the paperSubstantial claim
  • Findingown data

    Semi-manual nugget creation with NIST post-editing takes roughly one hour per topic on average.

    “Based on our logs, this process takes roughly an hour per topic on average, which suggests that the NIST assessors are not merely “rubber stamping” automatically generated nuggets, but are actually carefully deliberating over how they are formulated.”
    Quote in the paperSubstantial claim
  • Findingown data

    Fully automatic end-to-end nugget evaluation scores show strong correlation or agreement with manual nugget evaluations at the run level.

    “Based on results from a community-wide evaluation, we observe strong agreement at the run level between scores derived from fully automatic nugget evaluation and human-based variants.”
    • Also stated in: § 1 Introduction · boosted
      “For RQ1, we find that scores from our fully automatic (end-toend) nugget evaluation framework show strong correlations with manual nugget evaluations at the run level.”
    • Also stated in: § 1 Introduction · boosted
      “We demonstrate that system rankings derived from our fully automatic nugget evaluation process strongly correlate with those obtained from manual or semi-manual evaluations by human assessors.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: NIST assessors provide high-quality human annotations suitable as ground truth for validating automatic evaluation.
  • May combine two claims. 13
  • Classified as probably not a substantial claim. 18
  • The quote only partly states this claim. 2

  • May combine two claims.

    Findingown data

    Automating only nugget assignment results in higher per-topic average and all topics/runs Kendall's tau correlations for both V_strict and A_strict.

    “Additionally, it results in a higher per-topic average and all topics/runs Kendall's 𝜏 correlations for both 𝑉 strict and 𝐴 strict .”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The automatic relevance labels used for AutoNuggets are sufficiently accurate for nuggetization.

  • May combine two claims.

    Findingown data

    Correlations are stronger with AutoNuggets+Edits than ManualNuggets, but Kendall's tau remains quite high even with fully manual nuggets.

    “We observe stronger correlations with AutoNuggets+Edits than ManualNuggets, but Kendall's 𝜏 remains quite high even with fully manual nuggets.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manual nugget evaluations (NIST assessors) provide a reliable reference standard for evaluating RAG systems.

  • May combine two claims.

    Findingown data

    For the ManualNuggets / Manual-Assign condition, the run-level Kendall's tau correlations are 0.727 for V_strict and 0.758 for A_strict.

    “For the ManualNuggets / Manual-Assign condition (bottom row), the run-level Kendall's 𝜏 correlations drop to 0.727 for 𝑉 strict and 0.758 for 𝐴 strict .”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manual nugget evaluations (NIST assessors) provide a reliable reference standard for evaluating RAG systems.
    • Rests on an assumption: The automatic relevance labels used for AutoNuggets are sufficiently accurate for nuggetization.

  • May combine two claims.

    Findingown data

    For the AutoNuggets+Edits / ManualAssign condition, the run-level Kendall's tau correlation is 0.887 for V_strict and 0.901 for A_strict.

    “For the AutoNuggets+Edits / ManualAssign condition (top row), we observe a strong run-level Kendall's 𝜏 correlation of 0.887 for 𝑉 strict and 0.901 for 𝐴 strict .”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manual nugget evaluations (NIST assessors) provide a reliable reference standard for evaluating RAG systems.
    • Rests on an assumption: The automatic relevance labels used for AutoNuggets are sufficiently accurate for nuggetization.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The same NIST assessor performed nugget creation and nugget assignment to ensure consistency in nugget interpretation.

    “In the manual nugget assignment process, we leveraged the expertise of the original NIST assessor involved in creating the list of nuggets (i.e., the same assessor performed nugget creation and nugget assignment).”
    • Also stated in: § 4.2.2 Manual Nugget Assignment. · plain
      “By involving the same assessor responsible for nugget creation, we ensure consistency in nugget interpretation.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Study design and conductreasoning

    The evaluation scores focus exclusively on recall of nuggets in RAG answers and do not cover citation support or fluency.

    “It is important to emphasize that this work focuses exclusively on evaluating the recall of information nuggets within RAG answers.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Study design and conductreasoning

    The test queries were constructed near the submission deadline rather than reusing publicly released queries to reduce relevance judgment contamination.

    “The set of test queries was constructed near the submission deadline (late July 2024), rather than reusing queries that had already been publicly released in Rosset et al. [34].”
    • Also stated in: § 3 The Setup of the TREC 2024 RAG Track · plain
      “This strategy primarily addresses concerns about relevance judgment contamination, though we note that the corpus is likely to be already contained in LLM pretraining data given its web-based nature and open-source availability.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [34]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Voorhees demonstrated the importance of constructing nuggets to evaluate definition questions.

    “Voorhees [46] demonstrated the importance of constructing nuggets to evaluate definition questions, thereby aligning complex question answering with retrieval-augmented generation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [46]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    Nuggets are formulated at the semantic or conceptual level and may not correspond to lexical realizations in the sources.

    “It is important to note that while nuggets manifest as short natural language phrases or sentences, they are formulated at the semantic or conceptual level, and thus may or may not correspond to phrases or other lexical realizations in the sources.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    In manual nugget assignment, assessors were not shown any LLM output, making it a fully manual process.

    “It is worth clarifying that assessors in this step are not shown any LLM output, so this is a fully manual process.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The nugget evaluation methodology was originally developed for the TREC Question Answering Track in 2003 and evaluates systems based on atomic facts that should be present in good answers.

    “This approach, originally developed for the TREC Question Answering (QA) Track in 2003, evaluates systems based on atomic facts that should be present in good answers.”
    • Also stated in: § 1 Introduction · plain
      “This methodology was originally articulated more than two decades ago in the context of the TREC Question Answering (QA) Track for evaluating answers to free-form “definition questions”.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [46]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [47]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisereasoning

    Most related nugget evaluation methods were primarily validated against ad hoc retrieval tasks, so it is unclear if they can be directly applied to RAG answer evaluation.

    “As most of these methods were primarily validated against ad hoc retrieval tasks, it is unclear if they can be directly applied to RAG answer evaluation in our context.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The recent advent of LLMs has enabled more reliable automatic nugget evaluation, as noted in concurrent papers.

    “Recently, the advent of LLMs has enabled researchers and practitioners to perform more reliable nugget evaluation automatically, an observation noted in concurrent papers [1,25,42].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [1]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [25]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [42]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    Nugget assignment is performed at the semantic or conceptual level, not merely lexical matching, and requires understanding, inference, and reasoning.

    “It is important to note that the nugget assignment process is performed at the semantic or conceptual level, not merely based on lexical matching (and hence requires understanding, inference, and reasoning).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The original nugget evaluation methodology emphasized identification of nuggets, or essential facts relevant to a good answer.

    “Introduced in the TREC QA Track in 2003 [47], the nugget evaluation methodology emphasized the identification of nuggets, or essential facts relevant to a good answer.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The corpus is likely already contained in LLM pretraining data given its web-based nature and open-source availability.

    “though we note that the corpus is likely to be already contained in LLM pretraining data given its web-based nature and open-source availability.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The condition replacing manual nugget assignment with automatic assignment isolates the impact of automating only the nugget assignment step.

    “This condition isolates the impact of automating only the nugget assignment step.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    While run-level rankings are well-preserved by automatic evaluation, there is more variability at the topic level.

    “This indicates that while run-level rankings are well-preserved by our automatic evaluation, there is more variability at the topic level.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    There is a need for calibration of LLM assignment labels.

    “indicating the need for calibration.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    For aspects not directly impacted by LLMs, the paper can continue to assert findings from the literature without revalidating them.

    “For aspects that are not directly impacted by our use of LLMs, we can continue to assert findings from the literature without needing to carefully validate those claims again.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    The experiments are grounded in the MS MARCO V2.1 segment collection, reflecting real-world search with greater fidelity than Wikipedia alone.

    “Our experiments are grounded in the MS MARCO V2.1 segment collection, reflecting real-world search with greater fidelity than (for example) Wikipedia alone (the target corpus of many other studies).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Doing the same with human assessors would have been prohibitively expensive.

    “while doing the same with human assessors would have been prohibitively expensive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    For aspects of the evaluation that are not dependent on LLMs, we can simply build on existing findings.

    “For aspects of the evaluation that are not dependent on LLMs, we can simply build on existing findings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The framework supports various degrees of human involvement to provide a tradeoff between effort and quality.

    “Although the entire process can be automated with AutoNuggetizer, we support various degrees of human involvement to provide a tradeoff between effort and quality.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    While good run-level correlations are achieved, per-topic correlations are quite low, suggesting the evaluation framework is inadequate for fine-grained debugging of individual answers.

    “We emphasize that while we achieve good run-level correlations, per-topic correlations are quite low, suggesting that our evaluation framework is inadequate for fine-grained debugging of individual answers.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manual nugget evaluations (NIST assessors) provide a reliable reference standard for evaluating RAG systems.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The information retrieval literature has a long tradition of deliberate and careful meta-evaluations that validate evaluation methodologies, and much work has examined different aspects of nugget evaluations.

    “The information retrieval literature has a long tradition of deliberate and careful meta-evaluations that validate evaluation methodologies, and much work has examined different aspects of nugget evaluations.”
    Quote in the paperNot a substantial claim0 of 6 sources checked
    • cites [4]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [14]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [37]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [45]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [53]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Building on the nugget evaluation methodology dating back over two decades minimizes reinvention of the wheel.

    “by building on the nugget evaluation methodology dating back over two decades, we minimize reinvention of the wheel.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Sharifymoghaddam et al. established correlations between nugget-based and human preferences in side-by-side comparisons of LLM answers.

    “Complementing these results, Sharifymoghaddam et al. [38] established correlations between nugget-based and human preferences in side-by-side comparisons of LLM answers.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    These findings are specific to the current implementation.

    “These findings are specific to the current implementation.”
    Quote in the paperNot a substantial claim
29 of 29

  • The paper relies on this without support.

    The lack of a standardized evaluation methodology presents a barrier to continued progress in information access, NLP, and AI.

    “We feel that the evaluation of RAG output remains deficient from many perspectives, and the lack of a standardized evaluation methodology presents a barrier to continued progress in information access, and more broadly, NLP as well as AI.”

  • The paper relies on this without support.

    NIST assessors provide high-quality human annotations suitable as ground truth for validating automatic evaluation.

    “Finally, our study includes correlations against highquality human annotations from NIST assessors, to validate and calibrate automated LLM-based processes for nugget creation and assignment.”

  • The paper relies on this without support.

    The manual nugget evaluations (NIST assessors) provide a reliable reference standard for evaluating RAG systems.

    “Our goal is to obtain high rank correlations against human-based metrics.”

  • The paper relies on this without support.

    The automatic relevance labels used for AutoNuggets are sufficiently accurate for nuggetization.

    “nuggetization is based on automatically generated relevance labels”
4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, et al., 2022. Improving Language Models by Retrieving from Trillions of Tokens. In Proceedings of the 39th International Conference on Machine Learning (ICML 2022). Baltimore, Maryland, 2206-2240.
  2. Hoa Trang Dang and Jimmy Lin. 2007. Different Structures for Evaluating Answers to Complex Questions: Pyramids Won't Topple, and Neither Will Human Assessors. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL 2007). Prague, Czech Republic, 768-775.
  3. Xingyu Ji, Aditya Parameswaran, and Madelon Hulsebos. 2024. TARGET: Benchmarking Table Retrieval for Generative Tasks. In Proceedings of the NeurIPS 2024 Third Table Representation Learning Workshop.

Cited by

Sorted by citations

Related

Authors
Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., Soboroff, I., Dang, H. T., & Lin, J.
Affiliations
  • University of Waterloo (CA)
  • Snowflake Inc. (United States)
  • Microsoft (United States) (US)
  • National Institute of Standards and Technology (US)
ORCID
Venue
Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2025-07-13
Pages
180-190
DOI
10.1145/3726302.3730090
OpenAlex
W4412377058
Semantic Scholar
4c41062c41f05810bcf583ceca4a136e932345e9
Cited by
14
References
35
Citations by year
2026: 11 · 2025: 3
FWCI
13.77
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
Funders
  • Natural Sciences and Engineering Research Council (NSERC) of Canada
  • Snowflake
  • Microsoft
  • Korean Government
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
retrieval-augmented generation, RAG evaluation, fact extraction, large language models, question answering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review