CoChat
Sign Up
PreprintCited by 21Information Retrieval and Search BehaviorTopic ModelingRecommender Systems and Techniques

UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor

Open access
Upadhyay, Shivani, Pradeep, Ronak, Thakur, Nandan, Craswell, Nick, Lin, Jimmy·2024·arXiv
DOI 10.48550/arxiv.2406.06519arXiv 2406.06519
Open this paper in CoChat — free accountOpen PDF

Abstract

Copious amounts of relevance judgments are necessary for the effective training and accurate evaluation of retrieval systems. Conventionally, these judgments are made by human assessors, rendering this process expensive and laborious. A recent study by Thomas et al. from Microsoft Bing suggested that large language models (LLMs) can accurately perform the relevance assessment task and provide human-quality judgments, but unfortunately their study did not yield any reusable software artifacts. Our work presents UMBRELA (a recursive acronym that stands for UMbrela is the Bing RELevance Assessor), an open-source toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o model and adds more nuance to the original paper. Across Deep Learning Tracks from TREC 2019 to 2023, we find that LLM-derived relevance judgments correlate highly with rankings generated by effective multi-stage retrieval systems. Our toolkit is designed to be easily extensible and can be integrated into existing multi-stage retrieval and evaluation pipelines, offering researchers a valuable resource for studying retrieval evaluation methodologies. UMBRELA will be used in the TREC 2024 RAG Track to aid in relevance assessments, and we envision our toolkit becoming a foundation for further innovation in the field. UMBRELA is available at https://github.com/castorini/umbrela.

AI snapshot

Written from the paper’s full text.

TL;DR

UMBRELA is an open-source toolkit that reproduces Thomas et al.’s Bing relevance assessor using OpenAI GPT-4o and DNA zero-shot prompting on TREC Deep Learning Track judgments from 2019–2023. It finds high correlation between human- and LLM-based system rankings, though label-level agreement with human qrels is only fair.

Key findings

  • Across TREC DL 2019–2023, system rankings evaluated with UMBRELA judgments correlated highly with rankings from human qrels using nDCG@10: Kendall τ ranged from 0.8728 to 0.9435 and Spearman ρ from 0.9729 to 0.9923 (Table 2, right).
  • Agreement between human and LLM labels was fair: Cohen κ was 0.3081–0.3730 on the four-scale labels and 0.4176–0.4989 on binary labels (Table 2, left).
  • Confusion matrices showed the LLM predicted non-relevant labels with roughly 75% accuracy, while accuracy dropped to roughly 50% for “related,” 30% for “highly relevant,” and 45% for “perfectly relevant” relative to human labels.
  • In a TREC DL 2019 case study, the authors found misclassifications where human label 3 passages were predicted as 0, and argued in two examples that the LLM judgment appeared more accurate due to ambiguous or incomplete human assessments.

Methods

  • The authors reassessed human qrels from TREC Deep Learning Tracks 2019–2023, originally assigned by NIST assessors on a 0–3 relevance scale. For DL 2022 and 2023, they removed near-duplicate passages using NIST clusters and retained only canonical passages before calculation.
  • UMBRELA used OpenAI GPT-4o via Microsoft Azure with a zero-shot DNA (Descriptive, Narrative, Aspects) prompt following Thomas et al.; parameters were temperature 0, top p = 1, no stopwords, frequency penalty 0.5, and presence penalty 0.
  • Evaluation compared human and LLM labels using Cohen κ (four-scale and binary) and confusion matrices, and compared system rankings using Kendall τ and Spearman ρ on nDCG@10 across TREC submissions.

Limitations

  • The paper has no formal limitations section. It reports only fair label-level agreement (Cohen κ 0.3081–0.3730 four-scale; 0.4176–0.4989 binary) and lower accuracy for related, highly relevant, and perfectly relevant labels.
  • Evident from the text: evaluation is limited to GPT-4o, TREC DL 2019–2023, and a zero-shot DNA prompt; DL 2022–23 results depend on near-duplicate preprocessing.
  • The case-study discrepancies are attributed to ambiguity or incomplete user intent and are based on manual spot-checking, not a systematic error analysis.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.06519

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

17 of 18 cited sources checked · 12 supported · 2 flagged · accuracy not yet measured

  • Interpretationcitation
    § Jimmy Lin · p. 1

    LLMs possess the ability to understand user needs and assign relevance labels with veracity comparable to human assessors.

    “As highlighted by Thomas et al. [21], LLMs possess the ability to understand user needs and assign relevance labels with veracity comparable to human assessors.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [21] [21] Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models Can Accurately Predict Searcher Preferences. In 2024 International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM. - source
    • The authors state this as an assumption: Human relevance judgments from TREC DL are reliable gold labels for evaluating retrieval systems.
  • Interpretationown data
    § Jimmy Lin · p. 1

    LLM assessment is an effective alternative to human-based assessment.

    “Our experiments with the TREC Deep Learning (DL) Tracks from 2019-2023 [4-8] verified the claims by Thomas et al. [21] about LLM assessment being an effective alternative to humanbased assessment, using OpenAI's latest model GPT-4o.”
    Quote in the paperSubstantial claim5 of 6 sources checked
    • cites [21] [21] Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models Can Accurately Predict Searcher Preferences. In 2024 International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM. - source
    • cites [4] [4] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 Deep Learning Track. arXiv:2102.07662 [cs.IR] - source
    • cites [5] [5] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2022. Overview of the TREC 2021 Deep Learning Track. In Text REtrieval Conference (TREC) . NIST, TREC. - source
    • cites [6] [6] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2023. Overview of the TREC 2022 Deep Learning Track. In Text REtrieval Conference (TREC) . NIST, TREC. -

      Not checked in the free run.

    • cites [7] [7] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the TREC 2019 Deep Learning Track. arXiv:2003.07820 [cs.IR] - source
    • cites [8] [8] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2024. Overview of the TREC 2023 Deep Learning Track. In Text REtrieval Conference (TREC) . NIST, TREC. - source
    • The authors state this as an assumption: Human relevance judgments from TREC DL are reliable gold labels for evaluating retrieval systems.
  • Findingown data
    § ABSTRACT · p. 1

    Across Deep Learning Tracks from TREC 2019 to 2023, LLM-derived relevance judgments correlate highly with rankings generated by effective multi-stage retrieval systems.

    “Across Deep Learning Tracks from TREC 2019 to 2023, we find that LLMderived relevance judgments correlate highly with rankings generated by effective multi-stage retrieval systems.”
    • Also stated in: § 4 RESULTS · p. 3 · plain
      “The high correlations among these judgment evaluations show alignment in the retrieval systems' rankings.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Human relevance judgments from TREC DL are reliable gold labels for evaluating retrieval systems.
  • Findingown data
    § 4 RESULTS · p. 3

    LLMs are able to assign appropriate labels to the majority of the judgments across all TREC DL tracks.

    “As expected, LLMs are able to assign appropriate labels to the majority of the judgments across all tracks.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Human relevance judgments from TREC DL are reliable gold labels for evaluating retrieval systems.
  • Findingown data
    § 4 RESULTS · p. 3

    We observe a high number of misclassifications in the judgments, predicted as label 0 but where their original (human) label is 3.

    “As shown in Figure 2(a), we observe a high number of misclassifications in the judgments, predicted as label 0 but where their original (human) label is 3.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 14
  • May combine two claims. 11
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Jimmy Lin · p. 1

    The zero-shot DNA prompting technique described by Kojima et al. [14] was carefully followed to reproduce the original setup as faithfully as possible.

    “In our study, we carefully follow the zero-shot DNA (Descriptive, Narrative, and Aspects) prompting technique described by Kojima et al. [14] to reproduce the original setup as faithfully as possible.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [14] [14] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large Language Models are Zero-Shot Reasoners. arXiv:2205.11916 [cs.CL] - source

      Section 3 (p. 3) identifies Kojima et al.’s technique as Zero-shot-CoT, which prompts a model with “Let’s think step by step” to elicit reasoning. Section 3.1 (pp. 3–4) describes a two-stage reasoning-and-answer procedure, not the attributed Descriptive, Narrative, and Aspects (DNA) technique. Whether the citing authors followed a procedure faithfully is their own claim, not something this source can establish. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation
    § 2 RELATED WORK · p. 1

    LLMs, when provided with proper guidance, can effectively 'understand' the relation between a given query and passage and accurately determine its relevance.

    “LLMs, when provided with proper guidance, can effectively 'understand' the relation between a given query and passage and accurately determine its relevance.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [11] [11] Guglielmo Faggioli, Laura Dietz, Charles Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth. 2023. Perspectives on Large Language Models for Relevance Judgment. arXiv:2304.09161 [cs.IR] - source

      The Test Collections section shows LLMs being prompted to judge whether a question and passage are relevant, and §7.1 says they can provide relevance judgments of a certain quality. But §5.2 reports substantial disagreement with human assessors on some pairs, while the Truthfulness & Misinformation section says it remains unclear how to translate detailed guidance into prompts. The source therefore supports the ability to make relevance judgments, not the unqualified claim that proper guidance makes them accurate. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [16] [16] Sean MacAvaney and Luca Soldaini. 2023. One-Shot Labeling for Automatic Relevance Estimation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '23) . - source

      Section 4 says the tested one-shot methods cannot reliably provide binary relevance labels. Section 4.1 reports that the prompted method, DuoPrompt, achieved an F1-score of at most 0.55 and was not accurate enough to serve as a fully automatic relevance labeler. That contradicts the claim that properly guided LLMs can accurately determine a passage’s relevance. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [21] [21] Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models Can Accurately Predict Searcher Preferences. In 2024 International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM. - source

      Section 4.1 (p. 5) concludes that, for some prompts, LLM relevance labels reached human-quality agreement on the collection; Section 2.2 (pp. 2–3) describes giving the model a query and document text with prompting guidance. The study evaluated query–document pairs, not specifically query–passage pairs, and its results varied considerably by prompt, so the passage-specific claim goes beyond what the paper establishes. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingreasoning
    § ABSTRACT · p. 1

    UMBRELA is an open-source toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o and adds more nuance to the original paper.

    “Our work presents UMBRELA (a recursive acronym that stands for UM brela is the B ing REL evance A ssessor), an open-source toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o model and adds more nuance to the original paper.”
    • Also stated in: § Jimmy Lin · p. 1 · plain
      “We reproduced results from Thomas et al. [21], which lacked reusable software components, and created UMBRELA, an opensource toolkit that we share with the broader community.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [21] [21] Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models Can Accurately Predict Searcher Preferences. In 2024 International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2 RELATED WORK · p. 2

    In the domain of text annotations in general, both proprietary and open-source LLMs have shown remarkable effectiveness.

    “In the domain of text annotations in general, both proprietary and open-source LLMs have shown remarkable effectiveness [1, 2, 13, 18, 25].”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [1] [1] Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikovo, and Fabrizio Gilardi. 2023. Open-source Large Language Models Outperform Crowd Workers and Approach ChatGPT in TextAnnotation Tasks. arXiv:2307.02179 [cs.CL] - source
    • cites [2] [2] Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Mohammadmasiha Zahedivafa, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2024. Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning. arXiv:2307.02179 [cs.CL] - source
    • cites [13] [13] Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators. arXiv:2303.16854 [cs.CL] - source

      The Experimental Setups and Experimental Results sections report that AnnoLLM, implemented with GPT-3.5, matched or surpassed crowdsourced annotators on three text-annotation tasks. The paper lists LLaMA as a zero-shot comparator on WiC, but does not establish that open-source LLMs are effective text annotators in general. Page numbers are unavailable in our copy. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation
    § 5 CONCLUSION · p. 4

    We have successfully replicated the results from Thomas et al. [21], thereby validating the capabilities of OpenAI's latest model, GPT-4o, in performing relevance assessments automatically.

    “In our research, we have successfully replicated the results from Thomas et al. [21], thereby validating the capabilities of OpenAI's latest model, GPT-4o, in performing relevance assessments automatically.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [21] [21] Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models Can Accurately Predict Searcher Preferences. In 2024 International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM. - source
    • The authors state this as an assumption: Human relevance judgments from TREC DL are reliable gold labels for evaluating retrieval systems.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Jimmy Lin · p. 1

    Relevance judgments facilitated by LLMs can be employed to 'fill holes' left by incomplete judgments, thereby contributing to a more thorough and accurate evaluation of retrieval systems.

    “In addition, relevance judgments facilitated by LLMs can be employed to 'fill holes' left by incomplete judgments, thereby contributing to a more thorough and accurate evaluation of retrieval systems [16, 24].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [16] [16] Sean MacAvaney and Luca Soldaini. 2023. One-Shot Labeling for Automatic Relevance Estimation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '23) . - source
    • cites [24] [24] Shivani Upadhyay, Ehsan Kamalloo, and Jimmy Lin. 2024. LLMs Can Patch Up Missing Relevance Judgments in Evaluation. arXiv:2405.04727 [cs.IR] - source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § ABSTRACT · p. 1

    UMBRELA is designed to be easily extensible and can be integrated into existing multi-stage retrieval and evaluation pipelines.

    “Our toolkit is designed to be easily extensible and can be integrated into existing multi-stage retrieval and evaluation pipelines, offering researchers a valuable resource for studying retrieval evaluation methodologies.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 4 RESULTS · p. 3

    LLM-derived relevance judgments show fair agreement with human assessors, as measured by Cohen kappa, across TREC DL 2019-2023.

    “These correlation values demonstrate fair agreement between human assessors and LLMs judgments.”
    • Also stated in: § 4 RESULTS · p. 3 · plain
      “Table 2 (left) presents Cohen 𝜅 scores to capture agreement between human and LLM assessments.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Human relevance judgments from TREC DL are reliable gold labels for evaluating retrieval systems.

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given
    § Jimmy Lin · p. 1

    UMBRELA takes a query and a set of passages as input, and based on the LLM configuration, attempts to 'understand' the search intent and labels the passages with different relevance grades.

    “At a high level, UMBRELA takes a query and a set of passages as input, and based on the LLM configuration, attempts to 'understand' the search intent and labels the passages with different relevance grades.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4 RESULTS · p. 3

    The LLM was able to predict non-relevant labels with roughly 75% accuracy, and for labels 'related', 'highly relevant', and 'perfectly relevant', the accuracy drops to roughly 50%, 30%, and 45%, respectively, with respect to human labels.

    “Across these datasets, the LLM was able to predict non-relevant labels with roughly 75% accuracy, whereas for labels 'related', 'highly relevant', and 'perfectly relevant', the accuracy drops to roughly 50%, 30%, and 45%, respectively (with respect to the human labels).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Human relevance judgments from TREC DL are reliable gold labels for evaluating retrieval systems.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning
    § 4 RESULTS · p. 4

    The query requests a definition for 'mechanical ventilation', but the passage provides details regarding 'Continuous Positive Airway Pressure'.

    “The query requests a definition for 'mechanical ventilation', but the passage provides details regarding 'Continuous Positive Airway Pressure'.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 3.1 DL TREC Judgments · p. 2

    The qrels for TREC 2022 and 2023 DL Tracks include relevance label propagation for near-duplicate passages, so the number of relevance judgments actually assigned by humans is much smaller than Table 1 suggests.

    “The qrels for the TREC 2022 and 2023 Deep Learning Tracks include relevance label propagation for near-duplicate passages that exist within the corpus. Thus, the number of relevance judgments that were actually assigned by humans is much smaller than the statistics in Table 1 suggest (we provide figures both with and without de-duplication).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning
    § 4 RESULTS · p. 4

    The passage mainly talks about the Thai flag but the query asks about the 'regular life of Thai people'.

    “The passage mainly talks about the Thai flag but the query asks about the 'regular life of Thai people'.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § 5 CONCLUSION · p. 4

    We have repackaged this methodology in an open-source tool, referred to as UMBRELA, that we share with the community.

    “Finally, we have repackaged this methodology in an open-source tool, referred to as UMBRELA, that we share with the community.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4 RESULTS · p. 4

    We notice many judgments similarly failing the classification for this query.

    “We notice many judgments similarly failing the classification for this query.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § 3.2 Prompting Details · p. 2

    The descriptive and narrative sections of the DNA prompt help the LLM understand the user query and the passage, whereas the aspects section details the step-by-step procedure for guiding the LLM's 'thinking process'.

    “The descriptive and narrative sections help the LLM understand the user query and the passage that it is supposed to label, whereas the aspects section details the step-by-step procedure for guiding the LLM's 'thinking process'.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4 RESULTS · p. 4

    In the Thai flag case, the LLM judgment appears to be more accurate than the human-assigned label.

    “In this case, the LLM judgment appears to be more accurate.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Human relevance judgments from TREC DL are reliable gold labels for evaluating retrieval systems.

  • May combine two claims.

    Interpretationreasoning
    § 4 RESULTS · p. 4

    Both the Thai flag and mechanical ventilation cases demonstrate ambiguity that could have been the result of either inaccurate assessments or incomplete information regarding the user intent.

    “Both the above-mentioned cases demonstrate ambiguity that could have been the result of either inaccurate assessments or incomplete information regarding the user intent.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Human relevance judgments from TREC DL are reliable gold labels for evaluating retrieval systems.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 2 RELATED WORK · p. 2

    The enhanced capabilities of LLMs and various prompting techniques have presented a strong substitute for manual text annotations.

    “The enhanced capabilities of LLMs and various prompting techniques have presented a strong substitute for manual text annotations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § Jimmy Lin · p. 1

    LLMs may actually outperform assessors with limited knowledge, who might not fully grasp the search intent, thereby leading to imprecise relevance judgments.

    “Moreover, LLMs may actually 'outperform' assessors with limited knowledge, who might not fully grasp the search intent, thereby leading to imprecise relevance judgments.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Human relevance assessment is time-consuming, labor-intensive, costly, and often inaccurate and misaligned with searcher intent.
20 of 20

  • The authors state this as an assumption.

    Human relevance judgments from TREC DL are reliable gold labels for evaluating retrieval systems.

    “Thus, we consider these as 'gold' labels.”

  • The paper relies on this without support.

    Human relevance assessment is time-consuming, labor-intensive, costly, and often inaccurate and misaligned with searcher intent.

    “However, human assessments can be time-consuming, laborintensive, and costly. Even after allocating these resources, assessments often remain inaccurate and misaligned with the searcher's intentions, particularly in the case where the information need did not arise from the assessor.”
2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. [5][5] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2022. Overview of the TREC 2021 Deep Learning Track. In Text REtrieval Conference (TREC) . NIST, TREC. -
  2. [6][6] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2023. Overview of the TREC 2022 Deep Learning Track. In Text REtrieval Conference (TREC) . NIST, TREC. -

Cited by

Sorted by citations

Related

Authors
Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., & Lin, J.
Affiliations
  • University of Waterloo (CA)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2406.06519
arXiv
2406.06519
OpenAlex
W4399554477
Semantic Scholar
d6077b167684567cfc35017709872b298a96a92d
Cited by
16
References
19
Citations by year
2026: 8 · 2025: 8
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Canada First Research Excellence Fund
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
relevance assessment, large language models, GPT-4o, open-source software toolkit, multi-stage retrieval, information retrieval evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review