Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index; none of the 28 sources its claims cite is checked for support yet.

Benchmarking LLM-based Relevance Judgment Methods

Open access
Arabzadeh, Negar, Clarke, Charles L. A. · 2025 · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval

Abstract

Large Language Models (LLMs) are increasingly deployed in both academic and industry settings to automate the evaluation of information seeking systems, particularly by generating graded relevance judgments. Several studies report Kendall τ correlations exceeding 0.85 when comparing system rankings derived from human versus LLM-generated relevance labels. Previous work on LLM-based relevance assessment has primarily focused on replicating graded human relevance judgments through various prompting strategies. However, there has been limited exploration of alternative assessment methods or comprehensive comparative studies. In this paper, we systematically compare multiple LLM-based relevance assessment methods, including binary relevance judgments, graded relevance assessments, pairwise preference-based methods, and two nugget-based evaluation methods~-~document-agnostic and document-dependent. Wherever possible, we employ state-of-the-art tools and optimized prompts tailored for these methods. In addition to a traditional comparison based on system rankings using Kendall correlations, we also examine how well LLM judgments align with human preferences, as inferred from relevance grades. We conduct extensive experiments on datasets from three TREC Deep Learning tracks 2019, 2020 and 2021 as well as the ANTIQUE dataset, which focuses on non-factoid open-domain question answering. Beyond dataset-specific results, our work offers a practical methodology for evaluating diverse LLM-based relevance assessment methods. As part of our data release, we include relevance judgments generated by both an open-source (Llama3.2b) and a commercial (gpt-4o) model. Our goal is to reproduce various LLM-based relevance judgment methods to provide a comprehensive comparison. We release all the relevance judgments as a resource that establishes a baseline for future work, ensuring a level playing field for evaluation of LLM-based relevance judgments. All code, data, and resources are publicly available in our GitHub Repository at https://github.com/Narabzad/llm-relevance-judgement-comparison

Overview

CoChat Summary

TL;DR

This paper systematically benchmarks five LLM-based relevance judgment methods—binary, graded (UMBRELA), pairwise preference, document-agnostic nugget (Exam), and document-dependent nugget (AutoNuggetizer)—on TREC DL 2019–2021 and ANTIQUE using GPT-4o and Llama 3.2B. It finds that pairwise preferences align best with human labels, while binary and UMBRELA judgments best preserve system rankings.

Key findings

  • Pairwise preference judgments consistently achieve the highest alignment with human labels: 91% average agreement for Best vs. UnAcceptable across all datasets and 71% average agreement for Best vs. Acceptable.
  • For agreement with system rankings, binary relevance is strongest on DL-20 (Kendall τ = 0.922) and DL-21 (τ = 0.904), while UMBRELA leads on DL-19 (τ = 0.920); original human labels show compatibility correlations above 0.9 (DL-19 0.953, DL-20 0.956, DL-21 0.916).
  • Nugget-based methods are variable: document-agnostic Exam-Binary tends to do better on DL-20, document-dependent methods slightly better on DL-19, but both are generally less consistent than pointwise or pairwise methods.
  • Alignment and system-ranking agreement do not always coincide: pairwise preferences have the highest human alignment but comparatively lower system-ranking agreement, whereas binary and UMBRELA have strong system-ranking agreement but not always equally high human alignment.

Methods

  • The study reproduces five LLM-based relevance assessment approaches: binary relevance, graded relevance via UMBRELA 2, pairwise preference judgments, document-agnostic nugget evaluation (Exam), and document-dependent nugget evaluation (AutoNuggetizer), using GPT-4o and Llama 3.2B with temperature 0; main paper results use GPT-4o, with Llama results in the GitHub repository.
  • Experiments use TREC Deep Learning tracks 2019 (43 queries), 2020 (45 queries), and 2021 (57 queries), plus ANTIQUE (200 queries); system-ranking analysis is conducted only on the TREC DL datasets because ANTIQUE had limited available runs.
  • Alignment with human labels is measured by pairwise category comparisons—Best vs. UnAcceptable, Acceptable vs. UnAcceptable, and Best vs. Acceptable—while agreement with system rankings uses the compatibility metric (RBO with p = 0.9) and Kendall τ correlations against nDCG@10.

Limitations

  • The paper states that Llama 3.2B results are not presented or analyzed in the paper due to space limitations, though all data, results, and analysis are included in the GitHub repository.
  • The paper states that ANTIQUE had limited available runs, so system-ranking agreement could not be studied on it; future work is needed to compare LLM-generated preference judgments with human preference judgments.
  • The text makes evident that nugget-based methods show high variability and lack consistency across datasets, binary judgments produce high tie rates and fail to capture fine-grained distinctions, and differences within the same human relevance category cannot be interpreted as either finer LLM distinctions or errors.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3726302.3730305

Cites

100 of 100 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 11 Not checked · 45 Own results and interpretations

17 more claims

Create a free account to see all 56 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 25
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 11

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 88 references

Research works printed but not linked: 7

  1. Ben Carterette, Paul N. Bennett, David Maxwell Chickering, and Susan T. Dumais. 2008. Here or there: Preference Judgments for Relevance. Computer Science Department Faculty Publication Series 46. University of Massachusetts Amherst.
  2. 10.1145/3451161 · DOI
  3. 10.1145/3451161 · DOI
  4. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2022. Overview of the TREC 2021 deep learning track. In Text REtrieval Conference (TREC). NIST, TREC. https://www.microsoft.com/en-us/research/publication/overview-of-the-trec-2021-deep-learning-track/
  5. Karen Sparck Jones and Julia R Galliers. 1995. Evaluating natural language processing systems: An analysis and review. (1995).
  6. Noriko Kando, Kazuko Kuriyama, Toshihiko Nozue, Koji Eguchi, Hiroyuki Kato, and Souichiro Hidaka. 1999. Overview of IR tasks at the first NTCIR workshop. In Proceedings of the first NTCIR workshop on research in Japanese text retrieval and term recognition. 11-44.
  7. 10.1145/2124295.2124343 · DOI

Cited by

See all 47

Metadata

Authors
Arabzadeh, N., & Clarke, C. L. A.
Affiliations
University of Waterloo (CA)
Venue
Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2025-07-13
Pages
3194-3204
OpenAlex
W4412396369
Cited by
46
References
88
Citations by year
2026: 33 · 2025: 13
FWCI
15.75
Type
conference paper
Language
en
Open access
Locations
  • International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
open-domain question answering, human preference alignment, prompt optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Benchmarking LLM-based Relevance Judgment Methods. https://app.cochat.ai/papers/10.1145/3726302.3730305
The paper
Arabzadeh, N., & Clarke, C. L. A. (2025). Benchmarking LLM-based Relevance Judgment Methods. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/3726302.3730305

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.