The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index; none of the 28 sources its claims cite is checked for support yet.
Benchmarking LLM-based Relevance Judgment Methods
Open accessAbstract
Large Language Models (LLMs) are increasingly deployed in both academic and industry settings to automate the evaluation of information seeking systems, particularly by generating graded relevance judgments. Several studies report Kendall τ correlations exceeding 0.85 when comparing system rankings derived from human versus LLM-generated relevance labels. Previous work on LLM-based relevance assessment has primarily focused on replicating graded human relevance judgments through various prompting strategies. However, there has been limited exploration of alternative assessment methods or comprehensive comparative studies. In this paper, we systematically compare multiple LLM-based relevance assessment methods, including binary relevance judgments, graded relevance assessments, pairwise preference-based methods, and two nugget-based evaluation methods~-~document-agnostic and document-dependent. Wherever possible, we employ state-of-the-art tools and optimized prompts tailored for these methods. In addition to a traditional comparison based on system rankings using Kendall correlations, we also examine how well LLM judgments align with human preferences, as inferred from relevance grades. We conduct extensive experiments on datasets from three TREC Deep Learning tracks 2019, 2020 and 2021 as well as the ANTIQUE dataset, which focuses on non-factoid open-domain question answering. Beyond dataset-specific results, our work offers a practical methodology for evaluating diverse LLM-based relevance assessment methods. As part of our data release, we include relevance judgments generated by both an open-source (Llama3.2b) and a commercial (gpt-4o) model. Our goal is to reproduce various LLM-based relevance judgment methods to provide a comprehensive comparison. We release all the relevance judgments as a resource that establishes a baseline for future work, ensuring a level playing field for evaluation of LLM-based relevance judgments. All code, data, and resources are publicly available in our GitHub Repository at https://github.com/Narabzad/llm-relevance-judgement-comparison
Overview
CoChat Summary
TL;DR
This paper systematically benchmarks five LLM-based relevance judgment methods—binary, graded (UMBRELA), pairwise preference, document-agnostic nugget (Exam), and document-dependent nugget (AutoNuggetizer)—on TREC DL 2019–2021 and ANTIQUE using GPT-4o and Llama 3.2B. It finds that pairwise preferences align best with human labels, while binary and UMBRELA judgments best preserve system rankings.
Key findings
- Pairwise preference judgments consistently achieve the highest alignment with human labels: 91% average agreement for Best vs. UnAcceptable across all datasets and 71% average agreement for Best vs. Acceptable.
- For agreement with system rankings, binary relevance is strongest on DL-20 (Kendall τ = 0.922) and DL-21 (τ = 0.904), while UMBRELA leads on DL-19 (τ = 0.920); original human labels show compatibility correlations above 0.9 (DL-19 0.953, DL-20 0.956, DL-21 0.916).
- Nugget-based methods are variable: document-agnostic Exam-Binary tends to do better on DL-20, document-dependent methods slightly better on DL-19, but both are generally less consistent than pointwise or pairwise methods.
- Alignment and system-ranking agreement do not always coincide: pairwise preferences have the highest human alignment but comparatively lower system-ranking agreement, whereas binary and UMBRELA have strong system-ranking agreement but not always equally high human alignment.
Methods
- The study reproduces five LLM-based relevance assessment approaches: binary relevance, graded relevance via UMBRELA 2, pairwise preference judgments, document-agnostic nugget evaluation (Exam), and document-dependent nugget evaluation (AutoNuggetizer), using GPT-4o and Llama 3.2B with temperature 0; main paper results use GPT-4o, with Llama results in the GitHub repository.
- Experiments use TREC Deep Learning tracks 2019 (43 queries), 2020 (45 queries), and 2021 (57 queries), plus ANTIQUE (200 queries); system-ranking analysis is conducted only on the TREC DL datasets because ANTIQUE had limited available runs.
- Alignment with human labels is measured by pairwise category comparisons—Best vs. UnAcceptable, Acceptable vs. UnAcceptable, and Best vs. Acceptable—while agreement with system rankings uses the compatibility metric (RBO with p = 0.9) and Kendall τ correlations against nDCG@10.
Limitations
- The paper states that Llama 3.2B results are not presented or analyzed in the paper due to space limitations, though all data, results, and analysis are included in the GitHub repository.
- The paper states that ANTIQUE had limited available runs, so system-ranking agreement could not be studied on it; future work is needed to compare LLM-generated preference judgments with human preference judgments.
- The text makes evident that nugget-based methods show high variability and lack consistency across datasets, binary judgments produce high tie rates and fail to capture fine-grained distinctions, and differences within the same human relevance category cannot be interpreted as either finer LLM distinctions or errors.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 11 Not checked · 45 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Methods generally achieve higher agreement rates when the relevance distinction between categories is more pronounced (the leftmost column).”
§ Benchmarking LLM-based Relevance Judgment Methods > 5.2 Alignment with Human Labels
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, nugget-based methods display high variability, often excelling on one dataset while underperforming on another, suggesting a lack of consistency.”
§ Benchmarking LLM-based Relevance Judgment Methods > 5.2 Alignment with Human Labels
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Binary relevance, especially in Best vs. Acceptable comparisons, shows the highest average tie rate (77%), likely because both answers meet a broad relevance threshold.”
§ Benchmarking LLM-based Relevance Judgment Methods > 5.2 Alignment with Human Labels
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The original human labels demonstrated consistently high compatibility, with Kendall's tau values exceeding 0.9 across all three datasets”
§ Benchmarking LLM-based Relevance Judgment Methods > 5.3 Agreement with System Rankings
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in the Best vs. UnAcceptable comparison, preferences achieves an average agreement of 91% across all datasets.”
§ Benchmarking LLM-based Relevance Judgment Methods > 5.2 Alignment with Human Labels
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
17 more claims
Create a free account to see all 56 claims and every source verdict.
- It may state two separate claims as one. 25
- The quote only partly states this claim. 1
- Probably not a substantial claim. 11
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Benchmarking LLM-based Relevance Judgment Methods > 5.3 Agreement with System Rankings
“identified as the top three bestperforming methods in Table 1.”
§ Benchmarking LLM-based Relevance Judgment Methods > 6 Concluding Remarks
“Table: 1 Table 1 : Scoring metrics for nugget evaluation.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Benchmarking LLM-based Relevance Judgment Methods > 5.3 Agreement with System Rankings
“Figure 2 presents scatter plots of all submitted runs from DL-20 and DL-21 for these three methods.”
§ Benchmarking LLM-based Relevance Judgment Methods > 5.3 Agreement with System Rankings
“we present scatter plots for Binary, UMBRELA, Exam-Graded max , Nuggets A and Pairwise Preferences”
Citations
Cites
(1999) · Cited by 12,854
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
(1996) · Institutional Research Information System University of Turin (University of Turin) · doi:10.1111/j · Cited by 5,486
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 1×
“Human preference labels are also central to the reinforcement learning from human feedback (RLHF) process used to tune LLMs and other models [21].”
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z. Y. A., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., et al. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.138 · Cited by 943
A similarity measure for indefinite rankings
Webber, W., Moffat, A., & Zobel, J. (2010) · ACM Transactions on Information Systems · doi:10.1145/1852102.1852106 · Cited by 781
Retrieval evaluation with incomplete information
Buckley, C., & Voorhees, E. M. (2004) · Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1008992.1009000 · Cited by 736
In Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715
Research works printed but not linked: 7
- Ben Carterette, Paul N. Bennett, David Maxwell Chickering, and Susan T. Dumais. 2008. Here or there: Preference Judgments for Relevance. Computer Science Department Faculty Publication Series 46. University of Massachusetts Amherst.
- 10.1145/3451161 · DOI
- 10.1145/3451161 · DOI
- Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2022. Overview of the TREC 2021 deep learning track. In Text REtrieval Conference (TREC). NIST, TREC. https://www.microsoft.com/en-us/research/publication/overview-of-the-trec-2021-deep-learning-track/
- Karen Sparck Jones and Julia R Galliers. 1995. Evaluating natural language processing systems: An analysis and review. (1995).
- Noriko Kando, Kazuko Kuriyama, Toshihiko Nozue, Koji Eguchi, Hiroyuki Kato, and Souichiro Hidaka. 1999. Overview of IR tasks at the first NTCIR workshop. In Proceedings of the first NTCIR workshop on research in Japanese text retrieval and term recognition. 11-44.
- 10.1145/2124295.2124343 · DOI
Cited by
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z. Y. A., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., et al. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.138 · Cited by 943
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.16594 · Cited by 271
A Survey of Vibe Coding with Large Language ModelsPreprint
Ge, Y., Mei, L., Duan, Z., Li, T., Zheng, Y., Wang, Y., Wang, L., Yao, J., Liu, T., Cai, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.12399 · Cited by 96
Criteria-Based LLM Relevance Judgments
Farzi, N., & Dietz, L. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744591 · Cited by 35
Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigation
Boytsov, L., Akinpelu, D., Katyal, N., Lin, T., Gao, F., Zhao, Y., Huang, J., & Nyberg, E. (2025) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2025.ijcnlp-long.46 · Cited by 18
Metadata
- Authors
- Arabzadeh, N., & Clarke, C. L. A.
- Affiliations
- University of Waterloo (CA)
- Venue
- Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2025-07-13
- Pages
- 3194-3204
- OpenAlex
- W4412396369
- Cited by
- 46
- References
- 88
- Citations by year
- 2026: 33 · 2025: 13
- FWCI
- 15.75
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- open-domain question answering, human preference alignment, prompt optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Benchmarking LLM-based Relevance Judgment Methods. https://app.cochat.ai/papers/10.1145/3726302.3730305
- The paper
- Arabzadeh, N., & Clarke, C. L. A. (2025). Benchmarking LLM-based Relevance Judgment Methods. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/3726302.3730305
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.