A Large-Scale Study of Relevance Assessments with Large Language Models Using UMBRELA
Abstract
There is substantial interest in applying large language models (LLMs) to provide relevance assessments in information retrieval (IR) applications from both industry and academia. To date, researchers and practitioners have presented several studies, but many questions remain. In this paper, we examine four different relevance assessment strategies: a fully manual process and three variants that rely on LLMs to different extents using our tool called UMBRELA. These were deployed in the TREC 2024 RAG Track on a diverse set of 77 runs from 19 teams in situ, which allowed us to correlate system rankings induced by the different approaches and to characterize tradeoffs between cost and quality. We find that system rankings produced by the three LLM-based strategies correlate well at the run level with those produced by fully manual assessments in terms of nDCG@20, nDCG@100, and Recall@100. On a topic-by-topic basis, the correlations are lower, and results using our setup indicate that increased human involvement does not improve correlations sufficiently to justify their costs. Our study suggests that LLMs can potentially replace fully manual judgments to measure run-level effectiveness in a coarse-grained manner.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408 · see citing works
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · see citing works
MS MARCO: A Human Generated MAchine Reading COmprehension DatasetPreprint
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T. G., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.giscience.2025.5 · Cited by 1,190 · see citing works
Self-citationRetrieval evaluation with incomplete information
Buckley, C., & Voorhees, E. M. (2004) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1008992.1009000 · Cited by 742 · see citing works
How reliable are the results of large-scale information retrieval experiments?
Zobel, J. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291014 · Cited by 579 · see citing works
Evaluating evaluation measure stability
Buckley, C., & Voorhees, E. M. (2000) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/345508.345543 · Cited by 399 · see citing works
Information retrieval system evaluation
Sanderson, M., & Zobel, J. (2005) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1076034.1076064 · Cited by 369 · see citing works
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · see citing works
Variations in relevance judgments and the measurement of retrieval effectiveness
Voorhees, E. M. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291017 · Cited by 303 · see citing works
Community-based bayesian aggregation models for crowdsourcing
Venanzi, M., Guiver, J., Kazai, G., Kohli, P., & Shokouhi, M. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2567989 · Cited by 247 · see citing works
Perspectives on Large Language Models for Relevance Judgment
Faggioli, G., Dietz, L., Clarke, C. L. A., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., Stein, B., et al. (2023) · Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval · doi:10.1145/3578337.3605136 · Cited by 184 · see citing works
Large Language Models can Accurately Predict Searcher Preferences
Thomas, P., Spielman, S., Craswell, N., & Mitra, B. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657707 · Cited by 168 · see citing works
Self-citationRelevance assessments and retrieval system evaluation
Lesk, M., & Salton, G. (1968) · Information Processing & Management · doi:10.1016/0020-0271(68)90029-6 · Cited by 144 · see citing works
InPars: Unsupervised Dataset Generation for Information Retrieval
Bonifacio, L., Abonizio, H., Fadaee, M., & Nogueira, R. F. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531863 · Cited by 108 · see citing works
Information Retrieval Evaluation
Harman, D. K. (2011) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-02276-0 · Cited by 103 · see citing works
Information Retrieval Evaluation
Harman, D. K. (2011) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00368ed1v01y201105icr019 · Cited by 92 · see citing works
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · arXiv · doi:10.48550/arxiv.2305.13281 · Cited by 85 · see citing works
One-Shot Labeling for Automatic Relevance Estimation
MacAvaney, S., & Soldaini, L. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3592032 · Cited by 65 · see citing works
Questioning the Survey Responses of Large Language Models
Dominguez-Olmedo, R., Hardt, M., & Mendler-Dünner, C. (2023) · arXiv · doi:10.48550/arxiv.2306.07951 · Cited by 48 · see citing works
Alaofi, M., Thomas, P., Scholer, F., & Sanderson, M. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698431 · Cited by 35 · see citing works
AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion PredictionPreprint
Kim, J. W., & Lee, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.09620 · Cited by 32 · see citing works
Evaluation of resources for question answering evaluation
Lin, J. (2005) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1076034.1076102 · Cited by 26 · see citing works
Self-citationLanguage Models Trained on Media Diets Can Predict Public Opinion
Chu, E., Andreas, J., Ansolabehere, S., & Roy, D. (2023) · arXiv · doi:10.48550/arxiv.2303.16779 · Cited by 24 · see citing works
Voorhees, E. M. (2003) · doi:10.3115/1073445.1073479 · Cited by 21 · see citing works
UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., & Lin, J. (2024) · arXiv · doi:10.48550/arxiv.2406.06519 · Cited by 16 · see citing works
Self-citationPassage-based query refinement
Cormack, G. V., Clarke, C. L. A., Palmer, C. R., et al. (2000) · Information Processing & Management · doi:10.1016/s0306-4573(99)00048-5 · Cited by 13 · see citing works
Investigating Data Contamination in Modern Benchmarks for Large Language Models
Deng, C., Zhao, Y., Tang, X., Gerstein, M., & Cohan, A. (2023) · arXiv · doi:10.48550/arxiv.2311.09783 · Cited by 8 · see citing works
Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents
Rosset, C., Chung, H.-L., Qin, G., Chau, E. C., Feng, Z., Awadallah, A., Neville, J., & Rao, N. (2024) · arXiv · doi:10.48550/arxiv.2402.17896 · Cited by 6 · see citing works
LLMs Can Patch Up Missing Relevance Judgments in Evaluation
Upadhyay, S., Kamalloo, E., & Lin, J. (2024) · arXiv · doi:10.48550/arxiv.2405.04727 · Cited by 4
Self-citationCan We Use Large Language Models to Fill Relevance Judgment Holes?Preprint
Abbasiantaeb, Z., Meng, C., Azzopardi, L. A., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.05600 · Cited by 4 · see citing works
Query Performance Prediction using Relevance Judgments Generated by Large Language Models
Meng, C., Arabzadeh, N., Askari, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.01012
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268
Self-citation
Flagged first, then by citations
Research works printed but not linked: 11
- Negar Arabzadeh and Charles L. A. Clarke. 2024. A Comparison of Methods for Evaluating Generative IR. arXiv:2404.04044 (2024). · arXiv
- Charles L. A. Clarke and Laura Dietz. 2025. LLM-based Relevance Assessment Still Can't Replace Human Relevance Assessment. arXiv:2412.17156 (2025). · arXiv
- Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2022. Overview of the TREC 2022 Deep Learning Track. In Proceedings of the Thirty-First Text REtrieval Conference (TREC 2022). Gaithersburg, Maryland.
- Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2023. Overview of the TREC 2023 Deep Learning Track. In Proceedings of the Thirty-Second Text REtrieval Conference (TREC 2023). Gaithersburg, Maryland.
- Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2023. Promptagator: Few-shot Dense Retrieval From 8 Examples. In Proceedings of the 11th International Conference on Learning Representations (ICLR 2023).
- Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. GPTScore: Evaluate as You Desire. arXiv:2302.04166 (2023). · arXiv
- Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like Summarization Evaluation with ChatGPT. arXiv:2304.02554 (2023). · arXiv
- Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, and Jimmy Lin. 2025. Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track. In Proceedings of the 47th European Conference on Information Retrieval (ECIR 2025), Part I. Lucca, Italy, 132-148.
- Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L. A. Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Faggioli. 2024. LLMJudge: LLMs for Relevance Judgments. arXiv:2408.08896 (2024). · arXiv
- Ian Soboroff. 2024. Don't Use LLMs to Make Relevance Judgments. arXiv:2409.15133 (2024). · arXiv
- Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. 2024b. A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look. arXiv:2411.08275 (2024). · arXiv
Cited by
Sorted by citations
Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Reranking
Fang, H., Tao, S., Chen, N., Chang, K.-X., & Sakai, T. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769493 · Cited by 6 · see citing works
Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?
Dietz, L., Li, B., Yang, E., Lawrie, D., Walden, W. G., & Mayfield, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_37 · Cited by 4 · see citing works
The Effect of Document Summarization on LLM-Based Relevance Judgments
Mohtadi, S., Roitero, K., Mizzaro, S., & Demartini, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_5 · Cited by 3 · see citing works
Query–Document Dense Vectors for LLM Relevance Judgment Bias Analysis
Mohtadi, S., & Demartini, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_6 · Cited by 1 · see citing works
Revisiting Human-vs-LLM Judgments Using the TREC Podcast Track
Mansour, W. A., Culpepper, J. S., Mackenzie, J., & Yates, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_34
Prompt Perturbation for reliable LLM evaluation over comparison graphs
Huang, D., Sun, J., & Yang, P. (2026) · Statistical Learning and Data Science · doi:10.1016/j.slads.2026.100022
When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessment
Yu, C., Li, H., Zuccon, G., Mackenzie, J., & Leelanupab, T. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809905
“Upadhyay et al. [27] showed that LLM-generated judgments can replace traditional human labels at the run level in the TREC 2024 RAG Track, with limited benefit from additional human-in-the-loop intervention.”
Revisiting BM25 Feedback Models using HyDE
Jedidi, N., & Lin, J. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809889
“For metrics, we report Recall@20 to reflect a common use of HyDE within retrieval-augmented generation systems, e.g., top-20 documents retrieved are fed into the LLM for answer generation [17].”
Self-citationOverview of the CLEF 2026 LongEval Lab on Longitudinal Evaluation of Model Performance
Bouaraba, S., Breuer, T., Cancellieri, M., El-Ebshihy, A., Fröbe, M., Galuščáková, P., Goeuriot, L., Iturra-Bocaz, G., Keller, J., Knoth, P., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39150-6_32
Generative AI-Based Uncertainty-Aware IR Evaluation with Distributional Evaluation
Nunzio, G. M. D., Faggioli, G., Marchesin, S., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39150-6_3
ChemSafe: Hybrid Retrieval-Augmented Decision Support for Laboratory Chemical Hazard Assessment
Elgedawy, R., Das, S. K., Seefried, E., Wiggins, G. M., Burchfield, R., Jeskie, K., Fiscor, S., Pramanik, S., Thomas, B., Schnell, C., et al. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846505
“For scalable evaluation, the LLM-as-judge framework [8, 20] has been validated at scale as a reliable substitute for manual relevance assessment [16].”
- Authors
- Upadhyay, S., Pradeep, R., Thakur, N., Campos, D., Craswell, N., Soboroff, I., & Lin, J.
- Affiliations
- University of Waterloo (CA)
- Snowflake Inc. (United States)
- Microsoft (United States) (US)
- National Institute of Standards and Technology (US)
- ORCID
- Venue
- Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR)
- Publisher
- ACM
- Published
- 2025-07-18
- Pages
- 358-368
- OpenAlex
- W4412673545
- Cited by
- 11
- References
- 17
- Citations by year
- 2026: 10 · 2025: 1
- FWCI
- 12.93
- Type
- conference paper
- Language
- en
- Open access
- closed
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- relevance assessment, large language models, retrieval-augmented generation, NDCG, cost-quality trade-off
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
