The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Large Language Models can Accurately Predict Searcher Preferences
Open accessAbstract
Much of the evaluation and tuning of a search system relies on relevance labels---annotations that say whether a document is useful for a given search and searcher. Ideally these come from real searchers, but it is hard to collect this data at scale, so typical experiments rely on third-party labellers who may or may not produce accurate annotations. Label quality is managed with ongoing auditing, training, and monitoring. We discuss an alternative approach. We take careful feedback from real searchers and use this to select a large language model (LLM), and prompt, that agrees with this feedback; the LLM can then produce labels at scale. Our experiments show LLMs are as accurate as human labellers and as useful for finding the best systems and hardest queries. LLM performance varies with prompt features, but also varies unpredictably with simple paraphrases. This unpredictability reinforces the need for high-quality "gold" labels.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, when provided with proper guidance, can effectively 'understand' the relation between a given query and passage and accurately determine its relevance.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Our experiments with the TREC Deep Learning (DL) Tracks from 2019-2023 [4-8] verified the claims by Thomas et al. [21] about LLM assessment being an effective alternative to humanbased assessment, using OpenAI's latest model GPT-4o.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“Our work presents UMBRELA (a recursive acronym that stands for UM brela is the B ing REL evance A ssessor), an open-source toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o model and adds more nuance to the original paper.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“As highlighted by Thomas et al. [21], LLMs possess the ability to understand user needs and assign relevance labels with veracity comparable to human assessors.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“In our research, we have successfully replicated the results from Thomas et al. [21], thereby validating the capabilities of OpenAI's latest model, GPT-4o, in performing relevance assessments automatically.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 2×
“3 Lastly, the ecological costs of these LLMs are still heavily debated [Bender et al. 2021; Bommasani et al. 2021; Dodge et al. 2022; Patterson et al. 2022, 2021; Wu et al. 2022] but represent an important aspect in which these models should”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Learning to Rank for Information Retrieval
Liu, T. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,715
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 1×
“Language models are known to reproduce and amplify harmful stereotypes and biases of social import [Bender et al. 2021; Blodgett et al. 2020; Bolukbasi et al. 2016; Caliskan et al. 2017; Gonen and Goldberg 2019] and therefore there is an immediate need to study if and how these biases may also manifest in relevance la…”
BackgroundG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
A similarity measure for indefinite rankings
Webber, W., Moffat, A., & Zobel, J. (2010) · ACM Transactions on Information Systems · doi:10.1145/1852102.1852106 · Cited by 864
OHSUMED: An Interactive Retrieval Evaluation and New Large Test Collection for Research
Hersh, W., Buckley, C., Leone, T. J., & Hickam, D. H. (1994) · Springer eBooks · doi:10.1007/978-1-4471-2099-5_20 · Cited by 787
Research works printed but not linked: 26
- Aashish Agarwal, Ankita Mandal, Matthias Schaffeld, Fangzheng Ji, Jhiao Zhan, Yiqi Sun, and Ahmet Aker. 2019. Good, neutral or bad news classification. In Proceedings of the Third International Workshop on Recent Trends in News Information Retrieval. 9--14.
- Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikovo, and Fabrizio Gilardi. 2023. Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks. arXiv:2307.02179 [cs.CL] · arXiv
- Valeria Bolotova, Vladislav Blinov, Yukun Zheng,WBruce Croft, Falk Scholer, and Mark Sanderson. 2020. Do people and neural nets pay attention to the samewords: studying eye-tracking data for non-factoid QA evaluation. In Proceedings of the ACM International Conference on Information and Knowledge Management. 85--94.
- Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasingword embeddings. Advances in neural information processing systems 29 (2016).
- Andrei Broder. 2002. A taxonomy of web search. In ACM Sigir forum, Vol. 36. ACM New York, NY, USA, 3--10.
- Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183--186.
- Ben Carterette, Paul N Bennett, David Maxwell Chickering, and Susan T Dumais. 2008. Here or there: Preference judgments for relevance. In Proceedings of the European Conference on Information Retrieval. 16--27.
- K. Alec Chrystal and Paul D. Mizen. 2001. Goodhart's law: Its origins, meaning and implications for monetary policy. Prepared for the Festschrift in honour of Charles Goodhart.
- Charles LAClarke Gianluca Demartini Laura Dietz Guglielmo Faggioli Matthias Hagen Claudia Hauff Noriko Kando Evangelos Kanoulas Martin Potthast Ian Soboroff Benno Stein and Henning Wachsmuth. 2023. HMC: A spectrum of human--machine-collaborative relevance judgment frameworks. In Frontiers of Information Access Experimentation for Research and Education Christine Bauer Ben Carterette Nicola Ferro and Norbert Fuhr (Eds.). Vol. 13. Leibniz-Zentrum für Informatik. Issue 1.
- Tadele T. Damessie, Taho P. Nghiem, Falk Scholer, and J. Shane Culpepper. 2017. Gauging the quality of relevance assessments using inter-rater agreement. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval.
- Susan Dumais, Robin Jeffries, , Daniel M. Russell, Diane Tang, and Jaime Teevan. 2014. Understanding user behavior through log data and analysis. In Ways of knowing in HCI, Judith S. Olson andWendy A. Kellogg (Eds.). Springer, New York, 349--372.
- Charles A E Goodhart. 1975. Problems of monetary management: The UK experience. In Papers in Monetary Economics. Vol. 1. Reserve Bank of Australia.
- Google LLC. 2022. General Guidelines. https://guidelines.raterhub.com/ searchqualityevaluatorguidelines.pdf, Downloaded 29 July 2023..
- Geoffrey Hinton Oriol Vinyals and Jeff Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation LearningWorkshop. http://arxiv.org/abs/1503.02531 · arXiv
- Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, YuhuaiWu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, ChristopherD. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, JueWang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic evaluation of language models. arXiv:2211.09110 [cs.CL] · arXiv
- OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] · arXiv
- David Patterson Joseph Gonzalez Quoc Le Chen Liang Lluis-Miquel Munguia Daniel Rothchild David So Maud Texier and Jeff Dean. 2021. Carbon emissions and large neural network training. (2021). arXiv:2104.10350 [cs.LG] · arXiv
- Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with ?gradient descent" and beam search. arXiv:2305.03495 · arXiv
- Rachel L. Thomas and David Uminsky. 2022. Reliance on metrics is a fundamental challenge for AI. Patterns 3, 5 (2022).
- EllenMVoorhees. 1998. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. 315--323.
- EllenMVoorhees. 2004. Overview of the TREC 2004 Robust Retrieval Track. In Proceedings of the Text REtrieval Conference.
- Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, DavidWadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How far can camels go? Exploring the state of instruction tuning on open resources. arXiv:2306.04751 [cs.CL] · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. arXiv:2201.11903 [cs.CL] · arXiv
- Chengrun Yang Xuezhi Wang Yifeng Lu Hanxiao Liu Quoc V. Le Denny Zhou and Xinyun Chen. 2023. Large language models as optimisers. arXiv:2309.03409 [cs.LG] · arXiv
- Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. 2022. TEMPERA: Test-time prompt editing via reinforcement learning. arXiv:2211.11890 [cs.CL] · arXiv
- Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv:2211.01910 [cs.LG] · arXiv
Cited by
Sorted by citations
AI–Human Hybrids for Marketing Research: Leveraging Large Language Models (LLMs) as Collaborators
Arora, N., Chakraborty, I., & Nishimura, Y. (2024) · Journal of Marketing · doi:10.1177/00222429241276529 · Cited by 211
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 86
Advancing the search frontier with AI agents
White, R. W. (2024) · Communications of the ACM · doi:10.1145/3655615 · Cited by 55
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47
A survey on privacy risks and protection in large language models
Chen, K., Zhou, X., Lin, Y., Feng, S., Shen, L., & Wu, P. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00177-1 · Cited by 67
Alaofi, M., Thomas, P., Scholer, F., & Sanderson, M. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698431 · Cited by 47
Self-citationFrom Prompt Engineering to Prompt Science with Humans in the Loop
Shah, C. S. (2025) · Communications of the ACM · doi:10.1145/3709599 · Cited by 33
Principles and Guidelines for the Use of LLM Judges
Dietz, L., Zendel, O., Bailey, P., Clarke, C. L. A., Cotterill, E., Dalton, J., Hasibi, F., Sanderson, M., & Craswell, N. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744588 · Cited by 50
Self-citationUsing Large Language Models to Generate, Validate, and Apply User Intent Taxonomies
Shah, C., White, R., Andersen, R., Buscher, G., Counts, S., Das, S., Montazer, A., Manivannan, S., Neville, J., Rangan, N., et al. (2025) · ACM Transactions on the Web · doi:10.1145/3732294 · Cited by 33
Query Performance Prediction Using Relevance Judgments Generated by Large Language Models
Meng, C., Arabzadeh, N., Askari, A., Aliannejadi, M., & de Rijke, M. (2025) · ACM Transactions on Information Systems · doi:10.1145/3736402 · Cited by 61
Balog, K., Metzler, D., & Qin, Z. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730348 · Cited by 41
Chen, N., Liu, J., Dong, X., Liu, Q., Sakai, T., & Wu, X.-M. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698420 · Cited by 20
Ortloff, A.-M., Martius, F., Meier, M., Sans-Raimbault, T., Geierhaas, L., & Smith, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713671 · Cited by 26
Exploring the Potential of Large Language Models in Self-adaptive Systems
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z. G., & Tei, K. (2024) · International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) · doi:10.1145/3643915.3644088 · Cited by 23
UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., & Lin, J. (2024) · arXiv · doi:10.48550/arxiv.2406.06519 · Cited by 16
“Our experiments with the TREC Deep Learning (DL) Tracks from 2019–2023 [4–8] verified the claims by Thomas et al. [21] about LLM assessment being an effective alternative to human-based assessment, using OpenAI’s latest model GPT-4o.”
BackgroundMethodInfluentialSelf-citationSociotechnical Implications of Generative Artificial Intelligence for Information Access
Mitra, B., Cramer, H. S. M., & Gurevich, O. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_7 · Cited by 28
Self-citationAn Investigation of Prompt Variations for Zero-Shot LLM-Based Rankers
Sun, S., Zhuang, S., Wang, S., & Zuccon, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88711-6_12 · Cited by 25
Generative Information Retrieval Evaluation
Alaofi, M., Arabzadeh, N., Clarke, C. L. A., & Sanderson, M. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_6 · Cited by 34
ChatGPT Goes Shopping: LLMs Can Predict Relevance in eCommerce Search
Soviero, B., Kuhn, D., Salle, A., & Moreira, V. P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56066-8_1 · Cited by 15
Goodbye human annotators? Content analysis of social policy debates using ChatGPT
Gielens, E., Sowula, J., & Leifeld, P. (2025) · Journal of Social Policy · doi:10.1017/s0047279424000382 · Cited by 18
Software testing for extended reality applications: a systematic mapping study
Gu, R., Rojas, J. M., & Shin, D. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00523-7 · Cited by 18
Yi, F., Singh, A., & Tao, Z. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000101 · Cited by 13
Conversational Gold: Evaluating Personalized Conversational Search System Using Gold Nuggets
Abbasiantaeb, Z., Lupart, S., Azzopardi, L. A., Dalton, J., & Aliannejadi, M. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730316 · Cited by 15
A Large-Scale Study of Relevance Assessments with Large Language Models Using UMBRELA
Upadhyay, S., Pradeep, R., Thakur, N., Campos, D., Craswell, N., Soboroff, I., & Lin, J. (2025) · Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) · doi:10.1145/3731120.3744605 · Cited by 11
Self-citationReport on the Search Futures Workshop at ECIR 2024
Azzopardi, L. A., Clarke, C. L. A., Kantor, P. B., Mitra, B., Trippas, J. R., Ren, Z., Aliannejadi, M., Arabzadeh, N., Chandrasekar, R., de Rijke, M., et al. (2024) · ACM SIGIR Forum · doi:10.1145/3687273.3687288 · Cited by 14
Self-citationBenchmarking LLM-based Relevance Judgment Methods
Arabzadeh, N., & Clarke, C. L. A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730305 · Cited by 30
Adapting Standard Retrieval Benchmarks to Evaluate Generated Answers
Arabzadeh, N., Bigdeli, A., & Clarke, C. L. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_26 · Cited by 20
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 10
Rahmani, H. A., Siro, C., Aliannejadi, M., Craswell, N., Clarke, C. L. A., Faggioli, G., Mitra, B., Thomas, P., & Yılmaz, E. (2024) · ACM SIGIR Forum · doi:10.1145/3722449.3722461 · Cited by 9
Self-citationMulti-stage Large Language Model Pipelines Can Outperform GPT-4o in Relevance Assessment
Schnabel, J. A., Trippas, J. R., Scholer, F., & Hettiachchi, D. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715488 · Cited by 9
Criteria-Based LLM Relevance Judgments
Farzi, N., & Dietz, L. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744591 · Cited by 9
Large Language Models for Information Retrieval: Challenges and Chances
Breuer, T., Frihat, S., Fuhr, N., Lewandowski, D., Schaer, P., & Schenkel, R. (2025) · Datenbank-Spektrum · doi:10.1007/s13222-025-00503-x · Cited by 19
Wang, Y., Lin, Y., Huang, R., Wang, J., & Liu, S. (2024) · Computing and artificial intelligence. · doi:10.59400/cai.v2i1.535 · Cited by 12
Data Fusion of Synthetic Query Variants With Generative Large Language Models
Breuer, T. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698423 · Cited by 10
Yu, L., Alégroth, E., Chatzipetrou, P., & Gorschek, T. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107802 · Cited by 14
Italiani, P., Moro, G., & Ragazzi, L. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09463-9 · Cited by 15
Analyzing Adversarial Attacks on Sequence-to-Sequence Relevance Models
Parry, A., Fröbe, M., MacAvaney, S., Potthast, M., & Hagen, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_19 · Cited by 12
Generative AI in Crowdwork for Web and Social Media Research: A Survey of Workers at Three Platforms
Christoforou, E., Demartini, G., & Otterbacher, J. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31452 · Cited by 8
Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback
Choi, J. (2025) · Applied Sciences · doi:10.3390/app15095198 · Cited by 15
The Viability of Crowdsourcing for RAG Evaluation
Gienapp, L., Hagen, T., Fröbe, M., Hagen, M., Stein, B., Potthast, M., & Scells, H. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730093 · Cited by 7
Large Language Model Relevance Assessors Agree With One Another More Than With Human Assessors
Fröbe, M., Parry, A., Schlatt, F., MacAvaney, S., Stein, B., Potthast, M., & Hagen, M. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730218 · Cited by 7
Towards Understanding Bias in Synthetic Data for Evaluation
Rahmani, H. A., Ramineni, V., Yılmaz, E., Craswell, N., & Mitra, B. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760908 · Cited by 16
Self-citationWhat Matters in a Measure? A Perspective from Large-Scale Search Evaluation
Thomas, P., Kazai, G., Craswell, N., & Spielman, S. E. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657845 · Cited by 8
Self-citationRevolutionizing Marketing Research with a Large Language Model: A Hybrid AI-Human ApproachPreprint
Arora, N. K., Chakraborty, I., & Nishimura, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4683054 · Cited by 7
Does UMBRELA work on other LLMs?
Farzi, N., & Dietz, L. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730317 · Cited by 6
Evaluating Podcast Recommendations with Profile-Aware LLM-as-a-Judge
Fabbri, F., Penha, G., D’Amico, E., Wang, A., Nadai, M. D., Doremus, J., Gigioli, P., Damianou, A., Stål, O., & Lalmas, M. (2025) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3705328.3759305 · Cited by 13
Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Reranking
Fang, H., Tao, S., Chen, N., Chang, K.-X., & Sakai, T. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769493 · Cited by 19
Improving the Reusability of Conversational Search Test Collections
Abbasiantaeb, Z., Meng, C., Azzopardi, L. A., & Aliannejadi, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_13 · Cited by 7
Jiang, T., Li, W., & Liu, J. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.25020 · Cited by 6
Otero, D., Parapar, J., & Barreiro, Á. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730221 · Cited by 5
Demographically-Inspired Query Variants Using an LLM
Alaofi, M., Ferro, N., Thomas, P., Scholer, F., & Sanderson, M. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744608 · Cited by 9
Self-citationContext Example Selection for LLM Generated Relevance Assessments
McKechnie, J., McDonald, G., & Macdonald, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_19 · Cited by 6
Who Will Evaluate the Evaluators? Exploring the Gen-IR User Simulation Space
Kiesel, J., Gohsen, M., Mirzakhmedova, N., Hagen, M., & Stein, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71736-9_11 · Cited by 5
Quati: A Brazilian Portuguese Information Retrieval Dataset from Native Speakers
Bueno, M. C., de Oliveira, E. S., Nogueira, R. F., Lotufo, R., & Pereira, J. (2024) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2024.245426 · Cited by 19
Wang, Z., Sun, J., Liu, H., Luo, X., Li, J., He, W., Yang, Z., Lv, H., Chen, Y., & Wang, Z. (2025) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70020 · Cited by 4
Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants
Richards, J., & Wessel, M. S. (2025) · IEEE/ACM International Workshop on Bots in Software Engineering (BotSE) · doi:10.1109/botse67031.2025.00009 · Cited by 4
A Comparative Analysis of Linguistic and Retrieval Diversity in LLM-Generated Search Queries
Zendel, O., Lawati, S. F. D. A., Rashidi, L., Scholer, F., & Sanderson, M. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761382 · Cited by 9
How Do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective
Liu, Q., Duan, H., Mao, J., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3774942 · Cited by 13
Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?
Dietz, L., Li, B., Yang, E., Lawrie, D., Walden, W. G., & Mayfield, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_37 · Cited by 4
Knowledge Distillation for Enhancing Walmart E-commerce Search Relevance Using Large Language Models
Shang, H., Vo, N., Yadav, N., Zhang, T. Y., Puthenputhussery, A., Cai, X., Chen, S., Chandran, P., & Kang, C. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715242 · Cited by 9
LLM-Driven Usefulness Labeling for IR Evaluation
Dewan, M., Liu, J., & Shah, C. S. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730223 · Cited by 7
A Cost-Effective Framework to Evaluate LLM-Generated Relevance Judgements
Merlo, S., Marchesin, S., Faggioli, G., & Ferro, N. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761200 · Cited by 7
Trippas, J. R. (2024) · ACM eBooks · doi:10.1145/3674127.3674135 · Cited by 5
Two Heads Are Better Than One: Improving Search Effectiveness Through LLM-Generated Query Variants
Ran, K., Alaofi, M., Sanderson, M., & Spina, D. (2025) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716468 · Cited by 7
Christoforou, E., Demartini, G., & Otterbacher, J. (2025) · Communications of the ACM · doi:10.1145/3706101 · Cited by 3
Sujau, M., Wada, M., Vallée, É., Hillis, N., & Sušnjak, T. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7020028 · Cited by 6
Movin, M., & Hauff, C. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730119 · Cited by 3
Palumbo, E., Isaksson, M., Tamborrino, A., Movin, M., Dincu, C., Vardasbi, A., Nikeshkin, L., Gorobets, O., Nyman, A., Newdick, P., et al. (2025) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3705328.3748127 · Cited by 3
TRUE: A Reproducible Framework for LLM-Driven Relevance Judgment in Information Retrieval
Dewan, M., Liu, J., & Shah, C. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3779397 · Cited by 7
The Effect of Document Summarization on LLM-Based Relevance Judgments
Mohtadi, S., Roitero, K., Mizzaro, S., & Demartini, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_5 · Cited by 6
Can we trust LLMs as relevance judges?
Bencke, L. R., Paula, F. S. F., dos Santos, B. G. T., & Moreira, V. P. (2024) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2024.243130 · Cited by 3
mFollowIR: A Multilingual Benchmark for Instruction Following in Retrieval
Weller, O., Chang, B., Yang, E., Yarmohammadi, M., Barham, S., MacAvaney, S., Cohan, A., Soldaini, L., Durme, B. V., & Lawrie, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88711-6_19 · Cited by 8
Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learning
Liu, H., Gong, S., Ji, Y., Wu, K., Xu, J., & Gu, J. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3680052 · Cited by 4
Effect of LLM's Personality Traits on Query Generation
Imasaka, Y., & Joho, H. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698433 · Cited by 6
JudgeBlender: Ensembling Automatic Relevance Judgments
Rahmani, H. A., Yılmaz, E., Craswell, N., & Mitra, B. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715536 · Cited by 3
Self-citationAn Instruction-Response Perspective on Large Language Models in Information Retrieval Tasks
Joho, H., & Jose, J. M. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730346 · Cited by 3
Temporal Extraction and Retrieval
Alonso, O., & Spaniol, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674137 · Cited by 2
Tonellotto, N. (2024) · ACM eBooks · doi:10.1145/3674127.3674130 · Cited by 3
Evaluating the Evaluations: A Perspective on Benchmarks
Alonso, O., & Church, K. (2024) · ACM SIGIR Forum · doi:10.1145/3722449.3722467 · Cited by 3
System Comparison Using Automated Generation of Relevance Judgements in Multiple Languages
Thomas, P., Oard, D. W., Yang, E., Lawrie, D., & Mayfield, J. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730252 · Cited by 2
Self-citationAlmutairi, W. A., & Ibrahim Almatrodi, N. A. (2025) · SAGE Open · doi:10.1177/21582440251380135 · Cited by 6
On the Use of LLMs for Relevance Labelling
Alaofi, M., Thomas, P., Scholer, F., & Sanderson, M. (2026) · ACM Transactions on Information Systems · doi:10.1145/3788872 · Cited by 4
Self-citationA Multi-LLM Pipeline for Retrieval-Grounded, Bloom’s Taxonomy-Aligned Question Generation
Abdullah, M., Mansoor, I., Rey, V. F., & Fraz, M. M. (2025) · International Conference on Digital Futures and Transformative Technologies (ICoDT2) · doi:10.1109/icodt269104.2025.11360748 · Cited by 3
UserSimCRS v2: Simulation-Based Evaluation for Conversational Recommender Systems
Bernard, N., & Balog, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21321-1_55 · Cited by 9
Verberne, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674141 · Cited by 5
Ferro, N., & Maistro, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674132 · Cited by 3
Privacy in Information Retrieval
Soboroff, I. M. (2024) · ACM eBooks · doi:10.1145/3674127.3674139 · Cited by 2
The Effects of Demographic Instructions on LLM Personas
de Paula, A. F. M., Culpepper, J. S., Moffat, A., Cherumanal, S. P., Scholer, F., & Trippas, J. R. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730255 · Cited by 4
Refined Medical Search via Dense Retrieval and User Interaction
Goli, R., Moffat, A., & Buchanan, G. R. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730292 · Cited by 2
eCommerce Search and Discovery
Kallumadi, S. T., & King, T. H. (2024) · ACM eBooks · doi:10.1145/3674127.3674140 · Cited by 1
Towards Web-scale Recommendations with LLMs: From Quality-aware Ranking to Candidate Generation
Shah, J. J., Barjasteh, I., Barapatre, A., Forsati, R., Luo, G., Wu, F., Fang, Y., Deng, X., Shepard, B., Shah, R., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709413 · Cited by 4
The Second International Workshop on Open Web Search (WOWS)
Farzana, S. M., Fröbe, M., Granitzer, M., Hendriksen, G., Hiemstra, D., Potthast, M., de Vries, A. P., & Zerhoudi, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88720-8_55 · Cited by 1
Hosseini, K., Kober, T., Krapac, J., Vollgraf, R., Cheng, W., & Ramallo, A. P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_10 · Cited by 7
REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources
Engelmann, B., Haak, F., Schaer, P., Abdoust, M. E., Netze, L., & Bittkowski, M. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730342 · Cited by 3
Measuring Hypothesis Testing Errors in the Evaluation of Retrieval Systems
McKechnie, J., McDonald, G., & Macdonald, C. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730229 · Cited by 1
Classifying Term Variants in Query Formulation
Onq, N. A., Sanderson, M., & Scholer, F. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729924 · Cited by 1
Faggioli, G., Ferro, N., Perego, R., & Tonellotto, N. (2025) · ACM Transactions on Information Systems · doi:10.1145/3765619 · Cited by 2
JurisTCU: a Brazilian Portuguese information retrieval dataset with query relevance judgments
Fernandes, L. C., Ribeiro, L. D. S., de Castro, M. V. B., Pacheco, L. A. D. S., & Sandes, E. F. O. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-025-09881-w · Cited by 4
CLUE: Using Large Language Models for Judging Document Usefulness in Web Search Evaluation
Wang, X., Zhang, E., Chen, Y., Xuan, J., Hou, Y., Xu, Y., Nie, Y., Wang, S., Yin, D., & Mao, J. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761158 · Cited by 1
BloomIntent: Automating Search Evaluation with LLM-Generated Fine-Grained User Intents
Choi, Y., Kim, E., Kim, H., Park, D., Lee, H., Kim, J. Y., & Kim, J. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747677 · Cited by 6
Metadata
- Authors
- Thomas, P., Spielman, S., Craswell, N., & Mitra, B.
- Affiliations
- Microsoft (United States) (US)
- Microsoft Research Montréal (Canada) (CA)
- Venue
- Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2024-07-10
- Pages
- 1930-1940
- OpenAlex
- W4400526908
- Cited by
- 168
- References
- 27
- Citations by year
- 2026: 42 · 2025: 74 · 2024: 51
- FWCI
- 63.52
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- large language models, search evaluation, label quality, user feedback, automatic annotation, prompt engineering, query difficulty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
