The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Large Language Models can Accurately Predict Searcher Preferences
Open accessAbstract
Much of the evaluation and tuning of a search system relies on relevance labels---annotations that say whether a document is useful for a given search and searcher. Ideally these come from real searchers, but it is hard to collect this data at scale, so typical experiments rely on third-party labellers who may or may not produce accurate annotations. Label quality is managed with ongoing auditing, training, and monitoring. We discuss an alternative approach. We take careful feedback from real searchers and use this to select a large language model (LLM), and prompt, that agrees with this feedback; the LLM can then produce labels at scale. Our experiments show LLMs are as accurate as human labellers and as useful for finding the best systems and hardest queries. LLM performance varies with prompt features, but also varies unpredictably with simple paraphrases. This unpredictability reinforces the need for high-quality "gold" labels.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, when provided with proper guidance, can effectively 'understand' the relation between a given query and passage and accurately determine its relevance.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Our experiments with the TREC Deep Learning (DL) Tracks from 2019-2023 [4-8] verified the claims by Thomas et al. [21] about LLM assessment being an effective alternative to humanbased assessment, using OpenAI's latest model GPT-4o.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“Our work presents UMBRELA (a recursive acronym that stands for UM brela is the B ing REL evance A ssessor), an open-source toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o model and adds more nuance to the original paper.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“As highlighted by Thomas et al. [21], LLMs possess the ability to understand user needs and assign relevance labels with veracity comparable to human assessors.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“In our research, we have successfully replicated the results from Thomas et al. [21], thereby validating the capabilities of OpenAI's latest model, GPT-4o, in performing relevance assessments automatically.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 2×
“3 Lastly, the ecological costs of these LLMs are still heavily debated [Bender et al. 2021; Bommasani et al. 2021; Dodge et al. 2022; Patterson et al. 2022, 2021; Wu et al. 2022] but represent an important aspect in which these models should”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Learning to Rank for Information Retrieval
Liu, T. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,715
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 1×
“Language models are known to reproduce and amplify harmful stereotypes and biases of social import [Bender et al. 2021; Blodgett et al. 2020; Bolukbasi et al. 2016; Caliskan et al. 2017; Gonen and Goldberg 2019] and therefore there is an immediate need to study if and how these biases may also manifest in relevance la…”
BackgroundG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
A similarity measure for indefinite rankings
Webber, W., Moffat, A., & Zobel, J. (2010) · ACM Transactions on Information Systems · doi:10.1145/1852102.1852106 · Cited by 864
OHSUMED: An Interactive Retrieval Evaluation and New Large Test Collection for Research
Hersh, W., Buckley, C., Leone, T. J., & Hickam, D. H. (1994) · Springer eBooks · doi:10.1007/978-1-4471-2099-5_20 · Cited by 787
Discrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 1×
“This may intensify existing representational and allocative harms from search systems [37, 45].”
BackgroundTest Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485
Sustainable AI: Environmental Implications, Challenges and Opportunities
Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Behram, F. A., Huang, J., Bai, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.00364 · Cited by 405
The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
Patterson, D., Gonzalez, J., Holzle, U., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D. R., Texier, M., & Dean, J. (2022) · Computer · doi:10.1109/mc.2022.3148714 · Cited by 400
Accountability : power, ethos, and the technologies of managing
Munro, R., & Mouritsen, J. (1996) · Cited by 366
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331
Measuring the Carbon Intensity of AI in Cloud Instances
Dodge, J., Prewitt, T., Tachet des Combes, R., Odmark, E., Schwartz, R., Strubell, E., Luccioni, A. S., Smith, N. A., DeCario, N., & Buchanan, W. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533234 · Cited by 257
Efficient construction of large test collections
Cormack, G. V., Palmer, C. R., & Clarke, C. L. A. (1998) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/290941.291009 · Cited by 234
A reference collection for web spam
Castillo, C., Donato, D., Becchetti, L., Boldi, P., Leonardi, S., Santini, M., & Vigna, S. (2006) · ACM SIGIR Forum · doi:10.1145/1189702.1189703 · Cited by 205
Perspectives on Large Language Models for Relevance Judgment
Faggioli, G., Dietz, L., Clarke, C. L. A., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., Stein, B., et al. (2023) · Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval · doi:10.1145/3578337.3605136 · Cited by 184
Bailey, P., Craswell, N., Soboroff, I. M., Thomas, P., de Vries, A. P., & Yılmaz, E. (2008) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1390334.1390447 · Cited by 180
Self-citationTörnberg, P. (2023) · arXiv · doi:10.48550/arxiv.2304.06588 · Cited by 152
Local Self-Attention over Long Text for Efficient Document Retrieval
Hofstätter, S., Zamani, H., Mitra, B., Craswell, N., & Hanbury, A. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401224 · Cited by 92
Self-citationThe effect of threshold priming and need for cognition on relevance calibration and assessment
Scholer, F., Kelly, D., Wu, W., Lee, H. S., & Webber, W. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484090 · Cited by 70
Saračević, T. (2008) · Library trends · doi:10.1353/lib.0.0000 · Cited by 44
Examining the Limits of Crowdsourcing for Relevance Assessment
Clough, P., Sanderson, M., Tang, J., Gollins, T., & Warner, A. J. (2012) · IEEE Internet Computing · doi:10.1109/mic.2012.95 · Cited by 39
Akintayo, J. (2020) · Routledge eBooks · doi:10.4324/9780429399718-51 · Cited by 31
Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection
Inel, O., Draws, T., & Aroyo, L. M. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27547 · Cited by 14
Thomas, P., Kazai, G., White, R. W., & Craswell, N. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505815 · Cited by 14
Self-citationLess is Less: When are Snippets Insufficient for Human vs Machine Relevance Estimation?
Kazai, G., Mitra, B., Dong, A., Craswell, N., & Yang, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99739-7_18 · Cited by 3
Self-citation
Research works printed but not linked: 26
- Aashish Agarwal, Ankita Mandal, Matthias Schaffeld, Fangzheng Ji, Jhiao Zhan, Yiqi Sun, and Ahmet Aker. 2019. Good, neutral or bad news classification. In Proceedings of the Third International Workshop on Recent Trends in News Information Retrieval. 9--14.
- Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikovo, and Fabrizio Gilardi. 2023. Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks. arXiv:2307.02179 [cs.CL] · arXiv
- Valeria Bolotova, Vladislav Blinov, Yukun Zheng,WBruce Croft, Falk Scholer, and Mark Sanderson. 2020. Do people and neural nets pay attention to the samewords: studying eye-tracking data for non-factoid QA evaluation. In Proceedings of the ACM International Conference on Information and Knowledge Management. 85--94.
- Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasingword embeddings. Advances in neural information processing systems 29 (2016).
- Andrei Broder. 2002. A taxonomy of web search. In ACM Sigir forum, Vol. 36. ACM New York, NY, USA, 3--10.
- Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183--186.
- Ben Carterette, Paul N Bennett, David Maxwell Chickering, and Susan T Dumais. 2008. Here or there: Preference judgments for relevance. In Proceedings of the European Conference on Information Retrieval. 16--27.
- K. Alec Chrystal and Paul D. Mizen. 2001. Goodhart's law: Its origins, meaning and implications for monetary policy. Prepared for the Festschrift in honour of Charles Goodhart.
- Charles LAClarke Gianluca Demartini Laura Dietz Guglielmo Faggioli Matthias Hagen Claudia Hauff Noriko Kando Evangelos Kanoulas Martin Potthast Ian Soboroff Benno Stein and Henning Wachsmuth. 2023. HMC: A spectrum of human--machine-collaborative relevance judgment frameworks. In Frontiers of Information Access Experimentation for Research and Education Christine Bauer Ben Carterette Nicola Ferro and Norbert Fuhr (Eds.). Vol. 13. Leibniz-Zentrum für Informatik. Issue 1.
- Tadele T. Damessie, Taho P. Nghiem, Falk Scholer, and J. Shane Culpepper. 2017. Gauging the quality of relevance assessments using inter-rater agreement. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval.
- Susan Dumais, Robin Jeffries, , Daniel M. Russell, Diane Tang, and Jaime Teevan. 2014. Understanding user behavior through log data and analysis. In Ways of knowing in HCI, Judith S. Olson andWendy A. Kellogg (Eds.). Springer, New York, 349--372.
- Charles A E Goodhart. 1975. Problems of monetary management: The UK experience. In Papers in Monetary Economics. Vol. 1. Reserve Bank of Australia.
- Google LLC. 2022. General Guidelines. https://guidelines.raterhub.com/ searchqualityevaluatorguidelines.pdf, Downloaded 29 July 2023..
- Geoffrey Hinton Oriol Vinyals and Jeff Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation LearningWorkshop. http://arxiv.org/abs/1503.02531 · arXiv
- Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, YuhuaiWu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, ChristopherD. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, JueWang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic evaluation of language models. arXiv:2211.09110 [cs.CL] · arXiv
- OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] · arXiv
- David Patterson Joseph Gonzalez Quoc Le Chen Liang Lluis-Miquel Munguia Daniel Rothchild David So Maud Texier and Jeff Dean. 2021. Carbon emissions and large neural network training. (2021). arXiv:2104.10350 [cs.LG] · arXiv
- Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with ?gradient descent" and beam search. arXiv:2305.03495 · arXiv
- Rachel L. Thomas and David Uminsky. 2022. Reliance on metrics is a fundamental challenge for AI. Patterns 3, 5 (2022).
- EllenMVoorhees. 1998. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. 315--323.
- EllenMVoorhees. 2004. Overview of the TREC 2004 Robust Retrieval Track. In Proceedings of the Text REtrieval Conference.
- Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, DavidWadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How far can camels go? Exploring the state of instruction tuning on open resources. arXiv:2306.04751 [cs.CL] · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. arXiv:2201.11903 [cs.CL] · arXiv
- Chengrun Yang Xuezhi Wang Yifeng Lu Hanxiao Liu Quoc V. Le Denny Zhou and Xinyun Chen. 2023. Large language models as optimisers. arXiv:2309.03409 [cs.LG] · arXiv
- Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. 2022. TEMPERA: Test-time prompt editing via reinforcement learning. arXiv:2211.11890 [cs.CL] · arXiv
- Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv:2211.01910 [cs.LG] · arXiv
Cited by
Sorted by citations
AI–Human Hybrids for Marketing Research: Leveraging Large Language Models (LLMs) as Collaborators
Arora, N., Chakraborty, I., & Nishimura, Y. (2024) · Journal of Marketing · doi:10.1177/00222429241276529 · Cited by 211
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 86
Advancing the search frontier with AI agents
White, R. W. (2024) · Communications of the ACM · doi:10.1145/3655615 · Cited by 55
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47
A survey on privacy risks and protection in large language models
Chen, K., Zhou, X., Lin, Y., Feng, S., Shen, L., & Wu, P. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00177-1 · Cited by 67
Metadata
- Authors
- Thomas, P., Spielman, S., Craswell, N., & Mitra, B.
- Affiliations
- Microsoft (United States) (US)
- Microsoft Research Montréal (Canada) (CA)
- Venue
- Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2024-07-10
- Pages
- 1930-1940
- OpenAlex
- W4400526908
- Cited by
- 168
- References
- 27
- Citations by year
- 2026: 42 · 2025: 74 · 2024: 51
- FWCI
- 63.52
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- large language models, search evaluation, label quality, user feedback, automatic annotation, prompt engineering, query difficulty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
