CoChat
Sign Up
Cited by 169Information Retrieval and Search BehaviorTopic ModelingRecommender Systems and Techniques

Large Language Models can Accurately Predict Searcher Preferences

Open access
Thomas, Paul, Spielman, Seth, Craswell, Nick, Mitra, Bhaskar·2024·Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
DOI 10.1145/3626772.3657707
Open this paper in CoChat — free account

Abstract

Much of the evaluation and tuning of a search system relies on relevance labels---annotations that say whether a document is useful for a given search and searcher. Ideally these come from real searchers, but it is hard to collect this data at scale, so typical experiments rely on third-party labellers who may or may not produce accurate annotations. Label quality is managed with ongoing auditing, training, and monitoring. We discuss an alternative approach. We take careful feedback from real searchers and use this to select a large language model (LLM), and prompt, that agrees with this feedback; the LLM can then produce labels at scale. Our experiments show LLMs are as accurate as human labellers and as useful for finding the best systems and hardest queries. LLM performance varies with prompt features, but also varies unpredictably with simple paraphrases. This unpredictability reinforces the need for high-quality "gold" labels.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3626772.3657707

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs, when provided with proper guidance, can effectively 'understand' the relation between a given query and passage and accurately determine its relevance.

    “LLMs, when provided with proper guidance, can effectively 'understand' the relation between a given query and passage and accurately determine its relevance.”

    From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]

    Section 4.1 (p. 5) concludes that, for some prompts, LLM relevance labels reached human-quality agreement on the collection; Section 2.2 (pp. 2–3) describes giving the model a query and document text with prompting guidance. The study evaluated query–document pairs, not specifically query–passage pairs, and its results varied considerably by prompt, so the passage-specific claim goes beyond what the paper establishes. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • LLM assessment is an effective alternative to human-based assessment.

    “Our experiments with the TREC Deep Learning (DL) Tracks from 2019-2023 [4-8] verified the claims by Thomas et al. [21] about LLM assessment being an effective alternative to humanbased assessment, using OpenAI's latest model GPT-4o.”

    From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]

  • UMBRELA is an open-source toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o and adds more nuance to the original paper.

    “Our work presents UMBRELA (a recursive acronym that stands for UM brela is the B ing REL evance A ssessor), an open-source toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o model and adds more nuance to the original paper.”

    From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]

  • LLMs possess the ability to understand user needs and assign relevance labels with veracity comparable to human assessors.

    “As highlighted by Thomas et al. [21], LLMs possess the ability to understand user needs and assign relevance labels with veracity comparable to human assessors.”

    From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]

  • We have successfully replicated the results from Thomas et al. [21], thereby validating the capabilities of OpenAI's latest model, GPT-4o, in performing relevance assessments automatically.

    “In our research, we have successfully replicated the results from Thomas et al. [21], thereby validating the capabilities of OpenAI's latest model, GPT-4o, in performing relevance assessments automatically.”

    From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. [1][1] Aashish Agarwal, Ankita Mandal, Matthias Schaffeld, Fangzheng Ji, Jhiao Zhan, Yiqi Sun, and Ahmet Aker. 2019. Good, neutral or bad news classification. In Proceedings of the Third International Workshop on Recent Trends in News Information Retrieval . 9-14. -
  2. [2][2] Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikovo, and Fabrizio Gilardi. 2023. Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks. arXiv:2307.02179 [cs.CL] - · arXiv
  3. [6][6] Valeria Bolotova, Vladislav Blinov, Yukun Zheng, W Bruce Croft, Falk Scholer, and MarkSanderson. 2020. Do people and neural nets pay attention to the same words: studying eye-tracking data for non-factoid QA evaluation. In Proceedings of the ACMInternational Conference on Information and Knowledge Management . 85-94. -
  4. [7][7] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Advancesinneural information processing systems 29 (2016). -
  5. [9][9] Andrei Broder. 2002. A taxonomy of web search. In ACM Sigir forum , Vol. 36. ACMNewYork, NY, USA, 3-10. -
  6. [10][10] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183-186. -
  7. [11][11] Ben Carterette, Paul N Bennett, David Maxwell Chickering, and Susan T Dumais. 2008. Here or there: Preference judgments for relevance. In Proceedings of the European Conference on Information Retrieval . 16-27. -
  8. [15][15] Charles L A Clarke, Gianluca Demartini, Laura Dietz, Guglielmo Faggioli, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Ian Soboroff, Benno Stein, and Henning Wachsmuth. 2023. HMC: A spectrum of human-machine-collaborative relevance judgment frameworks. In Frontiers of Information Access Experimentation for Research and Education , Christine Bauer, Ben Carterette, Nicola Ferro, and Norbert Fuhr (Eds.). Vol. 13. Leibniz-Zentrum für Informatik. Issue 1. -
  9. [18][18] Tadele T. Damessie, Taho P. Nghiem, Falk Scholer, and J. Shane Culpepper. 2017. Gauging the quality of relevance assessments using inter-rater agreement. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval . -
  10. [20][20] Susan Dumais, Robin Jeffries, , Daniel M. Russell, Diane Tang, and Jaime Teevan. 2014. Understanding user behavior through log data and analysis. In Ways of knowing in HCI , Judith S. Olson and Wendy A. Kellogg (Eds.). Springer, New York, 349-372. -
  11. [24][24] Charles A E Goodhart. 1975. Problems of monetary management: The UK experience. In Papers in Monetary Economics . Vol. 1. Reserve Bank of Australia. -
  12. [27][27] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop . http://arxiv.org/abs/1503.02531 - · arXiv
  13. [34][34] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman,BinhangYuan,BobbyYan,CeZhang,ChristianCosgrove,ChristopherD. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic evaluation of language models. arXiv:2211.09110 [cs.CL] - · arXiv
  14. [38][38] OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] - · arXiv
  15. [40][40] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training. (2021). arXiv:2104.10350 [cs.LG] - · arXiv
  16. [41][41] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with 'gradient descent' and beam search. arXiv:2305.03495 - · arXiv
  17. [47][47] Rachel L. Thomas and David Uminsky. 2022. Reliance on metrics is a fundamental challenge for AI. Patterns 3, 5 (2022). -
  18. [48][48] PetterTörnberg.2023. ChatGPT-4outperformsexpertsandcrowdworkersinannotating political Twitter messages with zero-shot learning. arXiv:2304.06588 [cs.CL] - · arXiv
  19. [49][49] Ellen M Voorhees. 1998. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval . 315-323. -
  20. [50][50] Ellen M Voorhees. 2004. Overview of the TREC 2004 Robust Retrieval Track. In Proceedings of the Text REtrieval Conference . -
  21. [51][51] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How far can camels go? Exploring the state of instruction tuning on open resources. arXiv:2306.04751 [cs.CL] - · arXiv
  22. [53][53] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. arXiv:2201.11903 [cs.CL] - · arXiv
  23. [55][55] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimisers. arXiv:2309.03409 [cs.LG] - · arXiv
  24. [56][56] Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. 2022. TEMPERA: Test-time prompt editing via reinforcement learning. arXiv:2211.11890 [cs.CL] - · arXiv
  25. [57][57] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv:2211.01910 [cs.LG] · arXiv

Cited by

Sorted by citations

Related

Authors
Thomas, P., Spielman, S., Craswell, N., & Mitra, B.
Affiliations
  • Microsoft (United States) (US)
  • Microsoft Research Montréal (Canada) (CA)
ORCID
Venue
Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2024-07-10
Pages
1930-1940
DOI
10.1145/3626772.3657707
OpenAlex
W4400526908
Cited by
168
References
27
Citations by year
2026: 42 · 2025: 74 · 2024: 51 · 2023: 1
FWCI
63.52
Type
conference paper
Language
en
Open access
Licence
Locations
  • International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
large language models, search evaluation, label quality, user feedback, automatic annotation, prompt engineering, query difficulty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review