The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Large Language Models can Accurately Predict Searcher Preferences
Open accessAbstract
Much of the evaluation and tuning of a search system relies on relevance labels---annotations that say whether a document is useful for a given search and searcher. Ideally these come from real searchers, but it is hard to collect this data at scale, so typical experiments rely on third-party labellers who may or may not produce accurate annotations. Label quality is managed with ongoing auditing, training, and monitoring. We discuss an alternative approach. We take careful feedback from real searchers and use this to select a large language model (LLM), and prompt, that agrees with this feedback; the LLM can then produce labels at scale. Our experiments show LLMs are as accurate as human labellers and as useful for finding the best systems and hardest queries. LLM performance varies with prompt features, but also varies unpredictably with simple paraphrases. This unpredictability reinforces the need for high-quality "gold" labels.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, when provided with proper guidance, can effectively 'understand' the relation between a given query and passage and accurately determine its relevance.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Our experiments with the TREC Deep Learning (DL) Tracks from 2019-2023 [4-8] verified the claims by Thomas et al. [21] about LLM assessment being an effective alternative to humanbased assessment, using OpenAI's latest model GPT-4o.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“Our work presents UMBRELA (a recursive acronym that stands for UM brela is the B ing REL evance A ssessor), an open-source toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o model and adds more nuance to the original paper.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“As highlighted by Thomas et al. [21], LLMs possess the ability to understand user needs and assign relevance labels with veracity comparable to human assessors.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“In our research, we have successfully replicated the results from Thomas et al. [21], thereby validating the capabilities of OpenAI's latest model, GPT-4o, in performing relevance assessments automatically.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 2×
“3 Lastly, the ecological costs of these LLMs are still heavily debated [Bender et al. 2021; Bommasani et al. 2021; Dodge et al. 2022; Patterson et al. 2022, 2021; Wu et al. 2022] but represent an important aspect in which these models should”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Learning to Rank for Information Retrieval
Liu, T. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,715
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 1×
“Language models are known to reproduce and amplify harmful stereotypes and biases of social import [Bender et al. 2021; Blodgett et al. 2020; Bolukbasi et al. 2016; Caliskan et al. 2017; Gonen and Goldberg 2019] and therefore there is an immediate need to study if and how these biases may also manifest in relevance la…”
BackgroundG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
A similarity measure for indefinite rankings
Webber, W., Moffat, A., & Zobel, J. (2010) · ACM Transactions on Information Systems · doi:10.1145/1852102.1852106 · Cited by 864
OHSUMED: An Interactive Retrieval Evaluation and New Large Test Collection for Research
Hersh, W., Buckley, C., Leone, T. J., & Hickam, D. H. (1994) · Springer eBooks · doi:10.1007/978-1-4471-2099-5_20 · Cited by 787
Research works printed but not linked: 26
- Aashish Agarwal, Ankita Mandal, Matthias Schaffeld, Fangzheng Ji, Jhiao Zhan, Yiqi Sun, and Ahmet Aker. 2019. Good, neutral or bad news classification. In Proceedings of the Third International Workshop on Recent Trends in News Information Retrieval. 9--14.
- Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikovo, and Fabrizio Gilardi. 2023. Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks. arXiv:2307.02179 [cs.CL] · arXiv
- Valeria Bolotova, Vladislav Blinov, Yukun Zheng,WBruce Croft, Falk Scholer, and Mark Sanderson. 2020. Do people and neural nets pay attention to the samewords: studying eye-tracking data for non-factoid QA evaluation. In Proceedings of the ACM International Conference on Information and Knowledge Management. 85--94.
- Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasingword embeddings. Advances in neural information processing systems 29 (2016).
- Andrei Broder. 2002. A taxonomy of web search. In ACM Sigir forum, Vol. 36. ACM New York, NY, USA, 3--10.
- Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183--186.
- Ben Carterette, Paul N Bennett, David Maxwell Chickering, and Susan T Dumais. 2008. Here or there: Preference judgments for relevance. In Proceedings of the European Conference on Information Retrieval. 16--27.
- K. Alec Chrystal and Paul D. Mizen. 2001. Goodhart's law: Its origins, meaning and implications for monetary policy. Prepared for the Festschrift in honour of Charles Goodhart.
- Charles LAClarke Gianluca Demartini Laura Dietz Guglielmo Faggioli Matthias Hagen Claudia Hauff Noriko Kando Evangelos Kanoulas Martin Potthast Ian Soboroff Benno Stein and Henning Wachsmuth. 2023. HMC: A spectrum of human--machine-collaborative relevance judgment frameworks. In Frontiers of Information Access Experimentation for Research and Education Christine Bauer Ben Carterette Nicola Ferro and Norbert Fuhr (Eds.). Vol. 13. Leibniz-Zentrum für Informatik. Issue 1.
- Tadele T. Damessie, Taho P. Nghiem, Falk Scholer, and J. Shane Culpepper. 2017. Gauging the quality of relevance assessments using inter-rater agreement. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval.
- Susan Dumais, Robin Jeffries, , Daniel M. Russell, Diane Tang, and Jaime Teevan. 2014. Understanding user behavior through log data and analysis. In Ways of knowing in HCI, Judith S. Olson andWendy A. Kellogg (Eds.). Springer, New York, 349--372.
- Charles A E Goodhart. 1975. Problems of monetary management: The UK experience. In Papers in Monetary Economics. Vol. 1. Reserve Bank of Australia.
- Google LLC. 2022. General Guidelines. https://guidelines.raterhub.com/ searchqualityevaluatorguidelines.pdf, Downloaded 29 July 2023..
- Geoffrey Hinton Oriol Vinyals and Jeff Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation LearningWorkshop. http://arxiv.org/abs/1503.02531 · arXiv
- Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, YuhuaiWu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, ChristopherD. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, JueWang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic evaluation of language models. arXiv:2211.09110 [cs.CL] · arXiv
- OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] · arXiv
- David Patterson Joseph Gonzalez Quoc Le Chen Liang Lluis-Miquel Munguia Daniel Rothchild David So Maud Texier and Jeff Dean. 2021. Carbon emissions and large neural network training. (2021). arXiv:2104.10350 [cs.LG] · arXiv
- Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with ?gradient descent" and beam search. arXiv:2305.03495 · arXiv
- Rachel L. Thomas and David Uminsky. 2022. Reliance on metrics is a fundamental challenge for AI. Patterns 3, 5 (2022).
- EllenMVoorhees. 1998. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. 315--323.
- EllenMVoorhees. 2004. Overview of the TREC 2004 Robust Retrieval Track. In Proceedings of the Text REtrieval Conference.
- Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, DavidWadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How far can camels go? Exploring the state of instruction tuning on open resources. arXiv:2306.04751 [cs.CL] · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. arXiv:2201.11903 [cs.CL] · arXiv
- Chengrun Yang Xuezhi Wang Yifeng Lu Hanxiao Liu Quoc V. Le Denny Zhou and Xinyun Chen. 2023. Large language models as optimisers. arXiv:2309.03409 [cs.LG] · arXiv
- Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. 2022. TEMPERA: Test-time prompt editing via reinforcement learning. arXiv:2211.11890 [cs.CL] · arXiv
- Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv:2211.01910 [cs.LG] · arXiv
Cited by
Sorted by citations
Items 101–169 of 169
Li, X., & Li, J. (2026) · ACM Transactions on Information Systems · doi:10.1145/3793857 · Cited by 1
Marketing Keywords Exploration via LLMs With Domain Expertise
Bulut, A. (2026) · IEEE Access · doi:10.1109/access.2026.3659535 · Cited by 1
Evaluating Information Retrieval Models Along Time: The LongEval Lab at CLEF 2026
Breuer, T., Cancellieri, M., El-Ebshihy, A., Fröbe, M., Galuščáková, P., Goeuriot, L., Iturra-Bocaz, G., Keller, J., Knoth, P., Kruff, A. K., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21321-1_45 · Cited by 1
Query–Document Dense Vectors for LLM Relevance Judgment Bias Analysis
Mohtadi, S., & Demartini, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_6 · Cited by 1
Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents
Rafiee, M., Soudani, H., Abbasiantaeb, Z., Aliannejadi, M., Hasibi, F., & Zamani, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808629 · Cited by 5
Auto-Judge: A Cross-Task Benchmark for Comparing LLM Judges for Citation-Grounded RAG Systems
Farzi, N., Hagen, T. H., Yang, E., Fröbe, M., Pradeep, R., Rahmani, H. A., Wang, X., Zendel, O., Potthast, M., & Dietz, L. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808601 · Cited by 2
Identifying Key Terms in Prompts for Relevance Evaluation with GPT Models
Choi, J. (2024) · International Journal on Natural Language Computing · doi:10.5121/ijnlc.2024.13201 · Cited by 2
Toward Automatic Group Membership Annotation for Group Fairness Evaluation
Chen, F. L., Yang, D., & Fang, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70239-6_20 · Cited by 2
Galuščáková, P., Oard, D. W., & Nair, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674136 · Cited by 2
Gómez, E., Cuesta, H., Gkiokas, A., Gómez-Cañón, J. S., Porcaro, L., & Yesiler, F. (2024) · ACM eBooks · doi:10.1145/3674127.3702959 · Cited by 1
Baeza-Yates, R. A., Murgai, L., & Dudy, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674138 · Cited by 1
On the Necessity of World Knowledge for Mitigating Missing Labels in Extreme Classification
Prakash, J., Buvanesh, A., Santra, B., Saini, D., Yadav, S., Jiao, J., Prabhu, Y., Sharma, A., & Varma, M. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709290 · Cited by 1
RARR Unraveled: Component-Level Insights into Hallucination Detection and Mitigation
Ross, J. J., Khramtsova, E., van der Vegt, A. H., Koopman, B., & Zuccon, G. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730337 · Cited by 3
Goodbye Human Annotators? Content Analysis of Policy Debates using ChatGPTPreprint
Gielens, E., & Sowula, J. (2024) · doi:10.31235/osf.io/m5r3h · Cited by 1
Sycophancy Negatively Affects LLM-as-a-Judge in Conflict Evaluation
Farzi, N., Dietz, L., & Carton, S. (2026) · Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2026.gem-main.45
“Large Language Models (LLMs) are increasingly used as evaluators, and the LLM-as-a-judge paradigm has reshaped how information is assessed and scaled across diverse domains (Thomas et al., 2023; Upadhyay et al., 2024; Sun et al., 2023; Ma et al., 2023; Farzi and Dietz, 2024; Dietz and Farzi, 2025).”
Evaluating Parrots and Sociopathic Liars: A keynote at ICTIR 2023
Sakai, T. (2023) · ACM SIGIR Forum · doi:10.1145/3642979.3642984
Navigating the Evaluation Funnel to Optimize Iteration Speed for Recommender Systems
Schultzberg, C., & Ottens, B. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-73110-5_11
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674128
Searching Personal Collections
Bendersky, M., Metzler, D., Najork, M., & Wang, X. (2024) · ACM eBooks · doi:10.1145/3674127.3674142
Query-driven Search Functionality
Li, L. (2024) · ACM eBooks · doi:10.1145/3674127.3674133
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702962
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674129
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702961
Hersh, W., & Roberts, K. E. (2024) · ACM eBooks · doi:10.1145/3674127.3702960
Bast, H., Kalmbach, J., Klumpp, T., & Korzen, C. (2024) · ACM eBooks · doi:10.1145/3674127.3674134
Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Information
Long, K., Li, S., Xu, C., Tang, J., & Wang, T. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730061
Evaluating the Impact of Automated Labeling on Retrieval Instability in Neural IR
Ingram, W. A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730128
Towards Efficient and Effective Multi-modal Retrieval
Mansour, W. A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730129
ADORE: Autonomous Domain-Oriented Relevance Engine for E-commerce
Fang, Z., Xie, D., Pang, M., Yuan, C., Jiang, X. L., Peng, C., Lin, Z., & Luo, Z. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3731944
Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications
Tavakoli, L., & Zamani, H. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744574 · Cited by 4
Yang, D., Zhang, X., Han, T., & Liu, Y. (2025) · Natural language processing. · doi:10.1017/nlp.2025.10005 · Cited by 1
How Good are LLMs at Retrieving Documents in a Specific Domain?
Islam, N. T., & Zhao, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05607-8_26
How Faithful Are Your Summaries? A Study of NLI-Based Verification in Portuguese
Paula, F. S. F., Westhelle, M., Corrêa, M. C. M., Bencke, L. R., & Moreira, V. P. (2025) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2025.37834
Study on LLMs for Promptagator-Style Dense Retriever Training
Gwon, D., Jedidi, N., & Lin, J. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760960
Sakai, T., Rain, K. M., Takehi, R., Tao, S., & Song, Y.-I. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760934
Wang, Y., Chen, Y., Li, H., Xu, C., Luo, Y., Mao, X., Li, C.-P., Du, L., Ma, C., Jiang, Q., et al. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761545 · Cited by 2
Derived Topic Propagation Model Based on Topic Relevance and User Sentiment
Wei, S., Yang, Y., Guo, Y., Wang, R., Duan, S., & Xiao, Y. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3657357
Chen, N., Fang, H., Liu, J., Wei, W., Sakai, T., & Wu, X.-M. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3779384 · Cited by 3
“Inter-Inter-Coder” Analysis: A Multiset Approach for LLM Evaluation in Public Safety Analytics
Elrod, N. J., Kropczynski, J., & Grace, R. (2025) · IEEE World Forum on Public Safety Technology (WF-PST) · doi:10.1109/wf-pst65083.2025.00017
Audio Retrieval: Challenges and Solutions
Mansour, W. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21324-2_45
Reducing Human Effort to Validate LLM Relevance Judgements via Stratified Sampling
Merlo, S., Marchesin, S., Faggioli, G., & Ferro, N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_27 · Cited by 1
Query Harmfulness Prediction (QHP): A New Challenge for Safer Retrieval Systems
Carrera, X., Fernández-Pichel, M., & Losada, D. E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_4
On the Viability of Exploiting Large Language Models for Misinformation Annotation
Landrove, P., Fernández-Pichel, M., & Losada, D. E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_19
Revisiting Human-vs-LLM Judgments Using the TREC Podcast Track
Mansour, W. A., Culpepper, J. S., Mackenzie, J., & Yates, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_34
Large Language Models as Assessors: On the Impact of Relevance Scales
Zamolo, R., Lunardi, R., Soprano, M., Demartini, G., Mizzaro, S., & Roitero, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_24 · Cited by 1
Investigating the Potential of Large Language Models to Improve Urban Mobility
Feltus, C., & Kavka, C. (2025) · International Conference on Advanced Enterprise Information System (AEIS) · doi:10.1109/aeis68693.2025.00021
End-To-End LLM Systems: Data, Training, Evaluation, and Deployment
Dhanasekar, S. (2026) · International Conference on Machine Learning and Autonomous Systems (ICMLAS) · doi:10.1109/icmlas67792.2026.11483701
Does LLM Relevance Labelling Work for Arabic?
Alaofi, M., & Haouari, F. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809855
Reasoning with Large Language Models for Relevance Judgements
Geiger, L., Hettiachchi, D., Scholer, F., & Trippas, J. R. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820403
LLM-Driven Usefulness Judgment for Web Search Evaluation
Dewan, M., Liu, J., Gautam, A., & Shah, C. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820429 · Cited by 6
Prior-Data Fitted Networks as Tabular Foundation Models for Ranking in Low-Data Settings
Vos, D., Bhargav, S., de Rijke, M., & Oosterhuis, H. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820419
Does Form Affect Function? An Extended Study of LLM Re-Ranking Behavior
Goli, R., & Moffat, A. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820410
From Noise to Order: Learning to Rank via Denoising Diffusion
Ebrahimi, S., Mitra, B., Arabzadeh, N., Yuan, Y., Wu, H., Zarrinkalam, F., & Ebrahim Bagheri, A. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820437
Self-citationAuditing Query Drift: Do Users Actually Benefit from Pseudo-Relevance Feedback?
Liang, Z., McDonald, G., & Ounis, I. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809916
Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments
Christakopoulou, E., Patel, V., Velaga, H., & Gaikwad, S. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808460
Hybrid Pooling with LLMs via Relevance Context Learning
Otero, D., & Parapar, J. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809669
How Variability Influences Podcast Search: Queries, Transcriptions, and Judges
Mansour, W. A., Culpepper, J. S., Yates, A., & Mackenzie, J. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809668 · Cited by 1
When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessment
Yu, C., Li, H., Zuccon, G., Mackenzie, J., & Leelanupab, T. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809905 · Cited by 7
SAER: Scalable Assessment of E-commerce Recommendations using Large Language Models
Liu, X., Regulagedda, M. S., Patel, S., & Wong, W. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808462
Beyond Top-e: Simulation-Based Interactive Evaluation for Query Suggestions
Gabín, J., Parapar, J., & Wang, X. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808618
On the Robustness of LLM Re-Rankings
Goli, R., & Moffat, A. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809883
QAugment-Summary: Multi-stage Summarization and Question Decomposition for Open-Domain QA
Liu, M., Hou, K., Duan, R., Jiang, H., & Li, J. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-8447-5_24
NBQ: Next-Best-Question for Dynamic Profiling
Shi, Y., Wang, C., Wang, H., & Xiao, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817658
From Habits to Discovery: Deploying LLMs for Personalized Generative Recommendations at Spotify
D’Amico, E., Nadai, M. D., Chandar, P., Vohra, D., Lin, S., Lefarov, M., Gigioli, P., Penha, G., Kopysitsky, I., Senese, I. J., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818419 · Cited by 1
Overview of the CLEF 2026 LongEval Lab on Longitudinal Evaluation of Model Performance
Bouaraba, S., Breuer, T., Cancellieri, M., El-Ebshihy, A., Fröbe, M., Galuščáková, P., Goeuriot, L., Iturra-Bocaz, G., Keller, J., Knoth, P., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39150-6_32
Real-Time, Context-Specific Educational Metadata Enrichment of Learning Resources on the Web
Sebastian, R., & Hoppe, A. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846468
DAP: A diversity-aware pipeline for LLM-based utterance generation and API retrieval
dos Santos, V. G., Benatallah, B., Berro, A., & MacMahon, S. T. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108349
How Query Violations Relate to Relevance in Product Search
Paula, F. S. F., Kühn, D., & Moreira, V. P. (2026) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2026.249280
Metadata
- Authors
- Thomas, P., Spielman, S., Craswell, N., & Mitra, B.
- Affiliations
- Microsoft (United States) (US)
- Microsoft Research Montréal (Canada) (CA)
- Venue
- Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2024-07-10
- Pages
- 1930-1940
- OpenAlex
- W4400526908
- Cited by
- 168
- References
- 27
- Citations by year
- 2026: 42 · 2025: 74 · 2024: 51
- FWCI
- 63.52
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- large language models, search evaluation, label quality, user feedback, automatic annotation, prompt engineering, query difficulty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
