The DOI matches its Crossref record; 98 of the 103 references it cites were found in a registry or an index.
Rethinking search: making domain experts out of dilettantes
Open accessAbstract
When experiencing an information need, users want to engage with a domain expert, but often turn to an information retrieval system, such as a search engine, instead. Classical information retrieval systems do not answer information needs directly, but instead provide references to (hopefully authoritative) answers. Successful question answering systems offer a limited corpus created on-demand by human experts, which is neither timely nor scalable. Pre-trained language models, by contrast, are capable of directly generating prose that may be responsive to an information need, but at present they are dilettantes rather than domain experts - they do not have a true understanding of the world, they are prone to hallucinating, and crucially they are incapable of justifying their utterances by referring to supporting documents in the corpus they were trained over. This paper examines how ideas from classical information retrieval and pre-trained language models can be synthesized and evolved into systems that truly deliver on the promise of domain expert advice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Metzler et al. [133] has discussed several paradigms in which pre-trained language models can help IR systems answer user's information needs directly without listing references.”
From Foundations of GenIR (2025) · cites this paper as [133]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The shift toward generative information access is instructive: a vision articulated in IR [39] was realized mainly through LLMs developed outside the IR community (Section 1).”
From Agentic Memory Management Is the Next Frontier of Information Retrieval (2026) · cites this paper as [39]
In the citing paper
“Large language models (LLMs) are increasingly capable of providing users with advice in a wide range of professional domains, including legal advice.”
From (A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice (2024) · cites this paper as [50]
In the citing paper
“Most critically, as statistical models, LLMs can 'hallucinate' answers not grounded in their training data, severely compromising reliability [45, 50, 70].”
From (A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice (2024) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
The anatomy of a large-scale hypertextual Web search engine
Brin, S., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00110-x · Cited by 16,004
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Flagged first, then by citations
5 of the 103 references couldn’t be matched to a known paper.
Research works printed but not linked: 19
- Daniel Adiwardana , Minh-Thang Luong , David R So , Jamie Hall , Noah Fiedel , Romal Thoppilan , Zi Yang , Apoorv Kulshreshtha , Gaurav Nemade , Yifeng Lu , and Quoc V Le . Towards a humanlike open-domain chatbot. arXiv preprint arXiv:2001.09977 , 2020 . Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V Le. Towards a humanlike open-domain chatbot. arXiv preprint arXiv:2001.09977, 2020. · arXiv
- 10.5555/347319.346290 · DOI
- Tom B. Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel M. Ziegler , Jeffrey Wu , Clemens Winter , Christopher Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . Language models are few-shot learners . In Proceedings of the 34th Conference on Neural Information Processing Systems, NeurIPS '20 , 2020 . Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th Conference on Neural Information Processing Systems, NeurIPS '20, 2020.
- Avi Caciularu , Arman Cohan , Iz Beltagy , Matthew E Peters , Arie Cattan , and Ido Dagan . Cross-document language modeling. arXiv preprint arXiv:2101.00406 , 2021 . Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan, and Ido Dagan. Cross-document language modeling. arXiv preprint arXiv:2101.00406, 2021. · arXiv
- 10.5555/1516224 · DOI
- Rajarshi Das , Shehzaad Dhuliawala , Manzil Zaheer , and Andrew McCallum . Multi-step retriever-reader interaction for scalable open-domain question answering . In Proceedings of the 7th International Conference on Learning Representations, ICLR '19 , 2019 . Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. Multi-step retriever-reader interaction for scalable open-domain question answering. In Proceedings of the 7th International Conference on Learning Representations, ICLR '19, 2019.
- 10.5555/3295222.3295377 · DOI
- 10.5555/2999792.2999959 · DOI
- 10.5555/1859417 · DOI
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language models are unsupervised multitask learners . 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
- 10.5555/3305890.3306024 · DOI
- 10.5555/3327345.3327399 · DOI
- Yi Tay , Mostafa Dehghani , Samira Abnar , Yikang Shen , Dara Bahri , Philip Pham , Jinfeng Rao , Liu Yang , Sebastian Ruder , and Donald Metzler . Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006 , 2020 a. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020a. · arXiv
- 10.5555/3295222.3295349 · DOI
- 10.5555/2969442.2969540 · DOI
- Shuohang Wang , Mo Yu , Xiaoxiao Guo , Zhiguo Wang , Tim Klinger , Wei Zhang , Shiyu Chang , Gerry Tesauro , Bowen Zhou , and Jing Jiang . R3 : Reinforced ranker-reader for open-domain question answering . In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI '18 , pages 5981 -- 5988 , 2018 a. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. R3: Reinforced ranker-reader for open-domain question answering. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI '18, pages 5981--5988, 2018a.
- Tongzhou Wang , Jun-Yan Zhu , Antonio Torralba , and Alexei A Efros . Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018 b. Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018b. · arXiv
- Jason Weston , Sumit Chopra , and Antoine Bordes . Memory networks . In Proceedings of the 3rd International Conference on Learning Representations, ICLR '15 , 2015 . Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In Proceedings of the 3rd International Conference on Learning Representations, ICLR '15, 2015.
- Adams Wei Yu , David Dohan , Minh-Thang Luong , Rui Zhao , Kai Chen , Mohammad Norouzi , and Quoc V Le. Qanet : Combining local convolution with global self-attention for reading comprehension . In Proceedings of the 6th International Conference on Learning Representations, ICLR '18 , 2018 . Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. Qanet: Combining local convolution with global self-attention for reading comprehension. In Proceedings of the 6th International Conference on Learning Representations, ICLR '18, 2018.
Cited by
Sorted by citations
Items 101–119 of 119
User control of search algorithms would improve science
Dunivin, Z. O., & Smaldino, P. E. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02276-5
Generative Information Retrieval
Najork, M. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736806
Self-citationKG-ASI: A Knowledge Graph Enhanced Model-Based Retriever for Document Retrieval
Bi, Z., Shen, H., Zhang, W., & Zhong, W. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-93257-1_9
Nagatsuka, T. (2025) · Joho Chishiki Gakkaishi · doi:10.2964/jsik_2025_027
Confirmation, Framing, and Position Biases in LLM Responses
Магдановна, Е. Л., Firsov, A., & Kamps, J. (2026) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3786304.3787879
Efficient Optimization of Hierarchical Identifiers for Generative Recommendation
Valeau, F., Boufalis, O., Gkotsi, P., Rosenthal, J., & Vos, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21324-2_6
Zhang, A., Zhang, Y., Xie, H., & Zhang, X. (2026) · Discover Applied Sciences · doi:10.1007/s42452-026-08614-0
Boualili, L. (2022) · theses.fr (ABES) · doi:10.70675/29d19654z89d4z4f44z85efz0add721cac69
Towards Effective, Efficient and Explainable Neural Information Retrieval
Formal, T. (2023) · theses.fr (ABES) · doi:10.70675/0e6d7bcbz4ddbz45bczb909z78ab26702a56
Proactive models for open-domain conversational search
Erbacher, P. (2024) · theses.fr (ABES) · doi:10.70675/f7d68861zc694z49f1zb130z675cbf38ed1c
Répondre aux questions visuelles à propos d'entités nommées
Lerner, P. J. (2023) · theses.fr (ABES) · doi:10.70675/78c96766zfc95z4da8zb67aze67555da6c75
Design Knowledge Quality in Human-AI Co-Design
Nabavian, S., & Parsons, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-28570-6_11
The intention turn: AI and the first democratization of understanding
Gazit, L. (2026) · AI and Ethics · doi:10.1007/s43681-026-01208-x
Incremental refinement of relevance rankings: Balancing relevance depth and scope
Akbulut, M., & Tonta, Y. (2026) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70101
Information Seeking Behavior in LLM-Based RAG: Mental Models and Missing Information
Sotic, B. N., & Kamps, J. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808537 · Cited by 1
Tu, Y. L., Miao, S., Su, W., Liu, Y., & Ai, Q. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808536 · Cited by 3
Javadi, V. S., Naseer, S., Ather, A., Flek, L., & Trippas, J. R. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820401
What Does It Mean to “Google”? “AI Overviews” and “AI Mode” as Novel Information Objects
Silva, E. (2026) · Critical AI · doi:10.1215/2834703x-12689846
Metadata
- Authors
- Metzler, D., Tay, Y., Bahri, D., & Najork, M.
- Affiliations
- Google (United States) (US)
- Venue
- ACM SIGIR Forum
- Published
- 2021-06
- Volume
- 55
- Issue
- 1
- Pages
- 1-27
- ISSN
- 0163-5840 (ISSN-L)
- OpenAlex
- W3185146124
- Cited by
- 112
- References
- 103
- Citations by year
- 2026: 12 · 2025: 22 · 2024: 32 · 2023: 31 · 2022: 12 · 2021: 3
- FWCI
- 17.63
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM SIGIR Forum (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- information retrieval, question answering, pre-trained language models, hallucinations, information needs, retrieval-augmented generation, supporting documents, retrieval-augmented question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
