The DOI matches its Crossref record; 98 of the 103 references it cites were found in a registry or an index.
Rethinking search: making domain experts out of dilettantes
Open accessAbstract
When experiencing an information need, users want to engage with a domain expert, but often turn to an information retrieval system, such as a search engine, instead. Classical information retrieval systems do not answer information needs directly, but instead provide references to (hopefully authoritative) answers. Successful question answering systems offer a limited corpus created on-demand by human experts, which is neither timely nor scalable. Pre-trained language models, by contrast, are capable of directly generating prose that may be responsive to an information need, but at present they are dilettantes rather than domain experts - they do not have a true understanding of the world, they are prone to hallucinating, and crucially they are incapable of justifying their utterances by referring to supporting documents in the corpus they were trained over. This paper examines how ideas from classical information retrieval and pre-trained language models can be synthesized and evolved into systems that truly deliver on the promise of domain expert advice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Metzler et al. [133] has discussed several paradigms in which pre-trained language models can help IR systems answer user's information needs directly without listing references.”
From Foundations of GenIR (2025) · cites this paper as [133]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The shift toward generative information access is instructive: a vision articulated in IR [39] was realized mainly through LLMs developed outside the IR community (Section 1).”
From Agentic Memory Management Is the Next Frontier of Information Retrieval (2026) · cites this paper as [39]
In the citing paper
“Large language models (LLMs) are increasingly capable of providing users with advice in a wide range of professional domains, including legal advice.”
From (A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice (2024) · cites this paper as [50]
In the citing paper
“Most critically, as statistical models, LLMs can 'hallucinate' answers not grounded in their training data, severely compromising reliability [45, 50, 70].”
From (A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice (2024) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
The anatomy of a large-scale hypertextual Web search engine
Brin, S., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00110-x · Cited by 16,004
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Research works printed but not linked: 19
- Daniel Adiwardana , Minh-Thang Luong , David R So , Jamie Hall , Noah Fiedel , Romal Thoppilan , Zi Yang , Apoorv Kulshreshtha , Gaurav Nemade , Yifeng Lu , and Quoc V Le . Towards a humanlike open-domain chatbot. arXiv preprint arXiv:2001.09977 , 2020 . Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V Le. Towards a humanlike open-domain chatbot. arXiv preprint arXiv:2001.09977, 2020. · arXiv
- 10.5555/347319.346290 · DOI
- Tom B. Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel M. Ziegler , Jeffrey Wu , Clemens Winter , Christopher Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . Language models are few-shot learners . In Proceedings of the 34th Conference on Neural Information Processing Systems, NeurIPS '20 , 2020 . Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th Conference on Neural Information Processing Systems, NeurIPS '20, 2020.
- Avi Caciularu , Arman Cohan , Iz Beltagy , Matthew E Peters , Arie Cattan , and Ido Dagan . Cross-document language modeling. arXiv preprint arXiv:2101.00406 , 2021 . Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan, and Ido Dagan. Cross-document language modeling. arXiv preprint arXiv:2101.00406, 2021. · arXiv
- 10.5555/1516224 · DOI
- Rajarshi Das , Shehzaad Dhuliawala , Manzil Zaheer , and Andrew McCallum . Multi-step retriever-reader interaction for scalable open-domain question answering . In Proceedings of the 7th International Conference on Learning Representations, ICLR '19 , 2019 . Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. Multi-step retriever-reader interaction for scalable open-domain question answering. In Proceedings of the 7th International Conference on Learning Representations, ICLR '19, 2019.
- 10.5555/3295222.3295377 · DOI
- 10.5555/2999792.2999959 · DOI
- 10.5555/1859417 · DOI
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language models are unsupervised multitask learners . 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
- 10.5555/3305890.3306024 · DOI
- 10.5555/3327345.3327399 · DOI
- Yi Tay , Mostafa Dehghani , Samira Abnar , Yikang Shen , Dara Bahri , Philip Pham , Jinfeng Rao , Liu Yang , Sebastian Ruder , and Donald Metzler . Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006 , 2020 a. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020a. · arXiv
- 10.5555/3295222.3295349 · DOI
- 10.5555/2969442.2969540 · DOI
- Shuohang Wang , Mo Yu , Xiaoxiao Guo , Zhiguo Wang , Tim Klinger , Wei Zhang , Shiyu Chang , Gerry Tesauro , Bowen Zhou , and Jing Jiang . R3 : Reinforced ranker-reader for open-domain question answering . In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI '18 , pages 5981 -- 5988 , 2018 a. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. R3: Reinforced ranker-reader for open-domain question answering. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI '18, pages 5981--5988, 2018a.
- Tongzhou Wang , Jun-Yan Zhu , Antonio Torralba , and Alexei A Efros . Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018 b. Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018b. · arXiv
- Jason Weston , Sumit Chopra , and Antoine Bordes . Memory networks . In Proceedings of the 3rd International Conference on Learning Representations, ICLR '15 , 2015 . Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In Proceedings of the 3rd International Conference on Learning Representations, ICLR '15, 2015.
- Adams Wei Yu , David Dohan , Minh-Thang Luong , Rui Zhao , Kai Chen , Mohammad Norouzi , and Quoc V Le. Qanet : Combining local convolution with global self-attention for reading comprehension . In Proceedings of the 6th International Conference on Learning Representations, ICLR '18 , 2018 . Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. Qanet: Combining local convolution with global self-attention for reading comprehension. In Proceedings of the 6th International Conference on Learning Representations, ICLR '18, 2018.
Cited by
Sorted by citations
Precise Zero-Shot Dense Retrieval without Relevance Labels
Gao, L., Ma, X., Lin, J., & Callan, J. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.99 · Cited by 362
Large Language Models for Information Retrieval: A Survey
Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Chen, H., Liu, Z., Dou, Z., & Wen, J.-R. (2026) · ACM Transactions on Information Systems · doi:10.1145/3748304 · Cited by 225
Dense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 172
Information Retrieval: Recent Advances and Beyond
Hambarde, K., & Proença, H. P. M. C. (2023) · IEEE Access · doi:10.1109/access.2023.3295776 · Cited by 212
Metadata
- Authors
- Metzler, D., Tay, Y., Bahri, D., & Najork, M.
- Affiliations
- Google (United States) (US)
- Venue
- ACM SIGIR Forum
- Published
- 2021-06
- Volume
- 55
- Issue
- 1
- Pages
- 1-27
- ISSN
- 0163-5840 (ISSN-L)
- OpenAlex
- W3185146124
- Cited by
- 112
- References
- 103
- Citations by year
- 2026: 12 · 2025: 22 · 2024: 32 · 2023: 31 · 2022: 12 · 2021: 3
- FWCI
- 17.63
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM SIGIR Forum (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- information retrieval, question answering, pre-trained language models, hallucinations, information needs, retrieval-augmented generation, supporting documents, retrieval-augmented question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
