Cited by 117Information Retrieval and Search BehaviorTopic ModelingRecommender Systems and Techniques

Rethinking search: making domain experts out of dilettantes

Open access
Metzler, Donald, Tay, Yi, Bahri, Dara, Najork, Marc·2021·ACM SIGIR Forum

Abstract

When experiencing an information need, users want to engage with a domain expert, but often turn to an information retrieval system, such as a search engine, instead. Classical information retrieval systems do not answer information needs directly, but instead provide references to (hopefully authoritative) answers. Successful question answering systems offer a limited corpus created on-demand by human experts, which is neither timely nor scalable. Pre-trained language models, by contrast, are capable of directly generating prose that may be responsive to an information need, but at present they are dilettantes rather than domain experts - they do not have a true understanding of the world, they are prone to hallucinating, and crucially they are incapable of justifying their utterances by referring to supporting documents in the corpus they were trained over. This paper examines how ideas from classical information retrieval and pre-trained language models can be synthesized and evolved into systems that truly deliver on the promise of domain expert advice.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3476415.3476428

Cites

97 of 97 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Metzler et al. discussed several paradigms in which pre-trained language models can help IR systems answer user's information needs directly without listing references.

    “Metzler et al. [133] has discussed several paradigms in which pre-trained language models can help IR systems answer user's information needs directly without listing references.”

    From Foundations of GenIR (2025) · cites this paper as [133]

    The Introduction (p. 2) says pre-trained language models can generate prose responsive to an information need, and Section 3 (pp. 6–7) proposes model-based IR in place of traditional retrieval and ranking. This supports direct answers rather than merely listing references, but the paper presents one proposed model-based paradigm, not several such paradigms; it also stresses the need to justify answers with supporting documents. (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • The shift toward generative information access is instructive: a vision articulated in IR was realized mainly through LLMs developed outside the IR community.

    “The shift toward generative information access is instructive: a vision articulated in IR [39] was realized mainly through LLMs developed outside the IR community (Section 1).”
  • Large language models (LLMs) are increasingly capable of providing users with advice in a wide range of professional domains, including legal advice.

    “Large language models (LLMs) are increasingly capable of providing users with advice in a wide range of professional domains, including legal advice.”
  • LLMs can 'hallucinate' answers not grounded in their training data, severely compromising reliability.

    “Most critically, as statistical models, LLMs can 'hallucinate' answers not grounded in their training data, severely compromising reliability [45, 50, 70].”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

6 of the 103 references couldn’t be matched to a known paper.

Research works printed but not linked: 14

  1. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V Le. Towards a humanlike open-domain chatbot. arXiv preprint arXiv:2001.09977 , 2020. · arXiv
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th Conference on Neural Information Processing Systems , NeurIPS '20, 2020.
  3. Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan, and Ido Dagan. Crossdocument language modeling. arXiv preprint arXiv:2101.00406 , 2021. · arXiv
  4. Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. Multi-step retrieverreader interaction for scalable open-domain question answering. In Proceedings of the 7th International Conference on Learning Representations , ICLR '19, 2019.
  5. Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. Co-pacrr: A context-aware neural ir model for ad-hoc retrieval. In Proceedings of the 1th ACM International Conference on Web Search and Data Mining , WSDM '18, pages 279-287, 2018.
  6. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NeurIPS '17, pages 6294-6305, 2017.
  7. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , EMNLP '16, pages 1400-1409, 2016.
  8. Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: A core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web , WWW '07, pages 697-706, 2007.
  9. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006 , 2020a. · arXiv
  10. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Proceedings of the 29th International Conference on Neural Information Processing Systems , NeurIPS '15, pages 2692-2700, 2015.
  11. Denny Vrandeˇ ci´ c and Markus Kr¨ otzsch. Wikidata: A free collaborative knowledgebase. Communications of the ACM , 57(10):78-85, September 2014. ISSN 0001-0782.
  12. Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018b. · arXiv
  13. Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In Proceedings of the 3rd International Conference on Learning Representations , ICLR '15, 2015.
  14. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. Qanet: Combining local convolution with global self-attention for reading comprehension. In Proceedings of the 6th International Conference on Learning Representations , ICLR '18, 2018.

Cited by

Sorted by citations

Authors
Metzler, D., Tay, Y., Bahri, D., & Najork, M.
Affiliations
Google (United States) (US)
Venue
ACM SIGIR Forum
Published
2021-06
Volume
55
Issue
1
Pages
1-27
ISSN
0163-5840 (ISSN-L)
OpenAlex
W3185146124
Cited by
112
References
103
Citations by year
2026: 12 · 2025: 22 · 2024: 32 · 2023: 31 · 2022: 12 · 2021: 3
FWCI
17.63
Type
journal article
Language
en
Open access
Locations
  • ACM SIGIR Forum (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
information retrieval, question answering, pre-trained language models, hallucinations, information needs, retrieval-augmented generation, supporting documents, retrieval-augmented question answering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review