CoChat
Sign Up
PreprintCited by 5,311Topic ModelingInformation Retrieval and Search BehaviorMultimodal Machine Learning Applications

The DOI matches its DataCite record.

Dense Passage Retrieval for Open-Domain Question Answering

Open access
Karpukhin, Vladimir, Oğuz, Barlas, Min, Sewon, Lewis, Patrick, Wu, Ledell, Edunov, Sergey, Chen, Danqi, Yih, Wen-tau·2020·arXiv
DOI 10.48550/arxiv.2004.04906arXiv 2004.04906
Open this paper in CoChat — free accountOpen PDF

Abstract

Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets, our dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA benchmarks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2004.04906

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Hybrid models combining parametric and non-parametric memories can address some issues of pre-trained language models because knowledge can be directly revised and expanded, and accessed knowledge can be inspected and interpreted.

    “Hybrid models that combine parametric memory with non-parametric (i.e., retrieval-based) memories [20, 26, 48] can address some of these issues because knowledge can be directly revised and expanded, and accessed knowledge can be inspected and interpreted.”

    From Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) · cites this paper as [26]

    The Introduction describes a retrieval-based QA system that selects passages for a reader, and Related Work says DPR can be combined with generative models. These passages support the hybrid-model example, but our copy does not state that knowledge can be directly revised or expanded, or that accessed knowledge can be inspected and interpreted. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • RAG's retriever is initialized using DPR's retriever, which uses retrieval supervision on Natural Questions and TriviaQA.

    “It is worth noting that RAG's retriever is initialized using DPR's retriever, which uses retrieval supervision on Natural Questions and TriviaQA.”

    From Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) · cites this paper as [26]

    Section 5 (Experiments: Passage Retrieval) says DPR’s retriever was trained on Natural Questions and TriviaQA, supporting the retrieval-supervision portion. Section 7 (Related Work) says DPR has been combined with generation models, but our copy does not say that RAG’s retriever was initialized from DPR. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • There is significant work on learning to retrieve documents in information retrieval, more recently with pre-trained, neural language models similar to ours.

    “There is significant work on learning to retrieve documents in information retrieval, more recently with pre-trained, neural language models [44, 26] similar to ours.”

    From Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) · cites this paper as [26]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 47

  1. Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. Learn- ing to retrieve reasoning paths over Wikipedia graph for question answering. In International Conference on Learning Representations (ICLR).
  2. Petr Baudiš and Jan Šedivỳ. 2015. Modeling of the question answering task in the yodaqa system. In In- ternational Conference of the Cross-Language Eval- uation Forum for European Languages, pages 222- 228. Springer.
  3. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Empirical Methods in Nat- ural Language Processing (EMNLP).
  4. Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1994. Signature verifi- cation using a "Siamese" time delay neural network. In NIPS, pages 737-744.
  5. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89-96.
  6. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open- domain questions. In Association for Computa- tional Linguistics (ACL), pages 1870-1879.
  7. Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2019. Multi-step retriever- reader interaction for scalable open-domain question answering. In International Conference on Learn- ing Representations (ICLR).
  8. Scott Deerwester, Susan T Dumais, George W Fur- nas, Thomas K Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Jour- nal of the American society for information science, 41(6):391-407.
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In North American Association for Com- putational Linguistics (NAACL).
  10. David A Ferrucci. 2012. Introduction to "This is Wat- son". IBM Journal of Research and Development, 56(3.4):1-1.
  11. Ruiqi Guo, Sanjiv Kumar, Krzysztof Choromanski, and David Simcha. 2016. Quantization based fast inner product search. In Artificial Intelligence and Statis- tics, pages 482-490.
  12. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pa- supat, and Ming-Wei Chang. 2020. REALM: Retrieval-augmented language model pre-training. ArXiv, abs/2002.08909. · arXiv
  13. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun- hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Ku- mar, Balint Miklos, and Ray Kurzweil. 2017. Effi- cient natural language response suggestion for smart reply. ArXiv, abs/1705.00652. · arXiv
  14. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for Web search using clickthrough data. In ACM International Confer- ence on Information and Knowledge Management (CIKM), pages 2333-2338.
  15. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architec- tures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations (ICLR).
  16. Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open do- main question answering. ArXiv, abs/2007.01282. · arXiv
  17. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with GPUs. ArXiv, abs/1702.08734. · arXiv
  18. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading com- prehension. In Association for Computational Lin- guistics (ACL), pages 1601-1611.
  19. Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextu- alized late interaction over BERT. In ACM SIGIR Conference on Research and Development in Infor- mation Retrieval (SIGIR), pages 39-48.
  20. Brian Kulis. 2013. Metric learning: A survey. Foun- dations and Trends in Machine Learning, 5(4):287- 364.
  21. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: a benchmark for question answering research. Transactions of the Association of Compu- tational Linguistics (TACL).
  22. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Association for Com- putational Linguistics (ACL), pages 6086-6096.
  23. Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Association for Computa- tional Linguistics (ACL), pages 7871-7880.
  24. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS).
  25. Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In Association for Computa- tional Linguistics (ACL), pages 1736-1745.
  26. Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. A discrete hard EM ap- proach for weakly supervised question answering. In Empirical Methods in Natural Language Process- ing (EMNLP).
  27. Sewon Min, Danqi Chen, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2019b. Knowledge guided text re- trieval and reading for open domain question answer- ing. ArXiv, abs/1911.03868. · arXiv
  28. Dan Moldovan, Marius Pas ¸ca, Sanda Harabagiu, and Mihai Surdeanu. 2003. Performance issues and er- ror analysis in an open-domain question answering system. ACM Transactions on Information Systems (TOIS), 21(2):133-154.
  29. Stephen Mussmann and Stefano Ermon. 2016. Learn- ing and inference via maximum inner product search. In International Conference on Machine Learning (ICML), pages 2587-2596.
  30. Yixin Nie, Songhe Wang, and Mohit Bansal. 2019. Re- vealing the importance of semantic retrieval for ma- chine reading at scale. In Empirical Methods in Nat- ural Language Processing (EMNLP).
  31. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. ArXiv, abs/1901.04085. · arXiv
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. ArXiv, abs/1910.10683. · arXiv
  33. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 2383-2392.
  34. Parikshit Ram and Alexander G Gray. 2012. Maximum inner-product search using cone trees. In Proceed- ings of the 18th ACM SIGKDD international con- ference on Knowledge discovery and data mining, pages 931-939.
  35. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? ArXiv, abs/2002.08910. · arXiv
  36. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and be- yond. Foundations and Trends in Information Re- trieval, 3(4):333-389.
  37. Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019. Real-time open-domain question answering with dense-sparse phrase index. In Association for Com- putational Linguistics (ACL).
  38. Anshumali Shrivastava and Ping Li. 2014. Asymmet- ric LSH (ALSH) for sublinear time maximum inner product search (MIPS). In Advances in Neural In- formation Processing Systems (NIPS), pages 2321- 2329.
  39. Ellen M Voorhees. 1999. The TREC-8 question an- swering track report. In TREC, volume 99, pages 77-82.
  40. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018. Rˆ3: Reinforced ranker-reader for open-domain question answering. In Conference on Artificial Intelligence (AAAI).
  41. Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nalla- pati, and Bing Xiang. 2019. Multi-passage BERT: A globally normalized bert model for open-domain question answering. In Empirical Methods in Natu- ral Language Processing (EMNLP).
  42. Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gard- ner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question under- standing benchmark. Transactions of the Associa- tion of Computational Linguistics (TACL).
  43. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020a. Approximate nearest neighbor negative contrastive learning for dense text retrieval. ArXiv, abs/2007.00808. · arXiv
  44. Wenhan Xiong, Hankang Wang, and William Yang Wang. 2020b. Progressively pretrained dense corpus index for open-domain question answering. ArXiv, abs/2005.00038. · arXiv
  45. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019a. End-to-end open-domain question answering with bertserini. In North American Association for Com- putational Linguistics (NAACL), pages 72-77.
  46. Wei Yang, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019b. Data augmentation for bert fine-tuning in open-domain question answering. ArXiv, abs/1904.06652. · arXiv
  47. Wen-tau Yih, Kristina Toutanova, John C Platt, and Christopher Meek. 2011. Learning discriminative projections for text similarity measures. In Com- putational Natural Language Learning (CoNLL), pages 247-256.

Cited by

Sorted by citations

Metadata

Authors
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T.
Affiliations
  • Meta (Israel) (IL)
  • University of Washington (US)
  • Princeton University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
6769-6781
DOI
10.48550/arxiv.2004.04906
arXiv
2004.04906
OpenAlex
W3015883388
Cited by
5,311
References
0
Citations by year
2026: 1787 · 2025: 1150 · 2024: 733 · 2023: 764 · 2022: 493 · 2021: 323 · 2020: 57
FWCI
131.01
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • TIB Data Manager
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
dense passage retrieval, open-domain question answering, question answering, dense retrieval, passage retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review