The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Sparse, Dense, and Attentional Representations for Text Retrieval
Open accessAbstract
Abstract Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document by its inner product with the query. We investigate the capacity of this architecture relative to sparse bag-of-words models and attentional neural networks. Using both theoretical and empirical analysis, we establish connections between the encoding dimension, the margin between gold and lower-ranked documents, and the document length, suggesting limitations in the capacity of fixed-length encodings to support precise retrieval of long documents. Building on these insights, we propose a simple neural model that combines the efficiency of dual encoders with some of the expressiveness of more costly attentional architectures, and explore sparse-dense hybrids to capitalize on the precision of sparse retrieval. These models outperform strong alternatives in large-scale retrieval.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Based on our review of existing work, we identify two primary retrieval methods suitable for non-parametric memory including sparse methods based on bag-of-words models and dense methods based on deep representation learning for vector encoding and relevance score calculation [269, 270, 232, 271].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [270]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Self-citationAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,043
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Decoding by Linear Programming
Candès, E. J., & Tao, T. (2005) · IEEE Transactions on Information Theory · doi:10.1109/tit.2005.858979 · Cited by 7,134
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691
A statistical interpretation of term specificity and its application in retrieval
Jones, K. S. (2021) · J. Documentation · doi:10.1108/eb026526 · Cited by 5,541
Approximate nearest neighbors: towards removing the curse of dimensionality
Indyk, P., & Motwani, R. (1998) · Proceedings of the thirtieth annual ACM symposium on Theory of computing - STOC '98 · doi:10.1145/276698.276876 · Cited by 4,188
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
The probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374
Extensions of Lipschitz mappings into a Hilbert space
Johnson, W. B., & Lindenstrauss, J. (1984) · Contemporary mathematics - American Mathematical Society · doi:10.1090/conm/026/737400 · Cited by 2,632
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Learning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. P. (2013) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2505515.2505665 · Cited by 2,120
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
Self-citationDatabase-friendly random projections: Johnson-Lindenstrauss with binary coins
Achlioptas, D. (2003) · Journal of Computer and System Sciences · doi:10.1016/s0022-0000(03)00025-4 · Cited by 1,374
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Khattab, O., & Zaharia, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401075 · Cited by 1,301
A Deep Relevance Matching Model for Ad-hoc Retrieval
Guo, J., Fan, Y., Ai, Q., & Croft, W. B. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983769 · Cited by 858
Latent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771
Self-citationExtensions of Lipschitz mappings into a Hilbert space
Johnson, W. B. (1984) · Medical Entomology and Zoology · Cited by 661
Nogueira, R., & Cho, K. (2019) · arXiv · doi:10.48550/arxiv.1901.04085 · Cited by 615
Document Expansion by Query PredictionPreprint
Nogueira, R. F., Yang, W., Lin, J., & Cho, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08375 · Cited by 562
Vempala, S. (2005) · DIMACS series in discrete mathematics and theoretical computer science · doi:10.1090/dimacs/065 · Cited by 541
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 539
Multi-Stage Document Ranking with BERTPreprint
Nogueira, R. F., Yang, W., Cho, K., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14424 · Cited by 510
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K., Liu, J., Bennett, P. N., Ahmed, J., & Overwikj, A. (2021) · International Conference on Learning Representations · Cited by 376
Breaking the Softmax Bottleneck: A High-Rank RNN Language ModelPreprint
Yang, Z., Dai, Z., Salakhutdinov, R., & Cohen, W. W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.03953 · Cited by 341
Scaling question answering to the Web
Kwok, C. C. T., Etzioni, O., & Weld, D. S. (2001) · International World Wide Web Conference (WWW) · doi:10.1145/371920.371973 · Cited by 336
Yang, P., Fang, H., & Lin, J. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080721 · Cited by 321
An Introduction to Neural Information Retrieval
Mitra, B., & Craswell, N. (2018) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000061 · Cited by 317
The TREC question answering track
Voorhees, E. M., & Tice, D. M. (2000) · Natural Language Engineering · doi:10.1017/s1351324901002789 · Cited by 293
Humeau, S., Shuster, K., Lachaux, M.-A., & Weston, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.01969 · Cited by 291
Learning Dense Representations for Entity Retrieval
Gillick, D., Kulkarni, S., Lansing, L., Presta, A., Baldridge, J., Ie, E., & García-Olano, D. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1049 · Cited by 244
Scaling question answering to the web
Kwok, C., Etzioni, O., & Weld, D. S. (2001) · ACM Transactions on Information Systems · doi:10.1145/502115.502117 · Cited by 234
Yang, P., Fang, H., & Lin, J. (2018) · Journal of Data and Information Quality · doi:10.1145/3239571 · Cited by 222
Context-Aware Sentence/Passage Term Importance Estimation For First Stage RetrievalPreprint
Dai, Z., & Callan, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10687 · Cited by 222
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., & Overwijk, A. (2020) · arXiv · doi:10.48550/arxiv.2007.00808 · Cited by 204
Simple Applications of BERT for Ad Hoc Document RetrievalPreprint
Yang, W., Zhang, H., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.10972 · Cited by 176
Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index
Seo, M., Lee, J., Kwiatkowski, T., Parikh, A. P., Farhadi, A., & Hajishirzi, H. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1436 · Cited by 160
Context-Aware Term Weighting For First Stage Passage Retrieval
Dai, Z., & Callan, J. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401204 · Cited by 150
Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning
Narasimhan, K., Yala, A., & Barzilay, R. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1261 · Cited by 146
Accelerating Large-Scale Inference with Anisotropic Vector QuantizationPreprint
Guo, R., Sun, P., Lindgren, E., Geng, Q., Simcha, D., Chern, F., & Kumar, S. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.esa.2022.54 · Cited by 146
Context-Aware Document Term Weighting for Ad-Hoc Search
Dai, Z., & Callan, J. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380258 · Cited by 136
Optimal Bounds for Johnson-Lindenstrauss Transforms and Streaming Problems with Subconstant Error
Jayram, T. S., & Woodruff, D. P. (2013) · ACM Transactions on Algorithms · doi:10.1145/2483699.2483706 · Cited by 114
No Training Required: Exploring Random Encoders for Sentence ClassificationPreprint
Wieting, J., & Kiela, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10444 · Cited by 102
Optimality of the Johnson-Lindenstrauss Lemma
Larsen, K. G., & Nelson, J. (2017) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2017.64 · Cited by 90
Limitations of Learning via Embeddings in Euclidean Half-Spaces
Ben-David, S., Eiron, N., & Simon, H. U. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44581-1_25 · Cited by 89
TREC Complex Answer Retrieval Overview.
Dietz, L., Verma, M., Radlinski, F., & Craswell, N. (2017) · Text REtrieval Conference · Cited by 84
APPROXIMATE NEAREST NEIGHBOR SEARCH IN HIGH DIMENSIONS
Andoni, A., Indyk, P., & Razenshteyn, I. (2019) · International Congress of Mathematicians (ICM) · doi:10.1142/9789813272880_0182 · Cited by 77
Complementing Lexical Retrieval with Semantic Residual EmbeddingPreprint
Gao, L., Dai, Z., Chen, T., Fan, Z., Van Durme, B., & Callan, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13969 · Cited by 73
Text REtrieval Conference (TREC)
(2017) · CRC Press eBooks · doi:10.1081/e-elis4-120044470 · Cited by 70
Overview of the TREC 2019 deep learning track
Craswell, N., Mitra, B., Yilmaz, E., Campos, D., & Voorhees, E. M. (2020) · arXiv · doi:10.48550/arxiv.2003.07820 · Cited by 68
Accelerating Large-Scale Inference with Anisotropic Vector Quantization
Guo, R., Geng, Q., Simcha, D., Chern, F., Sun, P., Lindgren, E., & Kumar, S. (2020) · International Conference on Machine Learning · Cited by 62
Approxiamate Nearest Neighbor Search in High DimensionsPreprint
Andoni, A., Indyk, P., & Razenshteyn, I. (2018) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 61
Quantization based Fast Inner Product SearchPreprint
Guo, R., Kumar, S., Choromański, K., & Simcha, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01469 · Cited by 59
An Introduction to Neural Information Retrieval t
Mitra, B., & Craswell, N. (2018) · now publishers, Inc. eBooks · doi:10.1561/9781680835335 · Cited by 57
Almost Optimal Explicit Johnson-Lindenstrauss Families
Kane, D. M., Meka, R., & Nelson, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22935-0_53 · Cited by 42
The text REtrieval conference (TREC)
Voorhees, E. M., & Harman, D. K. (1999) · ACM SIGIR Forum · doi:10.1145/344250.344252 · Cited by 37
A Compressed Sensing View of Unsupervised Text Embeddings, Bag-of-n-Grams, and LSTMs
Arora, S., Khodak, M., Saunshi, N., & Vodrahalli, K. (2018) · International Conference on Learning Representations · Cited by 33
Optimal Compression of Approximate Inner Products and Dimension Reduction
Alon, N., & Klartag, B. (2017) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2017.65 · Cited by 29
Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval
Hasibi, F., Fang, Y., & Aizawa, A. (2021) · doi:10.1145/3471158 · Cited by 27
Yan, M., Li, C., Wu, C., Bi, B., Wang, W., Xia, J., & Si, L. (2019) · Text REtrieval Conference · Cited by 24
Can Unconditional Language Models Recover Arbitrary Sentences?Preprint
Subramani, N., Bowman, S. R., & Cho, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.04944 · Cited by 17
Zero-shot Neural Retrieval via Domain-targeted Synthetic Query Generation.Preprint
Ma, J., Korotkov, I., Yang, Y., Hall, K., & McDonald, R. (2020) · arXiv (Cornell University) · Cited by 17
Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question GenerationPreprint
Ma, J., Korotkov, I., Yang, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14503 · Cited by 6
The TREC-8 Question Answering Track
Voorhees, E. M., & Tice, D. M. (2000) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/23pge4fs6tdc
Vempala, S. (2005) · DIMACS Series in Discrete Mathematics and Theoretical Computer Science · doi:10.1090/dimacs/065/01
Flagged first, then by citations
Cited by
Sorted by citations
Qu, Y., Ding, Y., Liu, J., Liu, K., Ren, R., Zhao, W. X., Dong, D., Wu, H. R., & Wang, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.466 · Cited by 597
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K., Liu, J., Bennett, P. N., Ahmed, J., & Overwikj, A. (2021) · International Conference on Learning Representations · Cited by 376
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.272 · Cited by 353
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling
Hofstätter, S., Lin, S.-C., Yang, J.-H., Lin, J., & Hanbury, A. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462891 · Cited by 374
Pretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 328
Metadata
- Authors
- Luan, Y., Eisenstein, J., Toutanova, K., & Collins, M.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Luan, Yi
- Eisenstein, Jacob
- Toutanova, Kristina
- Collins, Michael
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2021
- Volume
- 9
- Pages
- 329-345
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3157758108
- Cited by
- 363
- References
- 61
- Citations by year
- 2026: 46 · 2025: 48 · 2024: 39 · 2023: 81 · 2022: 69 · 2021: 61
- FWCI
- 56.52
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- text retrieval, dual-encoder models, sparse retrieval, bag-of-words model, document length, retrieval precision, large-scale retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
