The DOI matches its DataCite record; 38 of the 38 references it cites were found in a registry or an index.
Optimizing Dense Retrieval Model Training with Hard Negatives
Open accessAbstract
Ranking has always been one of the top concerns in information retrieval researches. For decades, the lexical matching signal has dominated the ad-hoc retrieval process, but solely using this signal in retrieval may cause the vocabulary mismatch problem. In recent years, with the development of representation learning techniques, many researchers turn to Dense Retrieval (DR) models for better ranking performance. Although several existing DR models have already obtained promising results, their performance improvement heavily relies on the sampling of training examples. Many effective sampling strategies are not efficient enough for practical usage, and for most of them, there still lacks theoretical analysis in how and why performance improvement happens. To shed light on these research questions, we theoretically investigate different training strategies for DR models and try to explain why hard negative sampling performs better than random sampling. Through the analysis, we also find that there are many potential risks in static hard negative sampling, which is employed by many existing training methods. Therefore, we propose two training strategies named a Stable Training Algorithm for dense Retrieval (STAR) and a query-side training Algorithm for Directly Optimizing Ranking pErformance (ADORE), respectively. STAR improves the stability of DR training process by introducing random negatives. ADORE replaces the widely-adopted static hard negative sampling method with a dynamic one to directly optimize the ranking performance. Experimental results on two publicly available retrieval benchmark datasets show that either strategy gains significant improvements over existing competitive baselines and a combination of them leads to the best performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationRepresentation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Dense Passage Retrieval for Open-Domain Question Answering
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · arXiv · doi:10.48550/arxiv.2004.04906 · Cited by 5,311
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
Product Quantization for Nearest Neighbor Search
Jégou, H., Douze, M., & Schmid, C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.57 · Cited by 2,956
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Proceedings of the 19th International Conference on Artificial Intelligence and Statistics 2016
Murray, I., & Graham, M. M. (2016) · Cited by 2,115
Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval
Robertson, S., & Walker, S. (1994) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1007/978-1-4471-2099-5_24 · Cited by 1,947
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. U., & Hyvärinen, A. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 1,588
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Khattab, O., & Zaharia, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401075 · Cited by 1,301
From RankNet to LambdaRank to LambdaMART: An Overview
Burges, C. J. C. (2010) · Cited by 1,011
Large Batch Optimization for Deep Learning: Training BERT in 76 minutesPreprint
You, Y., Li, J., Reddi, S. J., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J. W., Keutzer, K., & Hsieh, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.00962 · Cited by 890
Nogueira, R., & Cho, K. (2019) · arXiv · doi:10.48550/arxiv.1901.04085 · Cited by 615
REALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574
Document Expansion by Query PredictionPreprint
Nogueira, R. F., Yang, W., Lin, J., & Cho, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08375 · Cited by 562
Embedding-based Retrieval in Facebook Search
Huang, J.-T., Sharma, A., Sun, S., Xia, L., Zhang, D., Pronin, P., Padmanabhan, J., Ottaviano, G., & Yang, L. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403305 · Cited by 386
Representation Learning: A Review and New PerspectivesPreprint
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.5538 · Cited by 235
A general approximation framework for direct optimization of information retrieval measures
Qin, T., Liu, T., & Li, H. (2009) · Information Retrieval · doi:10.1007/s10791-009-9124-x · Cited by 228
Yang, P., Fang, H., & Lin, J. (2018) · Journal of Data and Information Quality · doi:10.1145/3239571 · Cited by 222
Context-Aware Sentence/Passage Term Importance Estimation For First Stage RetrievalPreprint
Dai, Z., & Callan, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10687 · Cited by 222
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., & Overwijk, A. (2020) · arXiv · doi:10.48550/arxiv.2007.00808 · Cited by 204
RepBERT: Contextualized Text Embeddings for First-Stage RetrievalPreprint
Zhan, J., Mao, J., Liu, Y., Zhang, M., & Ma, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15498 · Cited by 152
Self-citationLearning Binary Codes for Maximum Inner Product Search
Shen, F., Liu, W., Zhang, S., Yang, Y., & Shen, H. T. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.472 · Cited by 142
Distilling Dense Representations for Ranking using Tightly-Coupled TeachersPreprint
Lin, S.-C., Yang, J.-H., & Lin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11386 · Cited by 141
Li, H., & Xu, J. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988058 · Cited by 116
Learning-to-Rank with BERT in TF-RankingPreprint
Han, S., Wang, X., Bendersky, M., & Najork, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08476 · Cited by 97
Sparse, Dense, and Attentional Representations for Text RetrievalPreprint
Luan, Y., Eisenstein, J., Toutanova, K., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00181 · Cited by 82
Complementing Lexical Retrieval with Semantic Residual EmbeddingPreprint
Gao, L., Dai, Z., Chen, T., Fan, Z., Van Durme, B., & Callan, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13969 · Cited by 73
Overview of the TREC 2019 deep learning track
Craswell, N., Mitra, B., Yilmaz, E., Campos, D., & Voorhees, E. M. (2020) · arXiv · doi:10.48550/arxiv.2003.07820 · Cited by 68
Qu, Y., Ding, Y., Liu, J., Liu, K., Ren, R., Zhao, W. X., Dong, D., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08191 · Cited by 66
Overview of the TREC 2021 deep learning trackPreprint
Craswell, N., Mitra, B., Yılmaz, E., Campos, D., & Lin, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.08191 · Cited by 43
Yan, M., Li, C., Wu, C., Bi, B., Wang, W., Xia, J., & Si, L. (2019) · Text REtrieval Conference · Cited by 24
Flagged first, then by citations
Research works printed but not linked: 2
- [19]19. Tie-Yan Liu. 2009. Learning to rank for information retrieval. Foundations and trends in information retrieval 3, 3 (2009), 225-331.
- [31]31. Peilin Yang, Hui Fang, and Jimmy Lin. 2018. Anserini: Reproducible ranking baselines using Lucene. Journal of Data and Information Quality (JDIQ) 10, 4 (2018), 1-20.
Cited by
Sorted by citations
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.272 · Cited by 353
The power of noise: Redefining retrieval for rag systems
Cuconasu, F., Trappolini, G., Siciliano, F., Filice, S., Campagnano, C., Maarek, Y., Tonellotto, N., & Silvestri, F. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657834 · Cited by 236
Dense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211
“, a sampler over the text collection based on some relevance model) is fixed or updated, hard negatives can be roughly divided into static hard negatives and dynamic hard negatives [98], [111].”
BackgroundMethodInfluentialInformation Retrieval: Recent Advances and Beyond
Hambarde, K., & Proença, H. P. M. C. (2023) · IEEE Access · doi:10.1109/access.2023.3295776 · Cited by 212
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
Ren, R., Qu, Y., Liu, J., Zhao, W. X., She, Q., Wu, H. R., Wang, H., & Wen, J.-R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.224 · Cited by 177
Metadata
- Authors
- Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M., & Ma, S.
- Affiliations
- Tsinghua University (CN)
- Renmin University of China (CN)
- Institute of Computing Technology (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 1503-1512
- arXiv
- 2104.08051
- OpenAlex
- W3155895380
- Cited by
- 251
- References
- 18
- Citations by year
- 2026: 20 · 2025: 35 · 2024: 62 · 2023: 76 · 2022: 46 · 2021: 12
- FWCI
- 40.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61732008, 61532011)
- Tsinghua University
- National Key Research and Development Program of China (2018YFC0831700, 61732008)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- dense retrieval, hard negative sampling, training strategies, ranking optimization, training stability, stars, ADORE, word matching, vocabulary mismatch, ad hoc retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
