The DOI matches its DataCite record; 38 of the 38 references it cites were found in a registry or an index.
Optimizing Dense Retrieval Model Training with Hard Negatives
Open accessAbstract
Ranking has always been one of the top concerns in information retrieval researches. For decades, the lexical matching signal has dominated the ad-hoc retrieval process, but solely using this signal in retrieval may cause the vocabulary mismatch problem. In recent years, with the development of representation learning techniques, many researchers turn to Dense Retrieval (DR) models for better ranking performance. Although several existing DR models have already obtained promising results, their performance improvement heavily relies on the sampling of training examples. Many effective sampling strategies are not efficient enough for practical usage, and for most of them, there still lacks theoretical analysis in how and why performance improvement happens. To shed light on these research questions, we theoretically investigate different training strategies for DR models and try to explain why hard negative sampling performs better than random sampling. Through the analysis, we also find that there are many potential risks in static hard negative sampling, which is employed by many existing training methods. Therefore, we propose two training strategies named a Stable Training Algorithm for dense Retrieval (STAR) and a query-side training Algorithm for Directly Optimizing Ranking pErformance (ADORE), respectively. STAR improves the stability of DR training process by introducing random negatives. ADORE replaces the widely-adopted static hard negative sampling method with a dynamic one to directly optimize the ranking performance. Experimental results on two publicly available retrieval benchmark datasets show that either strategy gains significant improvements over existing competitive baselines and a combination of them leads to the best performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationRepresentation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Dense Passage Retrieval for Open-Domain Question Answering
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · arXiv · doi:10.48550/arxiv.2004.04906 · Cited by 5,311
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
Product Quantization for Nearest Neighbor Search
Jégou, H., Douze, M., & Schmid, C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.57 · Cited by 2,956
Flagged first, then by citations
Research works printed but not linked: 2
- [19]19. Tie-Yan Liu. 2009. Learning to rank for information retrieval. Foundations and trends in information retrieval 3, 3 (2009), 225-331.
- [31]31. Peilin Yang, Hui Fang, and Jimmy Lin. 2018. Anserini: Reproducible ranking baselines using Lucene. Journal of Data and Information Quality (JDIQ) 10, 4 (2018), 1-20.
Cited by
Sorted by citations
Items 201–248 of 248
Deep Hashing Retrieval of Dynamic Corpora
Zou, A., Hao, W., Shao, T., Zhang, K., & Shen, Q. (2023) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp58490.2023.10248578
WSRR: Weighted Rank-Relevance Sampling for Dense Text Retrieval
Hambarde, K., & Proença, H. P. M. C. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-99-3758-5_22
Efficient Exploration and Exploitation for Sequential Music Recommendation
Pereira, B. L., Chaves, P. D. V., & Santos, R. L. T. (2023) · ACM Transactions on Recommender Systems · doi:10.1145/3625827
Improving Open-Domain Answer Sentence Selection by Distributed Clients with Privacy Preservation
Wang, W., Shen, T., Blumenstein, M., & Long, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46677-9_2
Stratified Ranking for Dense Passage Retrieval
Vu, L. L., Tien, D. N., & Dang-Minh, T. (2023) · International Symposium on Information and Communication Technology · doi:10.1145/3628797.3628810
Longtriever: a Pre-trained Long Text Encoder for Dense Document Retrieval
Yang, J., Liu, Z., Li, C., Sun, G., & Xie, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.223 · Cited by 1
Dense Retrieval as Indirect Supervision for Large-space Decision Making
Xu, N., Fei, W., Dong, M., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1002 · Cited by 3
Building an Efficient Retriever System with Limited Resources
Nguyen, Q. N., & Le, H. T. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-49529-8_5
SYNC: A Structurally Guided Hard Negative Curricula for Generalizable Neural Code Search
Naik, A., Das, S., Vedurada, J., & Aditya, S. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.7
DADR: A Denoising Approach for Dense Retrieval Model Training
Du, M., Li, S., Yu, J., Ma, J., Liu, H., Li, M., & Ji, B. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2387-4_11
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674128
Searching Personal Collections
Bendersky, M., Metzler, D., Najork, M., & Wang, X. (2024) · ACM eBooks · doi:10.1145/3674127.3674142
Query-driven Search Functionality
Li, L. (2024) · ACM eBooks · doi:10.1145/3674127.3674133
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702962
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674129
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702961
Hersh, W., & Roberts, K. E. (2024) · ACM eBooks · doi:10.1145/3674127.3702960
Bast, H., Kalmbach, J., Klumpp, T., & Korzen, C. (2024) · ACM eBooks · doi:10.1145/3674127.3674134
Foundations of Generative Information Retrieval
Ai, Q., Zhan, J., & Liu, Y. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_2 · Cited by 1
Self-citationZero-Shot Dense Retrieval Based on Query Expansion
Wu, Y., Chen, P., Ding, Z., & Yan, A. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1148-5_12
Exploring the Potential of Dimension Reduction in Building Efficient Dense Retrieval Systems
Xu, Z., Liu, Z., Gu, Y., & Yu, G. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1710-4_6
Rectifying and Discriminating Hard Negatives for Biomedical Retrieval Question Answering
Bai, J., Li, Z., Zhao, B., Li, C., Lin, C., & Rong, W. (2025) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2025.3551290
E2Rank: Efficient and Effective Layer-Wise Reranking
Campagnano, C., Mallia, A., Pertschuk, J., & Silvestri, F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_41
Optimizing Compound Retrieval Systems
Oosterhuis, H., Jagerman, R., Qin, Z., & Wang, X. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730051
Generating Synthetic Data with Large Language Models for Low-Resource Sentence Retrieval
Caffagni, D., Cocchi, F., Mambelli, A., Tutrone, F., Zanella, M., Cornia, M., & Cucchiara, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05409-8_4 · Cited by 2
Rethinking Data Use in Large Language Models
Min, S., Hajishirzi, H., & Zettlemoyer, L. (2025) · Computational Linguistics · doi:10.1162/coli.a.573
Nguyen, C. M., Tran, H. A. P., Thai, P. K., Nguyen, H. D., & Nguyen, V. T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06069-3_10
Dense Retrieval for Aggregated Search
Mei, L., Liu, S., Zhao, Z., Yan, Q., Mao, J., & Wen, J.-R. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761197
Self-citationOuyang, G., Yu, S., Wu, P., Xiao, R., & Luo, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3001-4_18
Learning Discrete Identifiers and Dense Vectors for Generative Retrieval
Xie, Y., Zou, L., Luo, X., Cai, H., Zhang, C., Dong, L., Lin, X., & Li, C. (2025) · ACM Transactions on Information Systems · doi:10.1145/3777550
Object Detection on Analog Water Meters Using Region-Based CNN
Fitriani, L., Sodikin, M. F., Rismala, R., Chalid, A., Siregar, C. A., & Avianty, R. (2025) · International Conference on Telecommunication Systems, Services, and Applications · doi:10.1109/tssa68467.2025.11303776
Improving Scientific Document Retrieval with Academic Concept Index
Lee, J., Lee, J., Kweon, W., Jin, B., Zhang, Y., Yoon, S., Lee, D., Yu, H., Han, J., & Kang, S. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3844616 · Cited by 2
A Loss Weighting Algorithm Based on In-batch Positive Passage Rankings for Dense Retrievers
Wang, Z., li, M., Chen, Y., Yang, Z., Qiao, Y., Li, X., Chen, S., & Ji, B. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11342524
Digital Precognition: Teaching Transformers to Map Cyber Threats Before Analysts Can
Waldrop, S., Doğdu, E., Choupani, R., & Mitchell, W. (2026) · IEEE International Conference on AI in Cybersecurity (ICAIC) · doi:10.1109/icaic67076.2026.11395807
Venkatesan, V., Saha, P., Vasudevan, G., Balakumar, G., & Balu, G. P. (2026) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc67433.2026.11393771
A Katz measure-enhanced incremental information retrieval approach based on spectral clustering
Xiang, L., Hao, W., Sun, F., Zheng, Y., Zou, S., & Zou, A. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-025-07944-x
Boualili, L. (2022) · theses.fr (ABES) · doi:10.70675/29d19654z89d4z4f44z85efz0add721cac69
Towards Effective, Efficient and Explainable Neural Information Retrieval
Formal, T. (2023) · theses.fr (ABES) · doi:10.70675/0e6d7bcbz4ddbz45bczb909z78ab26702a56
Wang, Z., Fan, Z., Li, Y., Zhang, X., Dai, P., & Tian, Z. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3682468
ARHN: Answer-Centric Relabeling of Hard Negatives with Open-Source LLMs for Dense Retrieval
Choi, H.-W., Choi, J., Jang, H., Kim, H., Yun, C., Jun, C., & Choi, S. J. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809628 · Cited by 2
A Replicability Study of Joint Product Quantisation for Effective Space-Efficient Dense Retrieval
Macdonald, C., Shen, Z., & Tonellotto, N. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808565 · Cited by 1
Yang, M., Zhao, Z., Chen, N., & Zhang, P. (2026) · Computers · doi:10.3390/computers15080493
Training Dense Retrievers with Multiple Positive Passages
Wang, B., Tang, M., Zhang, H., Guo, J., & Bi, K. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818053
Self-citationSTaR: a soft-labeling and triplet-aware retriever for efficient retrieval-augmented QA
Jiang, J.-Y., Chang, C., Li, Y., Huang, T.-C., & Roy, D. S. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02559-5
Revisiting Single-Table Retrieval: An Open Problem Under 360° Stress Tests
Yang, C., Jiang, Z., Li, J., Luo, Y., Fan, J., & Tang, N. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00262
A Multi-stage Method for Training Semantic Search Models
Leshchenko, S. S., Masliukhin, S. M., Makhnytkina, O. V., & Matveev, Y. N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37867-5_21
When Hard Negatives Are Not Enough: A Query-Difficulty Analysis of Dense Retrieval Fine-Tuning
Dura, M., Kaçıkan, E., & Onan, A. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11728071
Sequran: Composite Scoring and Reranking Techniques for Refining Quran Search Engine Results
Ramadita, R., Uriawan, W., & Zulfikar, W. B. (2026) · CommIT (Communication and Information Technology) Journal · doi:10.21512/commit.v20i2.13879
Metadata
- Authors
- Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M., & Ma, S.
- Affiliations
- Tsinghua University (CN)
- Renmin University of China (CN)
- Institute of Computing Technology (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 1503-1512
- arXiv
- 2104.08051
- OpenAlex
- W3155895380
- Cited by
- 251
- References
- 18
- Citations by year
- 2026: 20 · 2025: 35 · 2024: 62 · 2023: 76 · 2022: 46 · 2021: 12
- FWCI
- 40.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61732008, 61532011)
- Tsinghua University
- National Key Research and Development Program of China (2018YFC0831700, 61732008)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- dense retrieval, hard negative sampling, training strategies, ranking optimization, training stability, stars, ADORE, word matching, vocabulary mismatch, ad hoc retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
