The DOI matches its DataCite record.
Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data
Open accessAbstract
We investigate improving the retrieval effectiveness of embedding models through the lens of corpus-specific fine-tuning. Prior work has shown that fine-tuning with queries generated using a dataset's retrieval corpus can boost retrieval effectiveness for the dataset. However, we find that surprisingly, fine-tuning using the conventional InfoNCE contrastive loss often reduces effectiveness in state-of-the-art models. To overcome this, we revisit cross-encoder listwise distillation and demonstrate that, unlike using contrastive learning alone, listwise distillation can help more consistently improve retrieval effectiveness across multiple datasets. Additionally, we show that synthesizing more training data using diverse query types (such as claims, keywords, and questions) yields greater effectiveness than using any single query type alone, regardless of the query type used in evaluation. Our findings further indicate that synthetic queries offer comparable utility to human-written queries for training. We use our approach to train an embedding model that achieves state-of-the-art effectiveness among BERT embedding models. We release our model and both query generation and training code to facilitate further research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Cross-encoder listwise distillation with synthetic data can significantly improve dense retriever performance beyond conventional contrastive learning approaches.
“cross-encoder listwise distillation with synthetic data can significantly improve dense retriever performance beyond conventional contrastive learning approaches.”
From Cognis: Context-Aware Memory for Conversational AI Agents (2026) · cites this paper as Tamber et al., 2025
Citations
Cites
Still being checked.
Cited by
Sorted by citations
Dey, S., Braun, B., Ravipati, N., Wu, H., & Li, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.03628 · Cited by 5
Negative Sampling Techniques in Dense Retrieval: A Survey
Wischounig, L., Abdallah, A., & Jatowt, A. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.157 · Cited by 1
A Framework for Database Search with AI Models in Mass Spectrometry-Based Proteomics
Kalogeropoulos, K., Goey, J. V., Jenkins, T. P., & Eloff, K. (2026) · Journal of Proteome Research · doi:10.1021/acs.jproteome.5c01153 · Cited by 1
Cognis: Context-Aware Memory for Conversational AI AgentsPreprint
Daftari, P., Patel, K., Kapale, S., George, J., & Surendira, S. (2026) · arXiv (Cornell University) · doi:10.13140/rg.2.2.28445.27365 · Cited by 1
Cognis: Context-Aware Memory for Conversational AI Agents
Daftari, P., Patel, K., Kapale, S., George, J., & Surendira, S. (2026) · arXiv · doi:10.48550/arxiv.2604.19771
“Recent work on dense retrieval optimization includes CADET [Tamber et al., 2025], which demonstrates that cross-encoder listwise distillation with synthetic data can significantly improve dense retriever performance beyond conventional contrastive learning approaches.”
InfluentialNegative Sampling Techniques in Information Retrieval: A SurveyPreprint
Wischounig, L., Abdallah, A., & Jatowt, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.18005
Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation
Jang, Y., Hong, S., Moon, H., & Lim, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809843
Metadata
- Authors
- Tamber, M. S., Kazi, S., Sourabh, V., & Lin, J.
- ORCID
- Lin, Jimmy
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2505.19274
- OpenAlex
- W4414585866
- Cited by
- 1
- References
- 0
- Citations by year
- 2026: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- dense retrieval, synthetic query generation, synthetic training data, BERT embeddings, retrieval effectiveness, query types
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
