RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
In various natural language processing tasks, passage retrieval and passage re-ranking are two key procedures in finding and ranking relevant information.Since both the two procedures contribute to the final performance, it is important to jointly optimize them in order to achieve mutual improvement.In this paper, we propose a novel joint training approach for dense passage retrieval and passage reranking.A major contribution is that we introduce the dynamic listwise distillation, where we design a unified listwise training approach for both the retriever and the re-ranker.During the dynamic distillation, the retriever and the re-ranker can be adaptively improved according to each other's relevance information.We also propose a hybrid data augmentation strategy to construct diverse training instances for listwise training approach.Extensive experiments show the effectiveness of our approach on both MSMARCO and Natural Questions datasets.Our code is available at https:// github.com/PaddlePaddle/RocketQA.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“0 base (Sun et al., 2020), which is a BERT-like (Devlin et al., 2019) model with 12-layer transformers and introduces a continual pre-training framework on multiple pre-trained tasks.”
BackgroundDense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
Learning to rank: from pairwise approach to listwise approach
Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., & Li, H. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273513 · Cited by 1,960 · cited 1×
“…in a pointwise or pairwise manner (Nogueira and Cho, 2019; Nogueira et al., 2019b), where the model is 1 Instead of considering the total order as in learning to rank (Cao et al., 2007), we use “listwise” to indicate that relevance scores are derived based on a candidate list. a r X i v : 2110 .”
MethodColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Khattab, O., & Zaharia, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401075 · Cited by 1,301 · cited 3×
“…retrievers include DPR (Karpukhin et al., 2020), DPR-E, ANCE (Xiong et al., 2020a), ME-BERT (Luan et al., 2021), NPRINC (Lu et al., 2020), ColBERT (Khattab and Zaharia, 2020) RocketQA (Qu et al., 2021), TAS-Balanced (Hof-stätter et al., 2021), ADORE+STAR (Zhan et al., 2021) and PAIR (Ren et al.,…”
BackgroundLatent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771 · cited 2×
“Existing approaches of learning dense passage retriever can be divided into two categories: (1) self-supervised pretraining for retrieval (Chang et al., 2020; Lee et al., 2019; Guu et al., 2020) and (2) fine-tuning pre-trained language models (PLMs) on labeled data (Lu et al.”
BackgroundERNIE 2.0: A Continual Pre-Training Framework for Language Understanding
Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., & Wang, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6428 · Cited by 766
Self-citation
Cited by
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.272 · Cited by 353
Query2doc: Query Expansion with Large Language Models
Wang, L., Yang, N., & Wei, F. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.585 · Cited by 214
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., & Wei, F. (2022) · arXiv · doi:10.48550/arxiv.2212.03533 · Cited by 135
Promptagator: Few-shot Dense Retrieval From 8 Examples
Dai, Z., Zhao, V. Y., Ma, J., Luan, Y., Ni, J., Lu, J., Bakalov, A., Guu, K., Hall, K. B., & Chang, M.-W. (2022) · arXiv · doi:10.48550/arxiv.2209.11755 · Cited by 43
“Distilling interaction based models into representation based models has been shown effective in closing the gap between the two (Hofstätter et al., 2020; Ren et al., 2021a; Lin et al., 2021; Ren et al., 2021b; Reddi et al., 2021; Zhang et al., 2022).”
MethodTowards Robust QA Evaluation via Open LLMs
Kamalloo, E., Upadhyay, S., & Lin, J. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657675 · Cited by 15
Metadata
- Authors
- Ren, R., Qu, Y., Liu, J., Zhao, W. X., She, Q., Wu, H., Wang, H., & Wen, J.-R.
- Venue
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Published
- 2021
- Pages
- 2825-2835
- OpenAlex
- W3206455169
- Cited by
- 149
- References
- 18
- FWCI
- 26.68
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. https://app.cochat.ai/papers/10.18653/v1/2021.emnlp-main.224
- The paper
- Ren, R., Qu, Y., Liu, J., Zhao, W. X., She, Q., Wu, H., Wang, H., & Wen, J.-R. (2021). RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2021.emnlp-main.224
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.