CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks
Open accessAbstract
Knowledge-intensive language tasks (KILT) usually require a large body of information to provide correct answers. A popular paradigm to solve this problem is to combine a search system with a machine reader, where the former retrieves supporting evidences and the latter examines them to produce answers. Recently, the reader component has witnessed significant advances with the help of large-scale pre-trained generative models. Meanwhile most existing solutions in the search component rely on the traditional "index-retrieve-then-rank'' pipeline, which suffers from large memory footprint and difficulty in end-to-end optimization. Inspired by recent efforts in constructing model-based IR models, we propose to replace the traditional multi-step search pipeline with a novel single-step generative model, which can dramatically simplify the search process and be optimized in an end-to-end manner. We show that a strong generative retrieval model can be learned with a set of adequately designed pre-training tasks, and be adopted to improve a variety of downstream KILT tasks with further fine-tuning. We name the pre-trained generative retrieval model as CorpusBrain as all information about the corpus is encoded in its parameters without the need of constructing additional index. Empirical results show that CorpusBrain can significantly outperform strong baselines for the retrieval task on the KILT benchmark and establish new state-of-the-art downstream performances. We also show that CorpusBrain works well under zero- and low-resource settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Self-citationA vector space model for automatic indexing
Salton, G., Wong, A., & Yang, C. S. (1975) · Communications of the ACM · doi:10.1145/361219.361220 · Cited by 7,521 · see citing works
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
The probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851 · see citing works
Relevance weighting of search terms
Robertson, S. E., & Jones, K. S. (1976) · Journal of the American Society for Information Science · doi:10.1002/asi.4630270302 · Cited by 2,094 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
Reading Wikipedia to answer open-domain questions
Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1171 · Cited by 1,546 · see citing works
FEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · see citing works
Generating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · Cited by 1,266 · see citing works
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1912.08777 · Cited by 1,206 · see citing works
Robust Disambiguation of Named Entities in Text
Hoffart, J., Yosef, M. A., Bordino, I., Fürstenau, H., Pinkal, M., Spaniol, M., Taneva, B., Thater, S., & Weikum, G. (2011) · Cited by 922 · see citing works
A Deep Relevance Matching Model for Ad-hoc Retrieval
Guo, J., Fan, Y., Ai, Q., & Croft, W. B. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983769 · Cited by 821 · see citing works
Self-citationLatent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771 · see citing works
(2006) · doi:10.7551/mitpress/7503.003.0029 · Cited by 753 · see citing works
Wizard of Wikipedia: Knowledge-Powered Conversational agentsPreprint
Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., & Weston, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01241 · Cited by 610 · see citing works
Zero-Shot Relation Extraction via Reading Comprehension
Levy, O., Seo, M., Choi, E., & Zettlemoyer, L. (2017) · Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) · doi:10.18653/v1/k17-1034 · Cited by 482 · cited 2× · see citing works
“26 2,284,168 5,000 5,000 zsRE [27] Slot Filling 1 147,909 3,724 4,966 NQ [24] Open Domain QA 1.”
BackgroundZero-Shot Relation Extraction via Reading Comprehension
Levy, O., Seo, M., Choi, E., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1706.04115 · Cited by 482
Scalable Zero-shot Entity Linking with Dense Entity Retrieval
Wu, L., Petroni, F., Josifoski, M., Riedel, S., & Zettlemoyer, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.519 · Cited by 433 · see citing works
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K., Liu, J., Bennett, P. N., Ahmed, J., & Overwikj, A. (2021) · International Conference on Learning Representations · Cited by 376 · see citing works
KILT: a Benchmark for Knowledge Intensive Language Tasks
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.200 · Cited by 357 · see citing works
Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search
Dai, Z., Xiong, C., Callan, J., & Liu, Z. (2018) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3159652.3159659 · Cited by 350 · see citing works
ELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319 · see citing works
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling
Hofstätter, S., Lin, S.-C., Yang, J.-H., Lin, J., & Hanbury, A. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462891 · Cited by 318 · see citing works
Autoregressive Entity RetrievalPreprint
Cao, N. D., Izacard, G., Riedel, S., & Petroni, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00904 · Cited by 307 · see citing works
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTPreprint
Khattab, O., & Zaharia, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12832 · Cited by 270 · see citing works
Optimizing Dense Retrieval Model Training with Hard Negatives
Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M., & Ma, S. (2021) · arXiv · doi:10.48550/arxiv.2104.08051 · Cited by 251 · see citing works
Self-citationOptimizing Dense Retrieval Model Training with Hard Negatives
Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M., & Ma, S. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462880 · Cited by 248 · see citing works
Self-citation(2018) · doi:10.63317/2yu5nu76n7bu · Cited by 189 · see citing works
Robust named entity disambiguation with random walks
Guo, Z., & Barbosa, D. (2017) · Semantic Web · doi:10.3233/sw-170273 · Cited by 136 · see citing works
Context-Aware Term Weighting For First Stage Passage Retrieval
Dai, Z., & Callan, J. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401204 · Cited by 131 · see citing works
Semantic Models for the First-Stage Retrieval: A Comprehensive Review
Guo, J., Cai, Y., Fan, Y., Sun, F., Zhang, R., & Cheng, X. (2022) · ACM Transactions on Information Systems · doi:10.1145/3486250 · Cited by 130 · see citing works
Self-citationLearning to Reweight Terms with Distributed Representations
Zheng, G., & Callan, J. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767700 · Cited by 117 · see citing works
Rethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112 · see citing works
Learning to Rank for Information Retrieval and Natural Language Processing
Li, H. (2011) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-02141-1 · Cited by 91 · see citing works
Transformer Memory as a Differentiable Search IndexPreprint
Tay, Y., Tran, V. Q., Dehghani, M., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.06991 · Cited by 80 · see citing works
Autoregressive Search Engines: Generating Substrings as Document Identifiers
Bevilacqua, M., Ottaviano, G., Lewis, P., Yih, W.-T., Riedel, S., & Petroni, F. (2022) · arXiv · doi:10.48550/arxiv.2204.10628 · Cited by 80 · see citing works
Learning to Rank for Information Retrieval
Liu, T. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601982452 · Cited by 75 · see citing works
GERE: Generative Evidence Retrieval for Fact Verification
Chen, J., Zhang, R., Guo, J., Fan, Y., & Cheng, X. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531827 · Cited by 66 · see citing works
Self-citationMulti-Task Retrieval for Knowledge-Intensive Tasks
Maillard, J., Karpukhin, V., Petroni, F., Yih, W.-T., Oğuz, B., Stoyanov, V., & Ghosh, G. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.89 · Cited by 49 · see citing works
Pre-training for Ad-hoc Retrieval
Ma, Z., Dou, Z., Xu, W. X., Zhang, X., Jiang, H., Cao, Z., & Wen, J.-R. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482286 · Cited by 36 · see citing works
Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
Ma, X., Guo, J., Zhang, R., Fan, Y., & Cheng, X. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531772 · Cited by 35 · see citing works
Self-citationB-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc RetrievalPreprint
Ma, X., Guo, J., Zhang, R., Fan, Y., Li, Y., & Cheng, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.09791 · Cited by 18 · see citing works
Self-citationTABi: Type-Aware Bi-Encoders for Open-Domain Entity RetrievalPreprint
Leszczynski, M., Fu, D. Y., Chen, M. F., & Ré, C. (2022) · Findings · doi:10.48550/arxiv.2204.08173 · Cited by 13 · see citing works
DynamicRetriever: A Pre-training Model-based IR System with Neither Sparse nor Dense IndexPreprint
Zhou, Y., Yao, J., Dou, Z., Wu, L., & Wen, J.-R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.00537 · Cited by 10 · see citing works
Frej, J., Mulhem, P., Schwab, D., et al. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401211 · Cited by 9 · see citing works
TABi: Type-Aware Bi-Encoders for Open-Domain Entity Retrieval
Leszczynski, M., Fu, D. Y., Chen, M. F., & Ré, C. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.169 · Cited by 7 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 5
- [3]3. Christopher Burges, Robert Ragno, and Quoc Le. 2006. Learning to rank with nonsmooth cost functions. NIPS 19 (2006).
- [4]4. Wei-Cheng Chang, X Yu Felix, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2019. Pre-training Tasks for Embedding-based Large-scale Retrieval. In ICLR.
- [9]9. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Au- toregressive Entity Retrieval. In ICLR.
- [11]11. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In LREC.
- [35]35. Zhengyi Ma, Zhicheng Dou, Wei Xu, Xinyu Zhang, Hao Jiang, Zhao Cao, and Ji-Rong Wen. 2021. Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need. In CIKM. 1212-1221.
Cited by
Sorted by citations
Large Language Models for Information Retrieval: A Survey
Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Chen, H., Liu, Z., Dou, Z., & Wen, J.-R. (2026) · ACM Transactions on Information Systems · doi:10.1145/3748304 · Cited by 225 · see citing works
Dense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211 · see citing works
Continual Learning for Generative Retrieval over Dynamic Corpora
Chen, J., Zhang, R., Guo, J., de Rijke, M., Chen, W., Fan, Y., & Cheng, X. (2023) · Proceedings of the 32nd ACM International Conference on Information and Knowledge Management · doi:10.1145/3583780.3614821 · Cited by 40 · see citing works
Self-citationA Unified Generative Retriever for Knowledge-Intensive Language Tasks via Prompt Learning
Chen, J., Zhang, R., Guo, J., de Rijke, M., Liu, Y., Fan, Y., & Cheng, X. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591631 · Cited by 32 · see citing works
Self-citationRecent Advances in Generative Information Retrieval
Tang, Y., Zhang, R., Guo, J., & de Rijke, M. (2023) · Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region · doi:10.1145/3624918.3629547 · Cited by 22 · see citing works
Self-citationZeng, H., Luo, C., & Zamani, H. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657746 · Cited by 22 · see citing works
TOME: A Two-stage Approach for Model-based Retrieval
Ren, R., Zhao, W. X., Liu, J., Wu, H., Wen, J.-R., & Wang, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.336 · Cited by 14 · see citing works
Generative Retrieval for Book Search
Tang, Y., Zhang, R., Guo, J., de Rijke, M., Liu, S., Wang, S., Yin, D., & Cheng, X. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 · doi:10.1145/3690624.3709435 · Cited by 5 · see citing works
Self-citationRecent Advances in Generative Information Retrieval
Tang, Y., Zhang, R., Ren, Z., Guo, J., & de Rijke, M. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3661379 · Cited by 4 · see citing works
Self-citationLightweight and Direct Document Relevance Optimization for Generative Information Retrieval
Mekonnen, K. A., Tang, Y., & de Rijke, M. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730023 · Cited by 3 · see citing works
- Authors
- Chen, J., Zhang, R., Guo, J., Liu, Y., Fan, Y., & Cheng, X.
- Affiliations
- Institute of Computing Technology (CN)
- Beijing National Research Center for Information Science and Technology (CN)
- Venue
- Proceedings of the 31st ACM International Conference on Information & Knowledge Management
- Publisher
- ACM
- Published
- 2022-10-17
- Pages
- 191-200
- ISSN
- 2155-0751 (ISSN-L)
- OpenAlex
- W4292215729
- Cited by
- 57
- References
- 53
- Citations by year
- 2026: 2 · 2025: 13 · 2024: 25 · 2023: 15 · 2022: 2
- FWCI
- 12.29
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- the National Natural Science Foundation of China (62006218, 61902381)
- the Lenovo-CAS Joint Lab Youth Scientist Project
- the Youth Innovation Promotion Association CAS (20144310, 2021100)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- generative retrieval, pre-training, end-to-end optimization, machine reading
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
