Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations
Open accessThe DOI matches its Crossref record; 39 of the 41 references it cites were found in a registry or an index.
Abstract
Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations. It aims to provide effective, reproducible, and easy-to-use first-stage retrieval in a multi-stage ranking architecture. Our toolkit is self-contained as a standard Python package and comes with queries, relevance judgments, pre-built indexes, and evaluation scripts for many commonly used IR test collections. We aim to support, out of the box, the entire research lifecycle of efforts aimed at improving ranking with modern neural approaches. In particular, Pyserini supports sparse retrieval (e.g., BM25 scoring using bag-of-words representations), dense retrieval (e.g., nearest-neighbor search on transformer-encoded representations), as well as hybrid retrieval that integrates both approaches. This paper provides an overview of toolkit features and presents empirical results that illustrate its effectiveness on two popular ranking tasks. Around this toolkit, our group has built a culture of reproducibility through shared norms and tools that enable rigorous automated testing.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Interestingly, the classic BM25 model [18] remains competitive, surpassing some semantic models trained on search behavior datasets.”
From Generative Agents Navigating Digital Libraries (2025) · cites this paper as [18]
In the citing paper
“Sparse lexical methods, from classical BM25 [Lin et al., 2021] to lightweight full-text indexes such as SQLite FTS5, complement dense retrieval by matching exact terms.”
From MemX: A Local-First Long-Term Memory System for AI Assistants (2026) · cites this paper as Lin et al., 2021
Citations
Cites
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 1×
“This is particularly the case for work on deep learning today, since the major toolkits (PyTorch [29] and Tensorflow [1]) have both adopted Python as their front-end This work is licensed under a Creative Commons Attribution International 4.”
BackgroundTensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“also provides document expansion baselines using our doc2query method [28]; the latest model uses T5 [33] as described in Nogueira and Lin [26].”
MethodTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jégou, H. (2017) · arXiv · doi:10.48550/arxiv.1702.08734 · Cited by 4,140 · cited 3×
“Row (2a) refers to brute-force scans over the document vectors in Faiss [12], which provides exact nearest-neighbor search.”
BackgroundMethodDense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Malkov, Y. A., & Yashunin, D. A. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889473 · Cited by 1,904 · cited 3×
“To support dense and hybrid retrieval, Pyserini integrates Facebook’s Faiss library for efficient similarity search over dense vectors [12], which in turns integrates the HNSW library [23] to support low-latency querying.”
MethodColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Khattab, O., & Zaharia, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401075 · Cited by 1,301 · cited 8×
“Row (3a) shows the results of combining TCT-ColBERT with BM25 bag-of-words search over the original texts, while row (3b) shows results that combine document expansion using doc2query with the T5 model.”
MethodInfluential
Research works printed but not linked: 6
- Sebastian Hofstatter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. arXiv:2010.02666 (2020). · arXiv
- Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2021. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. arXiv:2010.02666 (2021). · arXiv
- Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Goharian. 2021. Simplified Data Wrangling with textttir_datasets. arXiv:2103.02280 (2021). · arXiv
- Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021 c. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. arXiv:2101.05667 (2021). · arXiv
- Chenyan Xiong, Zhenghao Liu, Si Sun, Zhuyun Dai, Kaitao Zhang, Shi Yu, Zhiyuan Liu, Hoifung Poon, Jianfeng Gao, and Paul Bennett. 2020 a. CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search. arXiv:2011.01580 (2020). · arXiv
- Yongze Yu, Jussi Karlgren, Hamed Bonab, Ann Clifton, Md Iftekhar Tanveer, and Rosie Jones. 2020. Spotify at the TREC 2020 Podcasts Track: Segment Retrieval. In Proceedings of the Twenty-Ninth Text REtrieval Conference (TREC 2020).
Cited by
Autonomous chemical research with large language models
Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023) · Nature · doi:10.1038/s41586-023-06792-0 · Cited by 1,050
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., & Liu, Z. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.137 · Cited by 757
“Following the official benchmark, we evaluate our method using Pyserini (Lin et al., 2021), and use nDCG@10 as the primary evaluation metric (Recall@100 is also measured and reported in Appendix C.1).”
Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Ma, X., Wang, L., Yang, N., Wei, F., & Lin, J. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657951 · Cited by 540
Self-citationIn-Context Retrieval-Augmented Language Models
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00605 · Cited by 449
Pretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 450
Self-citationLarge Language Models are Effective Text Rankers with Pairwise Ranking Prompting
Qin, Z., Jagerman, R., Hui, K., Zhuang, H., Wu, J., Yan, L. W., Shen, J., Liu, T., Liu, J., Metzler, D., et al. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.97 · Cited by 429
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407
InPars: Unsupervised Dataset Generation for Information Retrieval
Bonifacio, L., Abonizio, H., Fadaee, M., & Nogueira, R. F. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531863 · Cited by 404
“The collection is indexed using pyserini [18] and 1000”
MethodSelf-citationEvaluating Retrieval Quality in Retrieval-Augmented Generation
Salemi, A., & Zamani, H. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657957 · Cited by 155
“For document retrieval during training, we utilize BM25 [22] implemented in Pyserini [18] to retrieve 50 documents to augment the input with them.”
MethodSearching for Best Practices in Retrieval-Augmented Generation
Wang, X., Wang, Z., Gao, X., Zhang, F., Wu, Y., Xu, Z., Shi, T., Wang, Z., Li, S., Qian, Q., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.981 · Cited by 111
Precise Zero-Shot Dense Retrieval without Relevance Labels
Gao, L., Ma, X., Lin, J., & Callan, J. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.99 · Cited by 362
“A variety of meth-ods such as negative mining (Xiong et al., 2021; Qu et al., 2021), distillation (Qu et al., 2021; Lin et al., 2021b; Hofstätter et al., 2021) and task-specific pre-training (Izacard et al., 2021; Gao and Callan, 2021; Lu et al., 2021; Gao and Callan, 2022; Liu and Shao, 2022) have…”
Self-citationPrecise Zero-Shot Dense Retrieval without Relevance Labels
Gao, L., Ma, X., Lin, J., & Callan, J. (2022) · arXiv · doi:10.48550/arxiv.2212.10496 · Cited by 362
“A variety of meth-ods such as negative mining (Xiong et al., 2021; Qu et al., 2021), distillation (Qu et al., 2021; Lin et al., 2021b; Hofstätter et al., 2021) and task-specific pre-training (Izacard et al., 2021; Gao and Callan, 2021; Lu et al., 2021; Gao and Callan, 2022; Liu and Shao, 2022) have…”
Self-citationRankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!Preprint
Pradeep, R., Sharifymoghaddam, S., & Lin, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.02724 · Cited by 348
“For each query, the top 20 candidate passages were retrieved using BM25 with Pyserini (Lin et al., 2021a), which were then ranked by RankGPT 3.5 to create the teacher reorderings that were subsequently distilled into our student model, Rank-Zephyr.”
BackgroundMethodInfluentialSelf-citationBenchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.372 · Cited by 310
Fact-Checking Complex Claims with Program-Guided Reasoning
Pan, L., Wu, X., Lu, X., Luu, A. T., Wang, W. Y., Kan, M., & Nakov, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.386 · Cited by 291
MIRACL : A Multilingual Retrieval Dataset Covering 18 Diverse Languages
Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., & Lin, J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00595 · Cited by 274
Self-citationmMARCO: A Multilingual Version of the MS MARCO Passage Ranking DatasetPreprint
Bonifacio, L., Jeronymo, V., Abonizio, H. Q., Campiotti, I., Fadaee, M., Lotufo, R., & Nogueira, R. F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13897 · Cited by 264
“We use the Pyserini framework to perform this task (Lin et al., 2021).”
MethodSelf-citationA Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models
Zhuang, S., Zhuang, H., Koopman, B., & Zuccon, G. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657813 · Cited by 260
“The original Listwise method relies on the LLM’s next token generation to produce the complete ordered list of document labels at each step of the sliding window process, as illustrated in Figure 1b.”
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval
Lin, S.-C., Yang, J.-H., & Lin, J. (2021) · Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021) · doi:10.18653/v1/2021.repl4nlp-1.17 · Cited by 122
“To characterize end-to-end effectiveness and efficiency, we perform sparse retrieval with the Py-serini toolkit (Lin et al., 2021) and dense retrieval with Faiss (Johnson et al., 2017), but implement the score combination in separate custom code.”
BackgroundMethodSelf-citationQuery2doc: Query Expansion with Large Language Models
Wang, L., Yang, N., & Wei, F. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.585 · Cited by 214
Learning to Retrieve In-Context Examples for Large Language Models
Wang, L., Yang, N., & Wei, F. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2024.eacl-long.105 · Cited by 212
Dense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211
“Besides, Pyserini [82] is a toolkit that is designed to facilitate reproducible research for information retrieval.”
BackgroundOn the Risk of Misinformation Pollution with Large Language Models
Pan, Y., Pan, L., Chen, W., Nakov, P., Kan, M.-Y., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.97 · Cited by 210
How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval
Lin, S.-C., Asai, A., Li, M., Oğuz, B., Lin, J., Mehdad, Y., Yih, W.-T., & Chen, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.423 · Cited by 208
Self-citationGema, A. P., Leang, J. O. J., Hong, G., Devoto, A., Mancino, A. C. M., Saxena, R., He, X., Zhao, Y., Du, X., Madani, M. R. G., et al. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.262 · Cited by 193
Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval
Zhang, X., Ma, X., Shi, P., & Lin, J. (2021) · Proceedings of the 1st Workshop on Multilingual Representation Learning · doi:10.18653/v1/2021.mrl-1.12 · Cited by 51
Self-citationLlama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval
Li, C., Liu, Z., Xiao, S., Shao, Y., & Lian, D. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.191 · Cited by 185
“When the BEIR benchmark was first launched two years ago, none of the dense retrievers (BERT and many others) were able to outperform BM25 despite their competitive performances in the supervised scenarios.”
InfluentialCodeRAG-Bench: Can Retrieval Augment Code Generation?
Wang, Z. Z., Asai, A., Yu, X. V., Xu, F. F., Xie, Y., Neubig, G., & Fried, D. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.176 · Cited by 181
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., & Wei, F. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.125 · Cited by 42
BM25S: Orders of magnitude faster lexical search via eager sparse scoringPreprint
Lü, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.03618 · Cited by 158
“Those Java implementations, usually based on Lucene 3 , are accessible inside Python via the Pyserini reproducibility toolkit (Lin et al., 2021), and through HTTP by using the Elastic-search web client 4 .”
MethodConvGQR: Generative Query Reformulation for Conversational Search
Mo, F., Mao, K., Zhu, Y., Wu, Y., Huang, K., & Nie, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.274 · Cited by 149
Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting
Ye, F., Fang, M., Li, S., & Yılmaz, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.398 · Cited by 147
Reduce, Reuse, Recycle: Green Information Retrieval Research
Scells, H., Zhuang, S., & Zuccon, G. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531766 · Cited by 119
“BM25 We use the standard pyserini [45] For inference, we use the FAISS library [38] to index and retrieve dense vectors. monoBERT We followthe trainingpracticedescribedbyNogueira and Cho [62].”
BackgroundMethodGenerative Relevance Feedback with Large Language Models
Mackie, I., Chatterjee, S., & Dalton, J. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591992 · Cited by 116
“We use naver/splade-cocondenser-ensembledistil checkpoint and Pyserini’s [20] “impact” searcher for max-passage aggregation.”
MethodInfluentialParallel Context Windows for Large Language Models
Ratner, N., Levine, Y., Belinkov, Y., Ram, O., Magar, I., Abend, O., Karpas, E., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.352 · Cited by 111
“…the information extraction tasks ATIS airlines (Hemphill et al., 1990) and MIT movie genre (Liu et al., 2012), which were taken from Zhao et al. (2021), and Natural Questions (Kwiatkowski et al., 2019) which was loaded and incorporated with retrieved documents using Py-serini (Lin et al., 2021).”
MethodRank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning
Zhuang, S., Ma, X., Yao, Z., Wang, S., Koopman, B., Lin, J., & Zuccon, G. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809961 · Cited by 112
“Due to computational constraints, RL training is conducted on approximately 18% of the MSMARCO training set; at this step the model was found to achieve a fair convergence on training validation data.”
Self-citationDelucionQA: Detecting Hallucinations in Domain-specific Question Answering
Sadat, M., Zhou, Z., Lange, L., Araki, J., Gundroo, A., Wang, B., Menon, R. R., Parvez, M. R., & Feng, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.59 · Cited by 98
“In contrast, the dense retrieval function is based on FAISS index (Johnson et al., 2019) with a reproduced ColBERT transformer model (Lin et al., 2021b) for embedding and retrieval score.”
MethodContextualized Query Embeddings for Conversational Search
Lin, S.-C., Yang, J.-H., & Lin, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.77 · Cited by 95
“We perform dense retrieval using Faiss (Johnson et al., 2017) (Faiss-GPU, brute force) and sparse retrieval using Pyserini (Lin et al., 2021a) (BM25, k 1 = 0 .”
BackgroundMethodInfluentialSelf-citationMaking a MIRACL: Multilingual Information Retrieval Across a Continuum of LanguagesPreprint
Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., & Lin, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.09984 · Cited by 93
Self-citationT2Ranking: A Large-scale Chinese Benchmark for Passage Ranking
Xie, X., Dong, Q., Wang, B., Lv, F., Yao, T., Gan, W., Wu, Z., Li, X., Li, H., Liu, Y., et al. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591874 · Cited by 91
“• DPTDR [25] is the first work that employs prompt tuning for dense retrieval.”
BackgroundImproving Passage Retrieval with Zero-Shot Question Generation
Sachan, D., Lewis, M., Joshi, M., Aghajanyan, A., Yih, W.-T., Pineau, J., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.249 · Cited by 90
Retrieval-Augmented Retrieval: Large Language Models are Strong Zero-Shot Retriever
Shen, T., Long, G., Geng, X., Tao, C., Lei, Y., Zhou, T., Blumenstein, M., & Jiang, D. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.943 · Cited by 89
Zhuang, S., Ma, X., Koopman, B., Lin, J., & Zuccon, G. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.250 · Cited by 89
Self-citationHe, Z., Cao, P., Wang, C., Jin, Z., Chen, Y., Xu, J., Li, H., Liu, K., & Zhao, J. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.549 · Cited by 87
“Therefore, we propose corresponding metrics for legal articles and judgement evaluation.”
Query Performance Prediction Using Relevance Judgments Generated by Large Language Models
Meng, C., Arabzadeh, N., Askari, A., Aliannejadi, M., & de Rijke, M. (2025) · ACM Transactions on Information Systems · doi:10.1145/3736402 · Cited by 88
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
Min, S., Gururangan, S., Wallace, E., Shi, W., Hajishirzi, H., Smith, N. A., & Zettlemoyer, L. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2308.04430 · Cited by 89
“We use BM25 from Pyserini (Lin et al., 2021).”
Exploring Listwise Evidence Reasoning with T5 for Fact Verification
Jiang, K., Pradeep, R., & Lin, J. (2021) · doi:10.18653/v1/2021.acl-short.51 · Cited by 85
“In this work, our contributions are focused on the second and third sub-tasks; for document retrieval, we simply augment current best practices with BM25 (Yang et al., 2017; Lin et al., 2021).”
MethodInfluentialSelf-citationAPEER : Automatic Prompt Engineering Enhances Large Language Model Reranking
Jin, C., Peng, H., Zhao, S., Wang, Z. G., Xu, W., Han, L., Zhao, J., Zhong, K.-W., Rajasekaran, S., & Metaxas, D. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3717574 · Cited by 85
“Furthermore, we perform in-depth ablation studies to evaluate the impact of our novel preference optimization, as well as the effects of different training dataset sizes in APEER .”
Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls
Li, H., Mourad, A., Zhuang, S., Koopman, B., & Zuccon, G. (2023) · ACM Transactions on Information Systems · doi:10.1145/3570724 · Cited by 82
“We use the implementations provided in the pyserini toolkit [31].”
MethodCorpus-Steered Query Expansion with Large Language Models
Lei, Y., Cao, Y. H., Zhou, T., Shen, T., & Yates, A. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2024.eacl-short.34 · Cited by 80
Injecting the BM25 Score as Text Improves BERT-Based Re-rankers
Askari, A., Abolghasemi, A., Pasi, G., Kraaij, W., & Verberne, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_5 · Cited by 77
CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search
Mo, F., Ghaddar, A., Mao, K., Rezagholizadeh, M., Chen, B., Liu, Q., & Nie, J. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.135 · Cited by 75
The Information Retrieval Experiment Platform
Fröbe, M., Reimer, J. H., MacAvaney, S., Deckers, N., Reich, S., Bevendorff, J., Stein, B., Hagen, M., & Potthast, M. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591888 · Cited by 74
“We report the effectiveness as nDCG@10 (macro-averaged in case a corpus is associated with multiple tasks) for BM25, ColBERT, TAS-B, all three duoT5 variants, and monoT5 (in its default configuration with its default model) and the best, median, and worst approaches from the groups of 20 lexical, 17 bi-encoder, and 8…”
MethodInfluentialVector Search with OpenAI Embeddings: Lucene Is All You Need
Xian, J., Teofili, T., Pradeep, R., & Lin, J. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635691 · Cited by 76
Self-citationPradeep, R., Liu, Y., Zhang, X., Li, Y., Yates, A., & Lin, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_44 · Cited by 72
“3 We use the Pyserini IR toolkit [26] to generate the TCT-ColBERTv2 runs following the reproduction documentation in Pyserini.”
MethodSelf-citationCurriculum Contrastive Context Denoising for Few-shot Conversational Dense Retrieval
Mao, K., Dou, Z., & Qian, H. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531961 · Cited by 71
“For the first three query reformulation methods, we perform both sparse retrieval and dense retrieval with Pyserini BM25 [18] 3 and ANCE [41], respectively, on the reformulated queries for evaluation.”
Methods3: You Don’t Need That Much Data to Train a Search Agent via RL
Jiang, P., Xu, X., Lin, J., Xiao, J., Wang, Z., Sun, J., & Han, J. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1095 · Cited by 70
Self-citationExploring the Best Practices of Query Expansion with Large Language Models
Zhang, L., Wu, Y., Yang, Q., & Nie, J. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.103 · Cited by 69
“For BM25 searches, we use the Pyserini toolkit (Lin et al., 2021) with default settings.”
MethodAlaofi, M., Thomas, P., Scholer, F., & Sanderson, M. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698431 · Cited by 69
Overview of the TREC 2021 deep learning trackPreprint
Craswell, N., Mitra, B., Yılmaz, E., Campos, D., & Lin, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.08191 · Cited by 69
“In the reranking subtask, participants were provided with an initial ranking of 100 documents, which was retrieved using Pyserini [Lin et al., 2021b].”
Self-citationLin, J., Wang, T., & Qian, K. (2025) · Transactions on machine learning research · doi:10.48550/arxiv.2503.24289 · Cited by 70
“For sparse retrieval, we use Pyserini (Lin et al., 2021) with Lucene’s BM25 implementation.”
Self-citationFaster Learned Sparse Retrieval with Guided Traversal
Mallia, A., Mackenzie, J., Suel, T., & Tonellotto, N. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531774 · Cited by 67
“68 [18].”
Hofstätter, S., Khattab, O., Althammer, S., Sertkan, M., & Hanbury, A. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557367 · Cited by 67
“As baseline, we utilize BM25 as implemented by Pyserini [29].”
MethodSalient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?
Chen, X., Lakhotia, K., Oğuz, B., Gupta, A., Lewis, P., Peshterliev, S., Mehdad, Y., Gupta, S., & Yih, W.-T. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.19 · Cited by 66
Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering
Chuang, Y.-S., Fang, W., Li, S.-W. D., Yih, W.-T., & Glass, J. R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.768 · Cited by 65
Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification
Li, S., Zhao, R., Li, M., Ji, H., Callison-Burch, C., & Han, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.312 · Cited by 64
Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
Pradeep, R., Thakur, N., Sharifymoghaddam, S., Zhang, E., Nguyen, R., Campos, D., Craswell, N., & Lin, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_9 · Cited by 63
Self-citationProvence: efficient and robust context pruning for retrieval-augmented generation
Chirkova, N., Formal, T., Nikoulina, V., & Clinchant, S. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2501.16214 · Cited by 63
LoCal: Logical and Causal Fact-Checking with LLM-Based Multi-Agents
Ma, J., Hu, L., Li, R., & Fu, W. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714748 · Cited by 63
The ROOTS Search Tool: Data Transparency for LLMs
Piktus, A., Akiki, C., Villegas, P., Laurençon, H., Dupont, G., Luccioni, A. S., Jernite, Y., & Rogers, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.29 · Cited by 59
Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query EvaluationPreprint
Mackenzie, J., Trotman, A., & Lin, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.11540 · Cited by 59
“68, based on tuning on a selection of training instances on the MS MARCO passage ranking test collection [19].”
BackgroundSelf-citationSet the Clock: Temporal Alignment of Pretrained Language Models
Zhao, B., Brumbaugh, Z., Wang, Y., Hajishirzi, H., & Smith, N. A. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.892 · Cited by 59
Search Augmented Instruction Learning
Luo, H., Zhang, T., Chuang, Y.-S., Gong, Y., Kim, Y., Wu, X., Meng, H. M. L., & Glass, J. R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.242 · Cited by 57
Scaling Down, LiTting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder ModelsPreprint
Tamber, M. S., Pradeep, R., & Lin, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.16098 · Cited by 57
“In the first stage of training, the same ranked lists as in the RankZephyr and RankVicuna works formed from RankGPT 3.5 orderings of candidate passages retrieved using BM25 with Pyserini (Lin et al., 2021) LiT5-Distill models were trained using up to 20 input passages to maintain a window size of 20, matching previous…”
MethodInfluentialSelf-citationVisconde: Multi-document QA with GPT-3 and Neural Reranking
Pereira, J., Fidalgo, R., Lotufo, R., & Nogueira, R. F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_44 · Cited by 55
Self-citationSalemi, A., & Zamani, H. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657733 · Cited by 54
“We use Pyserini’s [27] implementation of BM25 with default parameters for implementing the first stage that takes the query string 𝑞 and retrieves 100 documents.”
MethodBalog, K., Metzler, D., & Qin, Z. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730348 · Cited by 57
“Specifically, LLM judges should not be used selectively to fill “relevance holes” in existing human judgments [1].”
Flashrag: A modular toolkit for efficient retrieval-augmented generation research
Jin, J., Zhu, Y., Dou, Z., Dong, G., Yang, X., Zhang, C., Zhao, T., Yang, Z., & Wen, J.-R. (2025) · Companion Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3701716.3715313 · Cited by 53
“FlashRAG supports both text-only and multimodal modes, providing researchers with more deployment options.”
What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary
Ram, O., Bezalel, L., Zicher, A., Belinkov, Y., Berant, J., & Globerson, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.140 · Cited by 53
Soft prompt tuning for augmenting dense retrieval with large language models
Peng, Z., Wu, X., Wang, Q., & Fang, Y. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112758 · Cited by 53
Large Language Models are Strong Zero-Shot RetrieverPreprint
Shen, T., Long, G., Geng, X., Tao, C., Zhou, T., & Jiang, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.14233 · Cited by 53
Zhuang, S., & Zuccon, G. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531951 · Cited by 51
Li, M., Lin, S.-C., Oğuz, B., Ghoshal, A., Lin, J., Mehdad, Y., Yih, W.-T., & Chen, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.663 · Cited by 51
Self-citationSalient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?Preprint
Chen, X., Lakhotia, K., Oğuz, B., Gupta, A., Lewis, P., Peshterliev, S., Mehdad, Y., Gupta, S., & Yih, W.-T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06918 · Cited by 51
“We use Pyserini (Lin et al., 2021a) for all sparse models used in this work including BM25 and UniCOIL.”
MethodInfluentialRethinking with Retrieval: Faithful Large Language Model Inference
He, H., Zhang, H., & Roth, D. (2023) · arXiv · doi:10.48550/arxiv.2301.00303 · Cited by 50
“In particular, we use the re-implementation of the sparse retrieval BM25 4 in Karpukhin et al. (2020) from Pyserini (Lin et al., 2021).”
MethodResources for Brewing BEIR: Reproducible Reference Models and Statistical Analyses
Kamalloo, E., Thakur, N., Lassance, C., Ma, X., Yang, J.-H., & Lin, J. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657862 · Cited by 50
“We make the following contributions: • We share with the community reproducible implementations of five popular retrieval models for BEIR in the open-source Pyserini IR toolkit [28].”
MethodInfluentialSelf-citationJin, J., Zhu, Y., Zhou, Y., & Dou, Z. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.42 · Cited by 50
CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering
Wu, Y., Huang, Y., Hu, N., Hua, Y., Qi, G., Chen, J., & Pan, J. Z. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.205 · Cited by 50
Lin, S.-C., Li, M., & Lin, J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00556 · Cited by 50
“We index and evaluate SPLADE-max and uniCOIL using the code provided by Formal et al. (2021a) 4 and Pyserini (Lin et al., 2021a), respectively.”
MethodResultSelf-citationUnderstanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
Zhao, S., Shao, Y., Huang, Y., Song, J., Wan, C., Wan, C., & Ма, Л. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3802824 · Cited by 49
“Given a query 𝑄 and the indexed database 1 2 ,𝐷 𝑛 , the retriever identifies the order of these documents according to a similarity criterion 𝑠𝑖𝑚 .”
HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer RerankingPreprint
Zhang, Y., Long, D., Xu, G., & Xie, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.10569 · Cited by 49
“We implement the BM25 retrieval process by the pyserini toolkit (Lin et al., 2021).”
MethodNearest Neighbor Speculative Decoding for LLM Generation and Attribution
Li, M., Chen, X., Holtzman, A., Chen, B., Lin, J., Yih, W.-T., & Lin, X. (2024) · neural information processing systems · doi:10.52202/079017-2574 · Cited by 48
Self-citationA Dense Representation Framework for Lexical and Semantic Matching
Lin, S.-C., & Lin, J. (2023) · ACM Transactions on Information Systems · doi:10.1145/3582426 · Cited by 47
“For BM25 and DeepImpact, we output term weights for each query and passage using Pyserini [29] and then randomly assign each unique term to a unique vocabulary ID to form a high-dimensional sparse vector.”
MethodInfluentialSelf-citationAsk Optimal Questions: Aligning Large Language Models with Retriever’s Preference in Conversation
Yoon, C., Kim, G., Jeon, B., Kim, S., Jo, Y., & Kang, J. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.328 · Cited by 47
Overview of the TREC 2022 deep learning trackPreprint
Craswell, N., Mitra, B., Yılmaz, E., Campos, D., Lin, J., Voorhees, E. M., & Soboroff, I. M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.10865 · Cited by 48
“In the top-100 reranking subtask, 100 passages per query were provided to participants, which were retrieved using Pyserini [Lin et al., 2021b].”
Self-citationRankLLM: A Python Package for Reranking with LLMs
Sharifymoghaddam, S., Pradeep, R., Slavescu, A., Nguyen, R., Xu, A., Chen, Z., Zhang, Y., Chen, Y., Xian, J., & Lin, J. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730331 · Cited by 46
“For each query, the top 20 passages are retrieved using BM25 with Pyserini [25] and then reordered by a teacher model, GPT 4 .”
Self-citationSTAIR: Learning Sparse Text and Image Representation in Grounded Tokens
Chen, C., Zhang, B., Cao, L., Shen, J., Gunter, T., Jose, A., Toshev, A., Zheng, Y., Shlens, J., Pang, R., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.932 · Cited by 45
DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions
Viswanathan, V., Gao, L., Wu, T., Liu, P., & Neubig, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.573 · Cited by 44
QACheck: A Demonstration System for Question-Guided Multi-Hop Fact-Checking
Pan, L., Lu, X., Kan, M., & Nakov, P. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.23 · Cited by 44
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors
Ivison, H., Smith, N. A., Hajishirzi, H., & Dasigi, P. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.576 · Cited by 44
Metadata
- Authors
- Lin, J., Ma, X., Lin, S.-C., Yang, J.-H., Pradeep, R., & Nogueira, R.
- Affiliations
- University of Waterloo (CA)
- Venue
- Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2021-07-11
- Pages
- 2356-2362
- OpenAlex
- W3180230246
- Cited by
- 360
- References
- 41
- Citations by year
- 2026: 67 · 2025: 72 · 2024: 61 · 2023: 97 · 2022: 44 · 2021: 18
- FWCI
- 45.53
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Waterloo-Huawei Joint Innovation Laboratory
- Canada First Research Excellence Fund
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- reproducibility, sparse retrieval, dense retrieval, hybrid retrieval, BM25, transformer representations, nearest neighbor search, neural ranking, relevance assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.