Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations
Open accessThe DOI matches its Crossref record; 39 of the 41 references it cites were found in a registry or an index.
Abstract
Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations. It aims to provide effective, reproducible, and easy-to-use first-stage retrieval in a multi-stage ranking architecture. Our toolkit is self-contained as a standard Python package and comes with queries, relevance judgments, pre-built indexes, and evaluation scripts for many commonly used IR test collections. We aim to support, out of the box, the entire research lifecycle of efforts aimed at improving ranking with modern neural approaches. In particular, Pyserini supports sparse retrieval (e.g., BM25 scoring using bag-of-words representations), dense retrieval (e.g., nearest-neighbor search on transformer-encoded representations), as well as hybrid retrieval that integrates both approaches. This paper provides an overview of toolkit features and presents empirical results that illustrate its effectiveness on two popular ranking tasks. Around this toolkit, our group has built a culture of reproducibility through shared norms and tools that enable rigorous automated testing.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Interestingly, the classic BM25 model [18] remains competitive, surpassing some semantic models trained on search behavior datasets.”
From Generative Agents Navigating Digital Libraries (2025) · cites this paper as [18]
In the citing paper
“Sparse lexical methods, from classical BM25 [Lin et al., 2021] to lightweight full-text indexes such as SQLite FTS5, complement dense retrieval by matching exact terms.”
From MemX: A Local-First Long-Term Memory System for AI Assistants (2026) · cites this paper as Lin et al., 2021
Citations
Cites
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 1×
“This is particularly the case for work on deep learning today, since the major toolkits (PyTorch [29] and Tensorflow [1]) have both adopted Python as their front-end This work is licensed under a Creative Commons Attribution International 4.”
BackgroundTensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“also provides document expansion baselines using our doc2query method [28]; the latest model uses T5 [33] as described in Nogueira and Lin [26].”
MethodTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jégou, H. (2017) · arXiv · doi:10.48550/arxiv.1702.08734 · Cited by 4,140 · cited 3×
“Row (2a) refers to brute-force scans over the document vectors in Faiss [12], which provides exact nearest-neighbor search.”
BackgroundMethodDense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Malkov, Y. A., & Yashunin, D. A. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889473 · Cited by 1,904 · cited 3×
“To support dense and hybrid retrieval, Pyserini integrates Facebook’s Faiss library for efficient similarity search over dense vectors [12], which in turns integrates the HNSW library [23] to support low-latency querying.”
MethodColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Khattab, O., & Zaharia, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401075 · Cited by 1,301 · cited 8×
“Row (3a) shows the results of combining TCT-ColBERT with BM25 bag-of-words search over the original texts, while row (3b) shows results that combine document expansion using doc2query with the T5 model.”
MethodInfluentialAnserini: Enabling the Use of Lucene for Information Retrieval Research
Yang, P., Fang, H., & Lin, J. (2017) · Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3077136.3080721 · Cited by 319 · cited 10×
“The above result corresponds to an Anserini baseline on the MS MARCO passage leaderboard.”
MethodInfluentialSelf-citationKandasamy, K., Vysyaraju, K. R., Neiswanger, W., Paria, B., Collins, C. R., Schneider, J., Póczos, B., & Xing, E. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.06694 · Cited by 252
Anserini: Reproducible Ranking Baselines Using Lucene
Yang, P., Fang, H., & Lin, J. (2018) · Journal of Data and Information Quality · doi:10.1145/3239571 · Cited by 221 · cited 10×
“The above result corresponds to an Anserini baseline on the MS MARCO passage leaderboard.”
MethodInfluentialSelf-citationApproximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., & Overwijk, A. (2020) · arXiv · doi:10.48550/arxiv.2007.00808 · Cited by 204 · cited 2×
“Row (4b) shows results from ANCE [37], which appears to be more effective than TCT-ColBERT, although the comparison isn’t quite fair since our models were not trained on MS MARCO document data.”
MethodA cascade ranking model for efficient ranked retrieval
Wang, L., Lin, J., & Metzler, D. (2011) · Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval · doi:10.1145/2009916.2009934 · Cited by 200 · cited 1×
“In particular, the importance of stable first-stage retrieval within a multi-stage ranking architecture [5, 25, 34] has become even more important, as it provides the foundation for increasingly-complex modern approaches that leverage hybrid techniques.”
BackgroundSelf-citationDocument Expansion by Query Prediction
Nogueira, R. F., Yang, W., Lin, J., & Cho, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08375 · Cited by 191 · cited 1×
“also provides document expansion baselines using our doc2query method [28]; the latest model uses T5 [33] as described in Nogueira and Lin [26].”
MethodSelf-citationMulti-Stage Document Ranking with BERT
Nogueira, R. F., Yang, W., Cho, K., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14424 · Cited by 186 · cited 2×
“957 Pyserini: multi-stage pipelines (4d) monoBERT [27] 0.”
MethodSelf-citationDeclarative Experimentation in Information Retrieval using PyTerrier
Macdonald, C., & Tonellotto, N. (2020) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3409256.3409829 · Cited by 170 · cited 1×
“This stands in contrast to, for example, PyTerrier, which aims to provide end-to-end search capabilities via a domain-specific language that allows advanced retrieval pipelines to be expressed in a declarative manner [22].”
ResultFrom Puppy to Maturity: Experiences in Developing Terrier.
MacDonald, C. M., McCreadie, R., Santos, R. L. T., & Ounis, I. (2012) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · Cited by 143
High accuracy retrieval with multiple nested ranker
Matveeva, I., Burges, C., Burkard, T., Laucius, A., & Wong, L. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148246 · Cited by 128 · cited 1×
“In particular, the importance of stable first-stage retrieval within a multi-stage ranking architecture [5, 25, 34] has become even more important, as it provides the foundation for increasingly-complex modern approaches that leverage hybrid techniques.”
BackgroundToward Reproducible Baselines: The Open-Source IR Reproducibility Challenge
Lin, J., Crane, M., Trotman, A., Callan, J., Chattopadhyaya, I., Foley, J., Ingersoll, G., Macdonald, C., & Vigna, S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-30671-1_30 · Cited by 114 · cited 2×
“Pyserini began as the Python interface to Anserini [38, 39], which our group has been developing for several years, with its roots in a community-wide reproducibility exercise dating back to 2015 [15].”
BackgroundMethodSelf-citationEsteva, A., Kale, A., Paulus, R., Hashimoto, K., Yin, W., Radev, D., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00437-0 · Cited by 109
Pretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06467 · Cited by 101 · cited 1×
“The advent of pretrained transformers has led to many exciting recent developments in information retrieval [16].”
BackgroundSelf-citationEffectiveness/efficiency tradeoffs for candidate generation in multi-stage retrieval architectures
Asadi, N., & Lin, J. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484132 · Cited by 96 · cited 1×
“In particular, the importance of stable first-stage retrieval within a multi-stage ranking architecture [5, 25, 34] has become even more important, as it provides the foundation for increasingly-complex modern approaches that leverage hybrid techniques.”
BackgroundSelf-citationA Replication Study of Dense Passage RetrieverPreprint
Ma, X., Sun, K., Pradeep, R., & Lin, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05740 · Cited by 72 · cited 1×
“This is a standard approach and Pyserini adopts the specific implementation in TCT-ColBERT [17], but similar techniques are used elsewhere as well [13, 18].”
MethodSelf-citationArguello, J., Crane, M., Díaz, F., Lin, J., & Trotman, A. (2016) · ACM SIGIR Forum · doi:10.1145/2888422.2888439 · Cited by 68 · cited 1×
“Our group’s efforts to promote and support reproducible research dates back to at least 2015 [4, 15], and the landscape has changed quite a bit since then.”
BackgroundSelf-citationOpenNIR: A Complete Neural Ad-Hoc Ranking Pipeline
MacAvaney, S. (2020) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3336191.3371864 · Cited by 51 · cited 1×
“Today, there is a vibrant ecosystem of search engines built by academic research groups such as Terrier [21] and PISA [24], common infrastructure such as ir_datasets [20], and neural ranking libraries as Capreolus [40, 41], OpenNIR [19], and OpenMatch [36].”
BackgroundZhang, E., Gupta, N., Tang, R., Han, X., Pradeep, R., Lu, K., Zhang, Y., Nogueira, R., Cho, K., Fang, H., et al. (2020) · Proceedings of the First Workshop on Scholarly Document Processing · doi:10.18653/v1/2020.sdp-1.5 · Cited by 51
Self-citationDistilling Dense Representations for Ranking using Tightly-Coupled Teachers
Lin, S.-C., Yang, J.-H., & Lin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11386 · Cited by 50 · cited 4×
“For dense retrieval, results using TCT-ColBERT [17] are shown in rows (2) using different indexes.”
MethodInfluentialSelf-citationEsteva, A., Kale, A., Paulus, R., Hashimoto, K., Yin, W., Radev, D., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09595 · Cited by 48 · cited 1×
“In the first use case, a researcher can reproduce a baseline (first-stage retrieval) run with a simple invocation, take the output run file (which is just plain text) to serve as input for downstream reranking, or as part of ensembles [7, 8].”
MethodVera: Prediction Techniques for Reducing Harmful Misinformation in Consumer Health Search
Pradeep, R., Ma, X., Nogueira, R. F., & Lin, J. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463120 · Cited by 37 · cited 1×
“Beyond search, Pyserini has also been used for other applications such as question answering [43], fact verification [30], and combating misinformation [31].”
MethodSelf-citationPISA: Performant indexes and search for academia
Mallia, A., Siedlaczek, M., Mackenzie, J., & Suel, T. (2019) · RMIT Research Repository (RMIT University Library) · doi:10.1000/604faketemporaryforimport · Cited by 35
Capreolus: A Toolkit for End-to-End Neural Ad Hoc Retrieval
Yates, A., Arora, S., Zhang, X., Yang, W., Jose, K. M., & Lin, J. (2020) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3336191.3371868 · Cited by 29 · cited 1×
“Today, there is a vibrant ecosystem of search engines built by academic research groups such as Terrier [21] and PISA [24], common infrastructure such as ir_datasets [20], and neural ranking libraries as Capreolus [40, 41], OpenNIR [19], and OpenMatch [36].”
BackgroundSelf-citationRRF102: Meeting the TREC-COVID Challenge with a 100+ Runs EnsemblePreprint
Bendersky, M., Zhuang, H., Ma, J., Han, S., Hall, K., & McDonald, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00200 · Cited by 24 · cited 1×
“In the first use case, a researcher can reproduce a baseline (first-stage retrieval) run with a simple invocation, take the output run file (which is just plain text) to serve as input for downstream reranking, or as part of ensembles [7, 8].”
MethodGrand, A., Muir, R., Ferenczi, J., & Lin, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45442-5_3 · Cited by 19
Self-citationFlexible IR Pipelines with Capreolus
Yates, A., Jose, K. M., Zhang, X., & Lin, J. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412780 · Cited by 13 · cited 1×
“Today, there is a vibrant ecosystem of search engines built by academic research groups such as Terrier [21] and PISA [24], common infrastructure such as ir_datasets [20], and neural ranking libraries as Capreolus [40, 41], OpenNIR [19], and OpenMatch [36].”
BackgroundSelf-citationA Lightweight Environment for Learning Experimental IR Research Practices
Yilmaz, Z. A., Clarke, C. L. A., & Lin, J. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401395 · Cited by 11 · cited 2×
“[2]), and occasionally subtle issues confuse experienced researchers as well.”
BackgroundSelf-citation
Research works printed but not linked: 6
- Sebastian Hofstatter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. arXiv:2010.02666 (2020). · arXiv
- Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2021. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. arXiv:2010.02666 (2021). · arXiv
- Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Goharian. 2021. Simplified Data Wrangling with textttir_datasets. arXiv:2103.02280 (2021). · arXiv
- Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021 c. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. arXiv:2101.05667 (2021). · arXiv
- Chenyan Xiong, Zhenghao Liu, Si Sun, Zhuyun Dai, Kaitao Zhang, Shi Yu, Zhiyuan Liu, Hoifung Poon, Jianfeng Gao, and Paul Bennett. 2020 a. CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search. arXiv:2011.01580 (2020). · arXiv
- Yongze Yu, Jussi Karlgren, Hamed Bonab, Ann Clifton, Md Iftekhar Tanveer, and Rosie Jones. 2020. Spotify at the TREC 2020 Podcasts Track: Segment Retrieval. In Proceedings of the Twenty-Ninth Text REtrieval Conference (TREC 2020).
Cited by
Autonomous chemical research with large language models
Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023) · Nature · doi:10.1038/s41586-023-06792-0 · Cited by 1,050
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., & Liu, Z. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.137 · Cited by 757
“Following the official benchmark, we evaluate our method using Pyserini (Lin et al., 2021), and use nDCG@10 as the primary evaluation metric (Recall@100 is also measured and reported in Appendix C.1).”
Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Ma, X., Wang, L., Yang, N., Wei, F., & Lin, J. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657951 · Cited by 539
Self-citationIn-Context Retrieval-Augmented Language Models
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00605 · Cited by 449
Pretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 450
Self-citation
Metadata
- Authors
- Lin, J., Ma, X., Lin, S.-C., Yang, J.-H., Pradeep, R., & Nogueira, R.
- Affiliations
- University of Waterloo (CA)
- Venue
- Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2021-07-11
- Pages
- 2356-2362
- OpenAlex
- W3180230246
- Cited by
- 360
- References
- 41
- Citations by year
- 2026: 67 · 2025: 72 · 2024: 61 · 2023: 97 · 2022: 44 · 2021: 18
- FWCI
- 45.53
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Waterloo-Huawei Joint Innovation Laboratory
- Canada First Research Excellence Fund
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- reproducibility, sparse retrieval, dense retrieval, hybrid retrieval, BM25, transformer representations, nearest neighbor search, neural ranking, relevance assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.