Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations
Open accessThe DOI matches its Crossref record; 39 of the 41 references it cites were found in a registry or an index.
Abstract
Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations. It aims to provide effective, reproducible, and easy-to-use first-stage retrieval in a multi-stage ranking architecture. Our toolkit is self-contained as a standard Python package and comes with queries, relevance judgments, pre-built indexes, and evaluation scripts for many commonly used IR test collections. We aim to support, out of the box, the entire research lifecycle of efforts aimed at improving ranking with modern neural approaches. In particular, Pyserini supports sparse retrieval (e.g., BM25 scoring using bag-of-words representations), dense retrieval (e.g., nearest-neighbor search on transformer-encoded representations), as well as hybrid retrieval that integrates both approaches. This paper provides an overview of toolkit features and presents empirical results that illustrate its effectiveness on two popular ranking tasks. Around this toolkit, our group has built a culture of reproducibility through shared norms and tools that enable rigorous automated testing.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Interestingly, the classic BM25 model [18] remains competitive, surpassing some semantic models trained on search behavior datasets.”
From Generative Agents Navigating Digital Libraries (2025) · cites this paper as [18]
In the citing paper
“Sparse lexical methods, from classical BM25 [Lin et al., 2021] to lightweight full-text indexes such as SQLite FTS5, complement dense retrieval by matching exact terms.”
From MemX: A Local-First Long-Term Memory System for AI Assistants (2026) · cites this paper as Lin et al., 2021
Citations
Cites
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 1×
“This is particularly the case for work on deep learning today, since the major toolkits (PyTorch [29] and Tensorflow [1]) have both adopted Python as their front-end This work is licensed under a Creative Commons Attribution International 4.”
BackgroundTensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“also provides document expansion baselines using our doc2query method [28]; the latest model uses T5 [33] as described in Nogueira and Lin [26].”
MethodTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jégou, H. (2017) · arXiv · doi:10.48550/arxiv.1702.08734 · Cited by 4,140 · cited 3×
“Row (2a) refers to brute-force scans over the document vectors in Faiss [12], which provides exact nearest-neighbor search.”
BackgroundMethodDense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Malkov, Y. A., & Yashunin, D. A. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889473 · Cited by 1,904 · cited 3×
“To support dense and hybrid retrieval, Pyserini integrates Facebook’s Faiss library for efficient similarity search over dense vectors [12], which in turns integrates the HNSW library [23] to support low-latency querying.”
MethodColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Khattab, O., & Zaharia, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401075 · Cited by 1,301 · cited 8×
“Row (3a) shows the results of combining TCT-ColBERT with BM25 bag-of-words search over the original texts, while row (3b) shows results that combine document expansion using doc2query with the T5 model.”
MethodInfluential
Research works printed but not linked: 6
- Sebastian Hofstatter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. arXiv:2010.02666 (2020). · arXiv
- Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2021. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. arXiv:2010.02666 (2021). · arXiv
- Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Goharian. 2021. Simplified Data Wrangling with textttir_datasets. arXiv:2103.02280 (2021). · arXiv
- Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021 c. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. arXiv:2101.05667 (2021). · arXiv
- Chenyan Xiong, Zhenghao Liu, Si Sun, Zhuyun Dai, Kaitao Zhang, Shi Yu, Zhiyuan Liu, Hoifung Poon, Jianfeng Gao, and Paul Bennett. 2020 a. CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search. arXiv:2011.01580 (2020). · arXiv
- Yongze Yu, Jussi Karlgren, Hamed Bonab, Ann Clifton, Md Iftekhar Tanveer, and Rosie Jones. 2020. Spotify at the TREC 2020 Podcasts Track: Segment Retrieval. In Proceedings of the Twenty-Ninth Text REtrieval Conference (TREC 2020).
Cited by
Items 601–700 of 809
Alaofi, M., Thomas, P., Scholer, F., & Sanderson, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.17969 · Cited by 1
“Retrieval was conducted using Pyterrier [21], except for Doc2Query for which Pyserini [19] was used over a pre-built augmented corpus with doc2query-T5 expansions.”
ERU-KG: Efficient Reference-aligned Unsupervised Keyphrase GenerationPreprint
Do, L. T., Bodke, A., Akash, P. S., & Chang, K. C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.24219 · Cited by 2
“We build our retrievers using Pyserini (Lin et al., 2021).”
HyReC: Exploring Hybrid-based Retriever for ChinesePreprint
Wang, Z., Shenpeng, Z., Shenglan, W., Zhao, M., & Li, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.21913 · Cited by 1
“Adhering to the official benchmark protocols, we evaluate our method using Pyserini (Lin et al., 2021) and utilize nDCG @10 as the primary evaluation metric.”
On the Merits of LLM-Based Corpus EnrichmentPreprint
Zur, G., Mordo, T., Tennenholtz, M., & Kurland, O. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.06015 · Cited by 1
“The MS-MARCO V2 passages (documents) [6] were indexed (Porter stemmed) using Pyserini [41], which was then used both for collecting corpus statistics and for computing ranking effec-tiveness measures.”
JIR-Arena: The First Benchmark Dataset for Just-in-time Information RecommendationPreprint
Yang, K., Ros, K., Kumar, S. K. S., & Zhai, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.13550 · Cited by 1
“V 2-HNP - MSMARCO [Lin et al., 2021b] as the dense indexing model.”
Edinburgh Dataset Analytics Working Group (2024) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.57967/hf/2507 · Cited by 1
Neuro-symbolic fact verification
Aly, R. (2024) · Apollo (University of Cambridge) · doi:10.17863/cam.118214 · Cited by 1
Retrieval-Augmented Generative AI Chatbot
Rodrigues, R. M. O. (2024) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · Cited by 1
MV-RAG: Retrieval Augmented Multiview Diffusion
Dayani, Y., Benishu, O., & Benaim, S. (2025) · Transactions on machine learning research · doi:10.48550/arxiv.2508.16577 · Cited by 1
“We additionally employ Pyserini [59, 43, 27] for text-based retrieval using a Bag-of-Words approach powered by the BM25 ranking function.”
Efficient and Effective Methods for Conversational Search and Summarization
Rocchietti, G. (2025) · Electronic Theses and Dissertations Repository (University of Pisa) · Cited by 1
ICR: Iterative Clarification and Rewriting for Conversational SearchPreprint
Cao, Z., Li, P., & Zhu, Q. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.05100 · Cited by 1
“For the retrieval system, we use Faiss (Johnson et al., 2019) for dense retrieval and Pyserini (Lin et al., 2021) for sparse retrieval.”
Revisiting Feedback Models for HyDEPreprint
Jedidi, N., & Lin, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.19349 · Cited by 1
“All retrieval experiments were conducted using the BM25 implementation from Pyserini [13] with default parameters.”
Self-citationInferential Question AnsweringPreprint
Mozafari, J., Zamani, H., Zuccon, G., & Jatowt, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.01239 · Cited by 1
Lawrie, D., Mayfield, J., Yang, E., Yates, A., MacAvaney, S., Pradeep, R., Miller, S., McNamee, P., & Soldaini, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.05334 · Cited by 1
“Patapsco is a framework built on top of Pyserini (Lin et al., 2021) specifically for experiments in many languages.”
Self-citationSTCALIR: Semi-Synthetic Test Collection for Algerian Legal Information RetrievalPreprint
Hatem, M. A., Batata, S., Mammasse, A., & Azouaou, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.00731 · Cited by 1
SelRoute: Query-Type-Aware Routing for Long-Term Conversational Memory RetrievalPreprint
McKee, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.02431 · Cited by 1
“The complementarity of lexical and semantic retrieval is well-established in information retrieval (Lin et al., 2021).”
Proactive models for open-domain conversational search
Erbacher, P. (2024) · theses.fr (ABES) · doi:10.70675/f7d68861zc694z49f1zb130z675cbf38ed1c · Cited by 1
A Unified Model and Document Representation for On-Device Retrieval-Augmented GenerationPreprint
Killingback, J., Meshi, O., Li, H., Zamani, H., & Karimzadehgan, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.14403 · Cited by 1
“For BM25 we use Pyserini (Lin et al., 2021) with default arguments.”
Diversed Model Discovery via Structured Table DiscoveryPreprint
Dong, Z., & Miller, R. J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.22766 · Cited by 1
“We also support sparse retrieval with Pyserini [36] and a hybrid variant that retrieves an expanded sparse candidate pool before dense reranking.”
TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMsPreprint
Di, Y., Huang, X., & Zhang, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.24079 · Cited by 1
“Following prior work, we use a sparse BM25 index implemented with Pyserini [28] and native Lucene analyzers 1 to score lexical relevance between paired tasks, and classify a task pair as contaminated when the BM25 score exceeds a decision threshold.”
More Efficient LLM Reranking with Whole-Pool, Setwise, Long-Context Language ModelsPreprint
Li, H., Yu, C., Koopman, B., & Zuccon, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.01782 · Cited by 1
“BM25 top-N initial candidate pools are taken from Py-serini [8].”
Retrieval-Oriented Code Representations in Agentic Bug LocalizationPreprint
Caumartin, G., Tse-Hsun, Chen, & Costa, D. E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.11046 · Cited by 1
“We use Pyserini’s implementation of BM25 (Robertson and Zaragoza, 2009; Lin et al., 2021) with default values for the parameters k 1 = 0 .”
Turenne, N., Sklab, Y., Chenin, E., & Zucker, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.14587 · Cited by 1
Query Expansion Should Be Coordinated: Dense Expands, Sparse AnchorsPreprint
Zhang, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.15851 · Cited by 1
“4 (Robertson and Zaragoza, 2009; Lin et al., 2021).”
Question's Gambit: The First Move Matters in Agentic Deep SearchPreprint
Rad, R. H., Bigdeli, A., Arabzadeh, N., Ebrahimi, S., Clarke, C. L. A., Fung, B. C. M., & Bagheri, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.14412 · Cited by 1
“In our proposed method, all first-stage retrieval uses BM25 implemented with Pyserini (Lin et al., 2021), with the same index shared by Question’s Gambit and the agent’s in-loop searches.”
System One models Are Not Better RerankersPreprint
Dinzinger, M., Dastidar, K. G., & Granitzer, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.32318 · Cited by 1
“Every system reranks the BM25 top-100 of Pyserini [16]; these are the candidates of S1Rank [27] and of earlier comparisons [1, 24, 31], whose published numbers we use where we did not rerun a system.”
MemX: A Local-First Long-Term Memory System for AI Assistants
Sun, L. (2026) · arXiv · doi:10.48550/arxiv.2603.16171
“Sparse lexical methods, from classical BM25 [Lin et al., 2021] to lightweight full-text indexes such as SQLite FTS5, complement dense retrieval by matching exact terms.”
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
Thakur, N., Lin, J., Havens, S., Carbin, M., Khattab, O., & Drozdov, A. (2025) · arXiv · doi:10.48550/arxiv.2504.13128
“We utilize the default implementation available in Pyserini [44].”
Self-citationCoverageBench: Evaluating Information Coverage across Tasks and Domains
Samuel, S., Yates, A., Lawrie, D., Soboroff, I., Adriaanse, T., Van Durme, B., & Yang, E. (2026) · Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3805712.3808625
Sparse and Dense Approaches for the Full-rank Retrieval of Responses for DialoguesPreprint
Penha, G., & Hauff, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.10558
A Common Framework for Exploring Document-at-a-Time and Score-at-a-Time Retrieval Methods
Trotman, A., Mackenzie, J., Parameswaran, P., & Lin, J. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531657
“Pyserini [16] began as a lightweight Python wrapper to Anserini for the Python-centric world of deep learning (as Anserini, like Lucene, was implemented in Java).”
MethodInfluentialSelf-citationDKBERT: An Efficient and Effective Ranking Model via Weight Sharing Decoupling and Kernel Matching
Li, X., & Li, D. (2022) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd55127.2022.9820107
MICO: Selective Search with Mutual Information Co-trainingPreprint
Wang, Z., Zhang, X., Yun, H., Teo, C. H., & Chilimbi, T. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.04378
“In the IR community, whether dense or sparse retrieval is better is still a prolonged debate (Lin, 2021; Luan et al., 2021), and recent studies have shown the improved performance of interpolating both (Li et al.”
BackgroundTan, S., & Paperno, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08946
“To validate what model can yield more satisfactory results with real-world data, we consider both sparse (BM25 in Lin et al. (2021)) and dense (DPR in Karpukhin et al. (2020)) retrievers in our retrieval task.”
MethodEnhancing BERT-based Passage Retriever with Word Recovery Method
Chen, H. Y., Qin, B., Yang, Y., & Zhang, J. (2022) · International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI) · doi:10.1109/mlbdbi58171.2022.00023
LeCAR: Leveraging Context for Enhanced Automotive Specification Retrieval
Wu, K.-W., Hsu, T.-H., Huang, Y.-H., Chen, Y.-S., Wang, H.-L., Hsieh, B.-J., & Hsu, C.-H. (2023) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri58017.2023.00038
“Examples of such methods include ColBERT [15], which employs efficient late interaction techniques for training and inference, and Dense Passage Retrieval (DPR) [16] [17], which uses a pair of BERT-based encoders to generate embeddings for queries and documents.”
MethodLossy Compression Options for Dense Index Retention
Mackenzie, J., & Moffat, A. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625316
“All three are publicly available as part of the Anserini/Pyserini open-source toolkit [11, 25].”
BackgroundBuilding Retrieval Systems for the ClueWeb22-B CorpusPreprint
Mehrotra, H., Callan, J., & Fan, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.04357
“Search was also implemented using the Pyserini wrapper for Faiss.”
MethodInfluentialKnowFIRES: A Knowledge-Graph Framework for Interpreting Retrieved Entities from Search
Arabzadeh, N., Golzadeh, K., Risi, C., Clarke, C. L. A., & Zhao, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56069-9_15
“Entity retrievers were implemented in Python 3.8 using the pyserini library [22].”
MethodInfluentialZong, Y., & Piwowarski, B. (2023) · HAL
Augmentation de jeux de données RI pour la recherche conversationnelle à initiative mixte
Erbacher, P., Preux, P., Nie, J., & Soulier, L. (2023) · HAL
Lesavourey, M., & Hubert, G. (2024) · HAL
PqE: Zero-Shot Document Expansion for Dense Retrieval with Large Language Models
Liu, J., Zou, D., Chai, N., Yang, Y., Wang, H., & Song, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9431-7_8
Zero-Shot Dense Retrieval Based on Query Expansion
Wu, Y., Chen, P., Ding, Z., & Yan, A. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1148-5_12
Multi-Document Question Answering with Lightweight Embeddings-based Document Reranker
Aitymbetov, N., Lee, M.-H., & Tu, N. A. (2024) · International Conference on Advanced Technologies for Communications · doi:10.1109/atc63255.2024.10908118
“First, the component retrieves the initial set of documents (k=500 documents for each decomposed question) using the BM25 sparse retriever implemented in the Py-serini package [8].”
Erect: Evidence Refinement Enhanced Complex Claim Verification With Large Language Models
Liu, X., Liu, D., & Wu, P. (2024) · International Conference on Sustainable Technology and Engineering (i-COSTE) · doi:10.1109/i-coste63786.2024.11024841
“For evidence retrieval setting, We use BM25 [14] implemented with the Pyserini toolkit [15] as the information retriever for both ERECT and the baselines, and adopt the Wikipedia corpus as the knowledge source.”
ReSHQ: Re-ranking Based on the Similarity Between Hypothetical Query and Query
Fukuoka, K., & Miyamori, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97141-9_32
ERU-KG: Efficient Reference-aligned Unsupervised Keyphrase Generation
Thanh, L., Bodke, A., Akash, P. S., & Chang, K. C. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1398
Mitigating Gender Bias in Job Ranking Systems Using Job Advertisement Neutrality
Kumar, D., Masoudian, S., Melchiorre, A. B., & Schedl, M. (2025) · Workshop on NLP for Positive Impact (NLP4PI) · doi:10.18653/v1/2025.nlp4pi-1.23
“We use BM25 (Lin et al., 2021) as our initial ranker and CrossEncoder as our final ranker for both training and evaluation.”
A Case Study of Balanced Query Recommendation on WikipediaPreprint
Mishra, H., & Soundarajan, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.20399
“Each original query and its candidate query are searched against the wiki corpus using the Pyserini framework implementation of BM25 [10] algorithm.”
Exploring Relations and Graphs for Information Retrieval
Kamphuis, C. (2025) · Radboud University Press eBooks · doi:10.54195/9789465151298
Ferrara, A., Pomo, C., & Tonellotto, N. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97207-2_38
A Generative Framework for Personalized Sticker Retrieval
Zhou, C., Zhang, R., Guo, J., Liu, Y., Zhang, F., Luo, G., & Cheng, X. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.753
TreeRare: Syntax Tree-Guided Retrieval and Reasoning for Knowledge-Intensive Question Answering
Zhang, B., Liu, Z., & He, H. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.947
“We use BM25 via Py-serini (Lin et al., 2021) as the retrieval backend.”
Bollikonda, M. (2025) · Preprints.org · doi:10.20944/preprints202511.0008.v1
Kovalev, V., & Kumar, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.00360
“BM25 [5]+mpnet dense hybrid; rerank top-50 with MiniLM-L6 [3] over ASR only.”
CRINN: Contrastive Reinforcement Learning for Approximate Nearest Neighbor SearchPreprint
Li, X., Wang, A. Z., Wang, A. Z., Shum, C., & Li, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.02091
“Approximate nearest-neighbor search (ANNS) [20, 21, 6, 7, 19, 12, 5] aims at finding data points that are closest to a query point in high-dimensional spaces while trading off a small amount of accuracy for significant speedup over exact search methods.”
Liu, L., & Zhang, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.04762
“Retrieval experiments are conducted using the Pyserini toolkit [16].”
Semantic Information Retrieval using Transformer-based Embeddings and Vector Similarity Search
Venkat Nivas Reddy, K., Jomy, J., Rose, S., Vasan, K., Sindhu, A., & Suresh, A. (2025) · International Conference on Innovative Data Communication Technologies and Application (ICIDCA) · doi:10.1109/icidca66325.2025.11280585
“The BEIR [22] benchmark provides zero-shot evaluation capabilities for retrieval systems while the open-source Pyserini [18] toolkit helps researchers reproduce dense and sparse retrieval methods and facilitates their wider adoption.”
Argument Retrieval for Controversial Questions
Moreira, B. C. (2022) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · doi:10.34626/hrd3-pq34
Efficient and Explainable Neural Ranking
Leonhardt, L. J. (2023) · Institutional Repository of Leibniz Universität Hannover (Leibniz Universität Hannover) · doi:10.15488/15769 · Cited by 1
Optimizing the Cost-Effectiveness Tradeoff in Neural Ranking
Hofstätter, S. (2023) · reposiTUm (TU Wien) · doi:10.34726/hss.2023.109581
Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query ExpansionPreprint
Yoon, Y., Jung, J., Yoon, S., & Park, K. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.14175
“Moreover, with a few exceptions, performance on unmatched claims was lower than that of the respective base-line methods without query expansion—BM25 for Query2doc and Contriever for HyDE.”
InfluentialExploring language models and question answering in biomedical and Arabic domains
Alrowili, S. (2024) · Library, Museums and Press - UDSpace (University of Delaware) · doi:10.58088/s63n-qp58
Designing and Diagnosing Models for Conversational Search and Recommendation
Penha, G. (2023) · Research Repository (Delft University of Technology) · doi:10.4233/uuid:acdfb704-6310-4b28-b884-4bd3e78b3f84
Pruning Optimization for Efficient Top-k Document Retrieval with Learned Sparse Representations
Qiao, Y. (2024) · eScholarship (California Digital Library)
Nguyen, H. D. (2023) · Institutional Repositories DataBase (IRDB)
言語モデルを用いた長文文書情報検索とクエリ書き換えの影響について
生方, 雄., & 133132 (2023) · Institutional Repositories DataBase (IRDB)
Efficient Inference in Open Retrieval Question Answering Systems
Rashid, M. S. (2024) · eScholarship (California Digital Library)
Identifying Gaps in Climate Impacts Research and Addressing them through a Software-based Workflow
Freedman, H. (2025) · eScholarship (California Digital Library)
Improving User Sparse Query Interpretation Through Pseudo-Relevance Retrieval Methods
Pei, Q. (2024) · YorkSpace (York University)
Understanding comparative questions and retrieving argumentative answers
Bondarenko, A. (2023) · Common Library Network (Der Gemeinsame Bibliotheksverbund)
Resource-lean transfer methods for cross-lingual information retrieval
Litschko, R. (2024) · MADOC (University of Mannheim)
Efficient Inference of Transformers in Natural Language Processing: Early Exiting and Beyond
Xin, J. (2023) · UWSpace (University of Waterloo)
Information Access Using Neural Networks For Diverse Domains And Sources
Xie, Y. (2023) · UWSpace (University of Waterloo)
An In-Depth Exploration of the High-Quality Entity Linking for Information Retrieval: MMEAD
Lin, L. (2023) · UWSpace (University of Waterloo)
Towards Generalized Open Domain Question Answering Systems
Liu, L. (2024) · UCL Discovery (University College London)
Semantic Caching for Mitigating LLM Inference Latency in Real-Time Query Understanding
Alam, P. K. (2025) · European Modern Studies Journal · doi:10.59573/emsj.9(5).2025.12
Tang, A., Soulier, L., & Guigue, V. (2025) · HAL
Minnu Helen, J. (2024) · University of Malaya Students Repository
Петровна, Б. И. (2025) · Vestnik NSUEM · doi:10.34020/2073-6495-2025-4-142-157
LLM-based stemming for improved Gujarati information retrieval
Kaur, J., & Patel, S. (2025) · Aslib Journal of Information Management · doi:10.1108/ajim-04-2025-0241
Integrating pre-trained language models into novel query expansion pipelines
Mackie, I. J. (2025) · Enlighten: Theses (The University of Glasgow) · doi:10.5525/gla.thesis.85661
Compositional Reasoning over Structured and Unstructured Data Using Hybrid Indexing Frameworks
Zhou, Q. (2025) · Multidisciplinary Research in Computing Information Systems · doi:10.71465/mrcis159
The Chairs for Databases and Information Systems at the Friedrich-Schiller-Universität Jena
Ahmed, W., Mustafa, T. A., Bachinger, S. T., Fröbe, M., Hagen, M., König‐Ries, B., Mauch, M., Reimer, J. H., Naik, D. S., Ostrowski, A., et al. (2026) · Datenbank-Spektrum · doi:10.1007/s13222-026-00534-y
“The library integrates with retrieval frameworks such as PyTerrier (Mac-donald et al. 2021), Pyserini (Lin et al. 2021), and ir explain (Saha et al. 2024).”
Doostmohammadi, E. (2026) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789181184440
Reference Recommendation for Large Language Models-Generated Text Using Deep Textual Representations
Bashir, S. (2026) · Romanian Journal of Information Science and Technology · doi:10.59277/romjist.2026.1.01
“The performance of the proposed approach was compared with TFIDF, BM25 [33], Citeo-matic [34], and SubRef [11].”
Truong, A. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402566
“Hybrid retrieval combining sparse and dense methods has shown promise in domain-specific applications [21], [22], yet the effectiveness of lexical matching for temporal queries remains an empirical question.”
Structured prediction for compute efficient and high accuracy NLP
Shavarani, H. S. (n.d.) · Summit (Simon Fraser University)
Li, X., Wang, B., Yang, X., & Luo, M. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i18.38537
Petrosyan, G. S., & Sahakyan, R. R. (2026) · P R O C E E D I N G S OF NATIONAL POLYTECHNIC UNIVERSITY OF ARMENIA INFORMATION TECHNOLOGIES ELECTRONICS RADIO ENGINEERING · doi:10.53297/18293336-2025.2-78
Li, Y., Krishnan, R., & Padman, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.21460
“This hybrid approach has been shown to outperform either retrieval modality alone on biomedical text [12].”
事前学習済み言語モデルを用いた検索モデルに対する教師なしドメイン適応
飯田, 大., & Iida, H. (2024) · Institutional Repositories DataBase (IRDB)
Susanto, B., Ferdiana, R., & Adji, T. B. (2026) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic68212.2026.11454352
“The combination of sparse and dense vector models (hybrid approach) in search systems is more effective than using either sparse or vector models individually [13].”
Li, D., Liang, Y., Qian, S., Sun, Y., Wu, H., Jia, W., Han, W., & Fu, Y. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114705
FreeRanker: A Latent Planning Framework for Zero-Shot Re-ranking
Thai, H. P., Pham, L. P. C., Le, T. A. N., & Le, N. H. T. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-21625-0_15
Giamphy, E. (2024) · theses.fr (ABES) · doi:10.70675/4e3c2514z6e1cz4242z9a97zde7f07c9a4bf
Multi-Faceted Self-Consistent Preference Alignment for Query Rewriting in Conversational SearchPreprint
Cao, Z., Li, P., & Zhu, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.06771
“We use Faiss (Johnson et al., 2019) and Pyserini (Lin et al., 2021) for dense retrieval and sparse retrieval respectively.”
Metadata
- Authors
- Lin, J., Ma, X., Lin, S.-C., Yang, J.-H., Pradeep, R., & Nogueira, R.
- Affiliations
- University of Waterloo (CA)
- Venue
- Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2021-07-11
- Pages
- 2356-2362
- OpenAlex
- W3180230246
- Cited by
- 360
- References
- 41
- Citations by year
- 2026: 67 · 2025: 72 · 2024: 61 · 2023: 97 · 2022: 44 · 2021: 18
- FWCI
- 45.53
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Waterloo-Huawei Joint Innovation Laboratory
- Canada First Research Excellence Fund
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- reproducibility, sparse retrieval, dense retrieval, hybrid retrieval, BM25, transformer representations, nearest neighbor search, neural ranking, relevance assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.