BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Open accessAbstract
Existing neural information retrieval (IR) models have often been studied in homogeneous and narrow settings, which has considerably limited insights into their out-of-distribution (OOD) generalization capabilities. To address this, and to facilitate researchers to broadly evaluate the effectiveness of their models, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous evaluation benchmark for information retrieval. We leverage a careful selection of 18 publicly available datasets from diverse text retrieval tasks and domains and evaluate 10 state-of-the-art retrieval systems including lexical, sparse, dense, late-interaction and re-ranking architectures on the BEIR benchmark. Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches, highlighting the considerable room for improvement in their generalization capabilities. We hope this framework allows us to better evaluate and understand existing retrieval systems, and contributes to accelerating progress towards better robust and generalizable systems in the future. BEIR is publicly available at https://github.com/UKPLab/beir.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Keyword-based retrieval methods like BM25 typically underperform semantic search methods.
“While keyword-based retrieval methods like BM25 typically underperform semantic search methods (Thakur et al., 2021), experimental results shown in Figure 2 (left) demonstrate BM25 remain surprisingly competitive with PREMem.”
From Pre-Storage Reasoning for Episodic Memory: Shifting Inference Burden to Memory for Personalized Dialogue (2025) · cites this paper as Thakur et al., 2021
BM25 outperforms neural retrievers across a wide array of tasks when applied in zero-shot settings.
“outperforms neural retrievers across a wide array of tasks, when applied in zero-shot settings (Thakur et al., 2021).”
From In-Context Retrieval-Augmented Language Models (2023) · cites this paper as Thakur et al., 2021
Existing RAG benchmarks typically assume a static or slowly changing knowledge base and short-lived interactions, emphasizing retrieval accuracy and grounding over continual updating and selective forgetting of information that is central to memory agents.
“However, existing RAG benchmarks typically assume a static or slowly changing knowledge base and short-lived interactions, emphasizing retrieval accuracy and grounding over the continual updating and selective forgetting of information that is central to memory agents.”
From Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (2025) · cites this paper as Thakur et al., 2021
Retrieval over the live web differs substantially from traditional information retrieval evaluations.
“A key component of generative research synthesis is retrieval over the live web, which differs substantially from traditional information retrieval evaluations [46, 32].”
From DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis (2025) · cites this paper as [46]
The MiniCPM-Embedding-Light model achieves a zero-shot performance of 55.27 in nDCG@10 on the BEIR benchmark, outperforming bge-large-en-v1.5 and jina-embeddings-v3.
“It achieves competitive performance on multiple benchmarks, and shows good generalization ability given a zero-shot performance of 55.27 in nDCG@10 on the BEIR benchmark (Thakur et al., 2021), outperforming other popular alternatives such as bge-large-en-v1.5 (BAAI, 2024) and jina-embeddings-v3 (JinaAI, 2024), which achieve 54.29 and 53.88 in nDCG@10, respectively.”
From DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research (2025) · cites this paper as Thakur et al., 2021
The citation lists haven’t been fetched yet
References 65 · Cited by 45
- Authors
- Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., & Gurevych, I.
- Affiliations
- Technische Universität Darmstadt (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2104.08663
- OpenAlex
- W3153833878
- Cited by
- 45
- References
- 65
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 12 · 2023: 15 · 2022: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- zero-shot evaluation, information retrieval models, LLM agents, Model Context Protocol, cross-references, search costs, reproducible benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
