Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index; 3 of the 22 sources its claims cite were checked for support.

Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study

Open access

Abstract

BACKGROUND: Within systematic reviews, when searching for relevant references, it is advisable to use multiple databases. However, searching databases is laborious and time-consuming, as syntax of search strategies are database specific. We aimed to determine the optimal combination of databases needed to conduct efficient searches in systematic reviews and whether the current practice in published reviews is appropriate. While previous studies determined the coverage of databases, we analyzed the actual retrieval from the original searches for systematic reviews. METHODS: Since May 2013, the first author prospectively recorded results from systematic review searches that he performed at his institution. PubMed was used to identify systematic reviews published using our search strategy results. For each published systematic review, we extracted the references of the included studies. Using the prospectively recorded results and the studies included in the publications, we calculated recall, precision, and number needed to read for single databases and databases in combination. We assessed the frequency at which databases and combinations would achieve varying levels of recall (i.e., 95%). For a sample of 200 recently published systematic reviews, we calculated how many had used enough databases to ensure 95% recall. RESULTS: A total of 58 published systematic reviews were included, totaling 1746 relevant references identified by our database searches, while 84 included references had been retrieved by other search methods. Sixteen percent of the included references (291 articles) were only found in a single database; Embase produced the most unique references (n = 132). The combination of Embase, MEDLINE, Web of Science Core Collection, and Google Scholar performed best, achieving an overall recall of 98.3 and 100% recall in 72% of systematic reviews. We estimate that 60% of published systematic reviews do not retrieve 95% of all available relevant references as many fail to search important databases. Other specialized databases, such as CINAHL or PsycINFO, add unique references to some reviews where the topic of the review is related to the focus of the database. CONCLUSIONS: Optimal searches in systematic reviews should search at least Embase, MEDLINE, Web of Science, and Google Scholar as a minimum requirement to guarantee adequate and efficient coverage.

Overview

CoChat Summary

TL;DR

This prospective exploratory study analyzed actual database retrieval for 58 published systematic reviews and found that searching Embase, MEDLINE, Web of Science Core Collection, and Google Scholar together achieved the best recall. The authors estimate that many published systematic reviews fail to retrieve 95% of relevant references because they omit key databases.

Key findings

  • Among 1,746 relevant references identified by database searches, 292 (17%) were found by only one database; Embase retrieved the most unique references (n = 132), followed by MEDLINE, Web of Science, and Google Scholar.
  • The combination of Embase, MEDLINE, Web of Science Core Collection, and Google Scholar achieved an overall recall of 98.3% and retrieved all included references in 72% of the studied systematic reviews.
  • Searching only Embase produced an average number needed to read (NNR) of 57, whereas the optimal four-database combination had an NNR of 73.
  • In a sample of 200 recent systematic reviews, 97% searched MEDLINE, 61% searched Embase, 35% searched Web of Science, and only 10% used Google Scholar; the authors estimate that 60% of published systematic reviews do not retrieve 95% of all available relevant references.

Methods

  • Since May 2013, the first author prospectively recorded results from systematic review searches performed at Erasmus MC; PubMed was searched in July 2016 for reviews published since 2014 with Erasmus MC first authors, yielding 58 included reviews.
  • Included reviews had documented search strategies and results and had used at least Embase, MEDLINE, Cochrane CENTRAL, Web of Science, and Google Scholar; included references were extracted and located in original EndNote libraries to determine which database retrieved them.
  • Recall, precision, and number needed to read were calculated for single databases and combinations in Excel; a separate sample of 200 recent PubMed systematic reviews was analyzed to assess current database selection practice.

Limitations

  • The authors state they did not investigate whether missing references changed review conclusions, and because the finding that Web of Science and Google Scholar improve recall is novel, they cannot conclude whether it is due to their dataset or is generalizable.
  • The authors note that all searches were performed by one experienced biomedical information specialist, that searcher skill strongly affects search effectiveness, and that their searches still missed some relevant references; they also state that Scopus as a replacement for Embase or Web of Science remains uncertain.
  • The text makes evident additional constraints: the study is exploratory, based at a single institution, and relies on a relatively small set of 58 systematic reviews, with specialized databases such as CINAHL and PsycINFO used only when topic-appropriate.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13643-017-0644-y

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Percentage of systematic reviews for which a certain database combination reached a certain recall. The X-axis represents the percentage of reviews for which a specific combination of databases, as shown on the y-axis, reached a certain recall (represented with bar colors). Abbreviations: EM Embase, ML MEDLINE, WoS Web of Science, GS Google Scholar. Asterisk indicates that the recall of all databases has been calculated over all included references. The recall of the database combinations was calculated over all included references retrieved by any database
Fig. 1 Percentage of systematic reviews for which a certain database combination reached a certain recall. The X-axis represents the percentage of reviews for which a specific combination of databases, as shown on the y-axis, reached a certain recall (represented with bar colors). Abbreviations: EM Embase, ML MEDLINE, WoS Web of Science, GS Google Scholar. Asterisk indicates that the recall of all databases has been calculated over all included references. The recall of the database combinations was calculated over all included references retrieved by any databaseCC BY · PubMed Central
Fig. 2 Percentage of systematic reviews of a certain domain for which the combination Embase, MEDLINE and Cochrane CENTRAL reached a certain recall
Fig. 2 Percentage of systematic reviews of a certain domain for which the combination Embase, MEDLINE and Cochrane CENTRAL reached a certain recallCC BY · PubMed Central
Fig. 3 Legend of Figs. 3 and 4
Fig. 3 Legend of Figs. 3 and 4CC BY · PubMed Central
Fig. 4 The ratio between number of results per database combination and the total number of results for all databases
Fig. 4 The ratio between number of results per database combination and the total number of results for all databasesCC BY · PubMed Central
Fig. 5 The ratio between precision per database combination and the total precision for all databases
Fig. 5 The ratio between precision per database combination and the total precision for all databasesCC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 1 Not valid · 6 Not checked · 91 Own results and interpretations

51 more claims

Create a free account to see all 100 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 26
  • It may state two separate claims as one. 24
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

5 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 27 references

Cited by

See all 2,114

Metadata

Authors
Bramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H.
Affiliations
  • Erasmus MC (NL)
  • University of Utah (US)
  • Maastricht University (NL)
Corresponding author
Bramer, Wichor M.
Venue
Systematic Reviews
Published
2017-12
Volume
6
Issue
1
Pages
245-245
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2774587119
Cited by
2,114
References
27
Citations by year
2026: 373 · 2025: 411 · 2024: 332 · 2023: 292 · 2022: 252 · 2021: 231 · 2020: 145 · 2019: 64 · 2018: 12 · 2017: 2
FWCI
14.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Research Publications (Maastricht University) (submittedVersion, other-oa)
Funders
National Center for Advancing Translational Sciences (UL1TR001067)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
literature search strategies, search recall, search precision, database coverage, Embase, MEDLINE, Web of Science, Google Scholar, CINAHL, PsycINFO, number needed to read
MeSH terms
Humans, Prospective Studies, Review Literature as Topic, Databases, Bibliographic, Information Storage and Retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. https://app.cochat.ai/papers/10.1186/s13643-017-0644-y
The paper
Bramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H. (2017). Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Systematic Reviews. https://doi.org/10.1186/s13643-017-0644-y

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.