The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index; 3 of the 22 sources its claims cite were checked for support.
Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study
Open accessAbstract
BACKGROUND: Within systematic reviews, when searching for relevant references, it is advisable to use multiple databases. However, searching databases is laborious and time-consuming, as syntax of search strategies are database specific. We aimed to determine the optimal combination of databases needed to conduct efficient searches in systematic reviews and whether the current practice in published reviews is appropriate. While previous studies determined the coverage of databases, we analyzed the actual retrieval from the original searches for systematic reviews. METHODS: Since May 2013, the first author prospectively recorded results from systematic review searches that he performed at his institution. PubMed was used to identify systematic reviews published using our search strategy results. For each published systematic review, we extracted the references of the included studies. Using the prospectively recorded results and the studies included in the publications, we calculated recall, precision, and number needed to read for single databases and databases in combination. We assessed the frequency at which databases and combinations would achieve varying levels of recall (i.e., 95%). For a sample of 200 recently published systematic reviews, we calculated how many had used enough databases to ensure 95% recall. RESULTS: A total of 58 published systematic reviews were included, totaling 1746 relevant references identified by our database searches, while 84 included references had been retrieved by other search methods. Sixteen percent of the included references (291 articles) were only found in a single database; Embase produced the most unique references (n = 132). The combination of Embase, MEDLINE, Web of Science Core Collection, and Google Scholar performed best, achieving an overall recall of 98.3 and 100% recall in 72% of systematic reviews. We estimate that 60% of published systematic reviews do not retrieve 95% of all available relevant references as many fail to search important databases. Other specialized databases, such as CINAHL or PsycINFO, add unique references to some reviews where the topic of the review is related to the focus of the database. CONCLUSIONS: Optimal searches in systematic reviews should search at least Embase, MEDLINE, Web of Science, and Google Scholar as a minimum requirement to guarantee adequate and efficient coverage.
Overview
CoChat Summary
TL;DR
This prospective exploratory study analyzed actual database retrieval for 58 published systematic reviews and found that searching Embase, MEDLINE, Web of Science Core Collection, and Google Scholar together achieved the best recall. The authors estimate that many published systematic reviews fail to retrieve 95% of relevant references because they omit key databases.
Key findings
- Among 1,746 relevant references identified by database searches, 292 (17%) were found by only one database; Embase retrieved the most unique references (n = 132), followed by MEDLINE, Web of Science, and Google Scholar.
- The combination of Embase, MEDLINE, Web of Science Core Collection, and Google Scholar achieved an overall recall of 98.3% and retrieved all included references in 72% of the studied systematic reviews.
- Searching only Embase produced an average number needed to read (NNR) of 57, whereas the optimal four-database combination had an NNR of 73.
- In a sample of 200 recent systematic reviews, 97% searched MEDLINE, 61% searched Embase, 35% searched Web of Science, and only 10% used Google Scholar; the authors estimate that 60% of published systematic reviews do not retrieve 95% of all available relevant references.
Methods
- Since May 2013, the first author prospectively recorded results from systematic review searches performed at Erasmus MC; PubMed was searched in July 2016 for reviews published since 2014 with Erasmus MC first authors, yielding 58 included reviews.
- Included reviews had documented search strategies and results and had used at least Embase, MEDLINE, Cochrane CENTRAL, Web of Science, and Google Scholar; included references were extracted and located in original EndNote libraries to determine which database retrieved them.
- Recall, precision, and number needed to read were calculated for single databases and combinations in Excel; a separate sample of 200 recent PubMed systematic reviews was analyzed to assess current database selection practice.
Limitations
- The authors state they did not investigate whether missing references changed review conclusions, and because the finding that Web of Science and Google Scholar improve recall is novel, they cannot conclude whether it is due to their dataset or is generalizable.
- The authors note that all searches were performed by one experienced biomedical information specialist, that searcher skill strongly affects search effectiveness, and that their searches still missed some relevant references; they also state that Scopus as a replacement for Embase or Web of Science remains uncertain.
- The text makes evident additional constraints: the study is exploratory, based at a single institution, and relies on a relatively small set of 58 systematic reviews, with specialized databases such as CINAHL and PsycINFO used only when topic-appropriate.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
2 Valid · 1 Not valid · 6 Not checked · 91 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using that combination, 93% of the systematic reviews in our study obtained levels of recall that could be considered acceptable (> 95%).”
§ Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study > Discussion
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For all but one domain, the traditional combination of Embase, MEDLINE, and Cochrane CENTRAL did not retrieve enough included references.”
§ Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study > Results > Differences between domains of reviews
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Embase was searched in 61% and Web of Science in 35%, and Google Scholar was only used in 10% of all reviews.”
§ Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study > Results > Status of current practice of database selection
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the five reviews that included only RCTs, four reached 100% recall if MEDLINE, Web of Science, and Google Scholar combined were complemented with Cochrane CENTRAL.”
§ Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study > Discussion
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Based on our findings, this combination achieves acceptable recall about half the time (47%).”
§ Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study > Results > Status of current practice of database selection
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
51 more claims
Create a free account to see all 100 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 24
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“although deduplication of references is essential to ensure the quality of SRs, there is currently no universal method to do so”
From Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research (2022) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although this database is one of the largest academic databases, offers extensive multidisciplinary coverage, and is widely used in bibliometric analyses, relevant publications indexed elsewhere or published in regional and non-indexed outlets may not have been captured [130].”
From Toward Holistic Aging in Lebanon: A Mapping of Gerontology and Geriatrics Research Trends, Gaps, and Future Directions (2026) · cites this paper as [130]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Meanwhile, others are highly specialized (e.g., PubMed, Medline, Compendix, etc.), which makes them unsuitable for more inclusive tasks, in particular, for the wide scope analysis and evaluation of multidisciplinary units [42,43].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [43]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These databases are widely used in review studies and the majority of them are included in the optimal database combination search19 which guarantee an adequate and efficient coverage of the scientific literature.”
From Occupational therapy interventions for adults with severe mental illness: a scoping review (2021) · cites this paper as [19]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These differences have led to recommendations that systematic review teams search multiple databases to maximize the likelihood of finding relevant studies [6, 39, 40].”
From PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews (2021) · cites this paper as [40]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study > Abstract > Results
“Sixteen percent of the included references (291 articles) were only found in a single database;”
§ Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study > Results > Unique references per database
“A total of 292 (17%) references were found by only one database.”
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1. The Cochrane Collaboration
Higgins, J. P. T., Green, S. R., & Higgins, J. P. T. (2008) · Cited by 7,175
A systematic approach to searching: an efficient and complete method to develop literature searches
Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018) · Journal of the Medical Library Association · doi:10.5195/jmla.2018.283 · Cited by 863
Self-citationComparing two sampling methods to engage hard-to-reach communities in research priority setting
Valerio, M. A., Rodriguez, N., Winkler, P., Lopez, J., Dennison, M., Liang, Y., & Turner, B. J. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0242-z · Cited by 543
Rethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.025 · Cited by 331
Self-citationSystematic reviews need systematic searchers.
(2005) · Europe PMC (PubMed Central) · Cited by 257
Bramer, W. M., Giustini, D., & Kramer, B. M. R. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0215-7 · Cited by 202
Self-citationHalladay, C. W., Trikalinos, T. A., Schmid, I. T., Schmid, C. H., & Dahabreh, I. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.12.017 · Cited by 166 · cited 1×
“Prior research on database importance for systematic reviews has looked primarily at whether included references could have theoretically been found in a certain database, but most have been unable to ascertain whether the researchers actually found the articles in those databases [10, 12, 16, 17, 26].”
BackgroundSearching multiple databases for systematic reviews: added value or diminishing returns?
Stevinson, C., & Lawlor, D. A. (2004) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2004.09.003 · Cited by 143
Bramer, W. M., Giustini, D., Kramer, B. M., & Anderson, P. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-115 · Cited by 139 · cited 1×
“Nevertheless others have concluded that a single database is not sufficient to retrieve all references for systematic reviews [18, 19].”
BackgroundSelf-citationSearching one or two databases was insufficient for meta-analysis of observational studies
Lemeshow, A. R., Blum, R. E., Berlin, J. A., Stoto, M. A., & Colditz, G. A. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.03.004 · Cited by 137
Cited by
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Self-citationRethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
Self-citationWeb of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696
The art of crafting a systematic literature review in entrepreneurship research
Kraus, S., Breier, M., & Dasí-Rodríguez, S. (2020) · International Entrepreneurship and Management Journal · doi:10.1007/s11365-020-00635-4 · Cited by 1,319
“Authors should use more than one database to cover most articles (Bramer et al. 2017).”
MethodAguirre Velasco, A., Cruz, I. S. S., Billings, J., Jimenez, M., & Rowe, S. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-02659-0 · Cited by 736
“Four databases were selected including MEDLINE®, Embase, PsycINFO, and Web of Science, as well as the search engine Google scholar, identified as an optimal database combination [20].”
Metadata
- Authors
- Bramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H.
- Affiliations
- Erasmus MC (NL)
- University of Utah (US)
- Maastricht University (NL)
- Corresponding author
- Bramer, Wichor M.
- Venue
- Systematic Reviews
- Published
- 2017-12
- Volume
- 6
- Issue
- 1
- Pages
- 245-245
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC5718002
- OpenAlex
- W2774587119
- Cited by
- 2,114
- References
- 27
- Citations by year
- 2026: 373 · 2025: 411 · 2024: 332 · 2023: 292 · 2022: 252 · 2021: 231 · 2020: 145 · 2019: 64 · 2018: 12 · 2017: 2
- FWCI
- 14.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor, 5 d)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- National Center for Advancing Translational Sciences (UL1TR001067)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- literature search strategies, search recall, search precision, database coverage, Embase, MEDLINE, Web of Science, Google Scholar, CINAHL, PsycINFO, number needed to read
- MeSH terms
- Humans, Prospective Studies, Review Literature as Topic, Databases, Bibliographic, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. https://app.cochat.ai/papers/10.1186/s13643-017-0644-y
- The paper
- Bramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H. (2017). Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Systematic Reviews. https://doi.org/10.1186/s13643-017-0644-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.