Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index; 2 of the 4 sources its claims cite were checked for support.
Abstract
BACKGROUND: Previously, we reported on the low recall of Google Scholar (GS) for systematic review (SR) searching. Here, we test our conclusions further in a prospective study by comparing the coverage, recall, and precision of SR search strategies previously performed in Embase, MEDLINE, and GS. METHODS: The original search results from Embase and MEDLINE and the first 1000 results of GS for librarian-mediated SR searches were recorded. Once the inclusion-exclusion process for the resulting SR was complete, search results from all three databases were screened for the SR's included references. All three databases were then searched post hoc for included references not found in the original search results. RESULTS: We checked 4795 included references from 120 SRs against the original search results. Coverage of GS was high (97.2 %) but marginally lower than Embase and MEDLINE combined (97.5 %). MEDLINE on its own achieved 92.3 % coverage. Total recall of Embase/MEDLINE combined was 81.6 % for all included references, compared to GS at 72.8 % and MEDLINE alone at 72.6 %. However, only 46.4 % of the included references were among the downloadable first 1000 references in GS. When examining data for each SR, the traditional databases' recall was better than GS, even when taking into account included references listed beyond the first 1000 search results. Finally, precision of the first 1000 references of GS is comparable to searches in Embase and MEDLINE combined. CONCLUSIONS: Although overall coverage and recall of GS are high for many searches, the database does not achieve full coverage as some researchers found in previous research. Further, being able to view only the first 1000 records in GS severely reduces its recall percentages. If GS would enable the browsing of records beyond the first 1000, its recall would increase but not sufficiently to be used alone in SR searching. Time needed to screen results would also increase considerably. These results support our assertion that neither GS nor one of the other databases investigated, is on its own, an acceptable database to support systematic review searching.
Overview
CoChat Summary
TL;DR
In a prospective study of 120 systematic reviews, Google Scholar had high coverage (97.2%) but much lower practical recall (46.4% among the first 1000 results) than Embase/MEDLINE combined (81.6%). The authors conclude that neither Google Scholar nor Embase or MEDLINE alone is acceptable as a single database for systematic review searching.
Key findings
- Coverage was high for Google Scholar (97.2%; 4659/4795 after excluding 49 citation-only results indexed after the original search), slightly lower than Embase (97.5%) and higher than MEDLINE alone (92.3%); Embase/MEDLINE combined coverage was 97.5%.
- Overall recall was 81.6% (3914/4795) for Embase/MEDLINE combined, 72.8% (3493) for all Google Scholar results, 72.6% (3481) for MEDLINE alone, but only 46.4% (2224) for the downloadable first 1000 Google Scholar results.
- Precision of the first 1000 Google Scholar results was 1.9% (2224/118,509), comparable to Embase at 2.0% (3940/192,935) and MEDLINE at 2.8% (3506/126,657); precision of complete Google Scholar results was 0.03% (3493/10,092,939).
- Per review, Google Scholar coverage was 100% for 68% of reviews, Embase for 63%, and MEDLINE for 34%; in 15 reviews fewer than 25% of included references were in the first 1000 Google Scholar results, while 9 reviews reached 100%, and all Google Scholar results reached at least 100% recall in 24 reviews (20.0%), exceeding 100% in 4 reviews because Google Scholar found references not found in traditional databases.
Methods
- Prospective study of librarian-mediated systematic review searches performed by the first author at Erasmus MC; 520 exhaustive searches were saved between May 2013 and August 2015, and by August 2015 reviewers of 120 systematic reviews had screened results, yielding 4795 included references.
- Original search results from MEDLINE via Ovid, Embase.com (searching Embase and MEDLINE records), and the first 1000 Google Scholar results (downloaded via Publish or Perish) were imported into EndNote; after inclusion-exclusion, included references were matched against original results, and post hoc Google Scholar searches were run for missed references using author names and title words combined with original strategies.
- Coverage, recall, and precision were calculated overall and per review; data were visualized in boxplots, and search strategies with exceptionally high or low Google Scholar recall were examined for topic and search-term characteristics.
Limitations
- The paper states that the research could be improved by using search strategies created by multiple independent information specialists at baseline, though the authors question whether this would alter their conclusions.
- The paper states that Google Scholar searches are never wholly replicable later because the algorithm changes, that citation-only results may not have been present at the time of the original search, and that only the first 1000 results can be viewed or downloaded.
- Evident from the text: all baseline strategies were created by a single information specialist; Google Scholar translations were constrained by a 256-character limit and lack of proximity operators; and reviewers used additional databases and extended search methods, so the total included references could exceed those retrieved from the three databases compared.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
2 Valid · 0 Not valid · 1 Not checked · 66 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 49 of these citation only results, the only citing paper in GS was the review based on our search strategy.”
§ Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study > Results > Overall coverage
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, 179 of these were present as citations only.”
§ Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study > Results > Overall coverage
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A rate of at least 100 % was reached in 24 SRs (20.0 %).”
§ Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study > Results > Recall per SR
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the precision of the total set of search results in GS was much lower than that of the other databases (0.03 %).”
§ Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study > Results > Precision per SR
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For four SRs, the recall of all search results in GS was even higher than 100 % because GS was able to find included references that had not been found in the traditional databases but were identified via other sources”
§ Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study > Results > Recall per SR
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
45 more claims
Create a free account to see all 69 claims and every source verdict.
- Probably not a substantial claim. 10
- It may state two separate claims as one. 9
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Third, although searching both Embase and PubMed is expected to cover approximately 97.5% of relevant published articles,33 and although we also included Web of Science in our search, we cannot exclude the possibility that additional relevant articles may have been missed as a result of restricting our search to these 3 databases.”
From Association of Levels of Physical Activity With Risk of Parkinson Disease: A Systematic Review and Meta-analysis (2018) · cites this paper as [33]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Previous studies have also calculated precision and recall of search systems from data reported by specific evidence‐synthesis studies.24, 32, 33, 34”
The cited source says what the claim attributes to it.
In the citing paper
“Our findings support the criticism of Bramer et al,33 Bramer et al,34 and Boeker et al25 and indicate that Google Scholar's coverage and recall is an inadequate reason to use it as principal search system in systematic searches.53”
The cited source says what the claim attributes to it.
In the citing paper
“These databases were selected because they are known to have good coverage of clinical research,39 40”
From Systematic review protocol assessing the processes for linking clinical trial registries and their published results (2016) · cites this paper as [39]
In the citing paper
“Bramer et al. [8] assessed systematic search results in various medical topics, from therapeutic effectiveness and diagnostic accuracy to ethics and public health.They found high database coverage in MEDLINE and Embase, but reported that in some cases even the cumulative recall of MEDLINE and Embase was below 50%.”
From Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study (2022) · cites this paper as [8]
Citations
Cites
The role of google scholar in evidence reviews and its applicability to grey literature searching
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138237 · Cited by 2,387 · cited 1×
“[10] who compared the recall of GS to that of Web of Science for SRs in the field of environmental science.”
BackgroundIs the coverage of google scholar enough to be used alone for systematic reviews
Gehanno, J.-F., Rollin, L., & Darmoni, S. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-7 · Cited by 357 · cited 1×
“[1] prompted a discussion around the utility of Google Scholar (GS) to support systematic review (SR) searching.”
BackgroundBoeker, M., Vach, W., & Motschall, E. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-131 · Cited by 225 · cited 1×
“Low precision has always been considered a problem in GS [11], but when accounting for actually usable search results (i.”
Backgroundde Vos, R.-J., Windt, J., & Weir, A. (2014) · British Journal of Sports Medicine · doi:10.1136/bjsports-2013-093281 · Cited by 186
Bramer, W. M., Giustini, D., Kramer, B. M., & Anderson, P. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-115 · Cited by 139
Self-citationShifting paradigms in eligibility criteria for live kidney donation: a systematic review
Ahmadi, A. R., Lafranca, J. A., Claessens, L. A., et al. (2014) · Kidney International · doi:10.1038/ki.2014.118 · Cited by 103 · cited 1×
“[6], Ambagtsheer (not yet published), Leermakers, Moreira (7)), the first author, together with the reviewer, had not been able to translate a complicated embase.”
BackgroundPrecision of healthcare systematic review searches in a cross‐sectional sample
Sampson, M., Tetzlaff, J., & Urquhart, C. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.42 · Cited by 80
Systematic review on cashew nut allergy
van der Valk, J. V. D., Dubois, A., van Wijk, R. V., Wichers, H., & de Jong, N. D. (2014) · Allergy. European Journal of Allergy and Clinical Immunology · doi:10.1111/all.12401 · Cited by 74
Brettle, A., Hulme, C., & Ormandy, P. (2007) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2007.00702.x · Cited by 28 · cited 1×
“Librarian-mediated searches (combining MeSH terms and free text terms in traditional databases) achieve better results than non-librarian searches [3, 4].”
BackgroundGardois, P., Calabrese, R., Colombi, N., Deplano, A., Lingua, C., Longo, F., Villanacci, M. C., Miniero, R., & Piga, A. (2011) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2011.00957.x · Cited by 26 · cited 1×
“Librarian-mediated searches (combining MeSH terms and free text terms in traditional databases) achieve better results than non-librarian searches [3, 4].”
Background
Cited by
Gusenbauer, M., & Haddaway, N. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1378 · Cited by 2,379
“Our findings support the criticism of Bramer et al, 33 Bramer et al, 34 and Boeker et al 25 and indicate that Google Scholar's coverage and recall is an inadequate reason to use it as principal search system in systematic searches.”
BackgroundResultBramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0644-y · Cited by 2,114
Self-citationKyu, H. H., Bachman, V. F., Alexander, L. T., Mumford, J. E., Afshin, A., Estep, K., Veerman, J. L., Delwiche, K., Iannarone, M. L., Moyer, M. L., et al. (2016) · BMJ · doi:10.1136/bmj.i3857 · Cited by 1,272
A systematic approach to searching: an efficient and complete method to develop literature searches
Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018) · Journal of the Medical Library Association · doi:10.5195/jmla.2018.283 · Cited by 863
“For medically oriented searches, the coverage and recall of Embase, which includes the MEDLINE database, are superior to those of MEDLINE [16].”
MethodResultSelf-citationHalevi, G., Moed, H., & Bar-Ilan, J. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.06.005 · Cited by 488
Metadata
- Authors
- Bramer, W. M., Giustini, D., & Kramer, B. M. R.
- Affiliations
- Erasmus MC (NL)
- University of British Columbia (CA)
- Utrecht University (NL)
- Corresponding author
- Bramer, Wichor M.
- Venue
- Systematic Reviews
- Published
- 2016-03-01
- Volume
- 5
- Issue
- 1
- Pages
- 39-39
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC4772334
- OpenAlex
- W2292818836
- Cited by
- 202
- References
- 11
- Citations by year
- 2026: 17 · 2025: 27 · 2024: 24 · 2023: 32 · 2022: 20 · 2021: 16 · 2020: 17 · 2019: 16 · 2018: 14 · 2017: 10 · 2016: 9
- FWCI
- 5.73
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (acceptedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- cIRcle (University of British Columbia) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- Google Scholar, MEDLINE, Embase, search coverage, search recall, search precision, bibliographic database comparison
- MeSH terms
- Humans, Prospective Studies, Review Literature as Topic, Databases, Bibliographic, MEDLINE, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. https://app.cochat.ai/papers/10.1186/s13643-016-0215-7
- The paper
- Bramer, W. M., Giustini, D., & Kramer, B. M. R. (2016). Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Systematic Reviews. https://doi.org/10.1186/s13643-016-0215-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.