CoChat

Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study

Open access
Bramer, Wichor M., Giustini, Dean, Kramer, Bianca M. R. · 2016 · Systematic Reviews

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index; 2 of the 4 sources its claims cite were checked for support.

Abstract

BACKGROUND: Previously, we reported on the low recall of Google Scholar (GS) for systematic review (SR) searching. Here, we test our conclusions further in a prospective study by comparing the coverage, recall, and precision of SR search strategies previously performed in Embase, MEDLINE, and GS. METHODS: The original search results from Embase and MEDLINE and the first 1000 results of GS for librarian-mediated SR searches were recorded. Once the inclusion-exclusion process for the resulting SR was complete, search results from all three databases were screened for the SR's included references. All three databases were then searched post hoc for included references not found in the original search results. RESULTS: We checked 4795 included references from 120 SRs against the original search results. Coverage of GS was high (97.2 %) but marginally lower than Embase and MEDLINE combined (97.5 %). MEDLINE on its own achieved 92.3 % coverage. Total recall of Embase/MEDLINE combined was 81.6 % for all included references, compared to GS at 72.8 % and MEDLINE alone at 72.6 %. However, only 46.4 % of the included references were among the downloadable first 1000 references in GS. When examining data for each SR, the traditional databases' recall was better than GS, even when taking into account included references listed beyond the first 1000 search results. Finally, precision of the first 1000 references of GS is comparable to searches in Embase and MEDLINE combined. CONCLUSIONS: Although overall coverage and recall of GS are high for many searches, the database does not achieve full coverage as some researchers found in previous research. Further, being able to view only the first 1000 records in GS severely reduces its recall percentages. If GS would enable the browsing of records beyond the first 1000, its recall would increase but not sufficiently to be used alone in SR searching. Time needed to screen results would also increase considerably. These results support our assertion that neither GS nor one of the other databases investigated, is on its own, an acceptable database to support systematic review searching.

Overview

CoChat Summary

TL;DR

In a prospective study of 120 systematic reviews, Google Scholar had high coverage (97.2%) but much lower practical recall (46.4% among the first 1000 results) than Embase/MEDLINE combined (81.6%). The authors conclude that neither Google Scholar nor Embase or MEDLINE alone is acceptable as a single database for systematic review searching.

Key findings

  • Coverage was high for Google Scholar (97.2%; 4659/4795 after excluding 49 citation-only results indexed after the original search), slightly lower than Embase (97.5%) and higher than MEDLINE alone (92.3%); Embase/MEDLINE combined coverage was 97.5%.
  • Overall recall was 81.6% (3914/4795) for Embase/MEDLINE combined, 72.8% (3493) for all Google Scholar results, 72.6% (3481) for MEDLINE alone, but only 46.4% (2224) for the downloadable first 1000 Google Scholar results.
  • Precision of the first 1000 Google Scholar results was 1.9% (2224/118,509), comparable to Embase at 2.0% (3940/192,935) and MEDLINE at 2.8% (3506/126,657); precision of complete Google Scholar results was 0.03% (3493/10,092,939).
  • Per review, Google Scholar coverage was 100% for 68% of reviews, Embase for 63%, and MEDLINE for 34%; in 15 reviews fewer than 25% of included references were in the first 1000 Google Scholar results, while 9 reviews reached 100%, and all Google Scholar results reached at least 100% recall in 24 reviews (20.0%), exceeding 100% in 4 reviews because Google Scholar found references not found in traditional databases.

Methods

  • Prospective study of librarian-mediated systematic review searches performed by the first author at Erasmus MC; 520 exhaustive searches were saved between May 2013 and August 2015, and by August 2015 reviewers of 120 systematic reviews had screened results, yielding 4795 included references.
  • Original search results from MEDLINE via Ovid, Embase.com (searching Embase and MEDLINE records), and the first 1000 Google Scholar results (downloaded via Publish or Perish) were imported into EndNote; after inclusion-exclusion, included references were matched against original results, and post hoc Google Scholar searches were run for missed references using author names and title words combined with original strategies.
  • Coverage, recall, and precision were calculated overall and per review; data were visualized in boxplots, and search strategies with exceptionally high or low Google Scholar recall were examined for topic and search-term characteristics.

Limitations

  • The paper states that the research could be improved by using search strategies created by multiple independent information specialists at baseline, though the authors question whether this would alter their conclusions.
  • The paper states that Google Scholar searches are never wholly replicable later because the algorithm changes, that citation-only results may not have been present at the time of the original search, and that only the first 1000 results can be viewed or downloaded.
  • Evident from the text: all baseline strategies were created by a single information specialist; Google Scholar translations were constrained by a 256-character limit and lack of proximity operators; and reviewers used additional databases and extended search methods, so the total included references could exceed those retrieved from the three databases compared.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13643-016-0215-7

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Legend of boxplot figures
Fig. 1 Legend of boxplot figuresCC BY · PubMed Central
Fig. 2 Total coverage and recall of MEDLINE (ML), Embase/MEDLINE (EM), and Google Scholar (GS)
Fig. 2 Total coverage and recall of MEDLINE (ML), Embase/MEDLINE (EM), and Google Scholar (GS)CC BY · PubMed Central
Fig. 3 Coverage per SR for GS, MEDLINE, and Embase/MEDLINE
Fig. 3 Coverage per SR for GS, MEDLINE, and Embase/MEDLINECC BY · PubMed Central
Fig. 4 Recall per SR for the first 1000 references in GS, total GS, MEDLINE, and Embase/MEDLINE
Fig. 4 Recall per SR for the first 1000 references in GS, total GS, MEDLINE, and Embase/MEDLINECC BY · PubMed Central
Fig. 5 Precision per SR for the first 1000 references in GS, GS, MEDLINE, Embase/MEDLINE, and overall
Fig. 5 Precision per SR for the first 1000 references in GS, GS, MEDLINE, Embase/MEDLINE, and overallCC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 0 Not valid · 1 Not checked · 66 Own results and interpretations

45 more claims

Create a free account to see all 69 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 10
  • It may state two separate claims as one. 9

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 11 references

Cited by

See all 202

Metadata

Authors
Bramer, W. M., Giustini, D., & Kramer, B. M. R.
Affiliations
  • Erasmus MC (NL)
  • University of British Columbia (CA)
  • Utrecht University (NL)
Corresponding author
Bramer, Wichor M.
Venue
Systematic Reviews
Published
2016-03-01
Volume
5
Issue
1
Pages
39-39
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2292818836
Cited by
202
References
11
Citations by year
2026: 17 · 2025: 27 · 2024: 24 · 2023: 32 · 2022: 20 · 2021: 16 · 2020: 17 · 2019: 16 · 2018: 14 · 2017: 10 · 2016: 9
FWCI
5.73
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (acceptedVersion, other-oa)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • cIRcle (University of British Columbia) (submittedVersion, cc-by)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
Google Scholar, MEDLINE, Embase, search coverage, search recall, search precision, bibliographic database comparison
MeSH terms
Humans, Prospective Studies, Review Literature as Topic, Databases, Bibliographic, MEDLINE, Information Storage and Retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. https://app.cochat.ai/papers/10.1186/s13643-016-0215-7
The paper
Bramer, W. M., Giustini, D., & Kramer, B. M. R. (2016). Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Systematic Reviews. https://doi.org/10.1186/s13643-016-0215-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.