CoChat
Sign Up
Cited by 44Meta-analysis and systematic reviewsscientometrics and bibliometrics researchInformation Retrieval and Search Behavior

PubMed coverage varied across specialties and over time: a large-scale study of included studies in Cochrane reviews

Open access
Frandsen, Tove Faber, Eriksen, Mette Brandt, Hammer, David Mortan Grøne, Christensen, Janne Buck·2019·Journal of Clinical Epidemiology
DOI 10.1016/j.jclinepi.2019.04.015
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective PubMed is one of the most commonly used search tools in biomedical and life sciences. Existing studies of database coverage generally conclude that searching PubMed may not be sufficient although some find that the contributions from other databases are modest at best. However, generalizability of the studies of the coverage of PubMed is typically restricted. The objective of this study is to analyze the coverage of PubMed across specialties and over time.Study design and setting We use the more than 50,000 included studies in all Cochrane reviews published from 2012 to 2016 as our population and examine if the studies and resulting publications can be identified in PubMed.Results The results show that PubMed has a coverage of 70.9, 95% confidence interval (CI) (68.40, 73.30) of all the included publications and 82.8%, 95% CI (80.9, 84.7) of the included studies. There are huge differences in coverage across and within specialties. In addition, coverage varies within groups over time.Conclusion Databases used for searching topics within the groups with highly varying or low coverage should be chosen with care as PubMed may have a relatively low coverage.

AI snapshot

Written from the paper’s full text.

TL;DR

This large-scale study examined PubMed coverage of included studies in all Cochrane reviews published 2012–2016. PubMed covered 70.9% of included publications and 82.8% of included studies, with large variation across and within Cochrane groups and over time.

Key findings

  • Overall PubMed coverage was 61,052 of 86,171 included publications, or 70.9%, 95% CI [68.40, 73.30], and 45,483 of 54,903 included studies, or 82.8%, 95% CI [80.9, 84.7].
  • Coverage varied widely by Cochrane group: mean publication coverage ranged from less than 50% for Cystic Fibrosis and Genetic Disorders and Airways to more than 90% for Metabolic and Endocrine Disorders and Childhood Cancer; individual years ranged from less than 40% to 100%.
  • Study-level coverage also varied across groups, from a little over 40% to more than 95%, and was higher than publication-level coverage because only one publication per study needed to be available in PubMed.
  • Within-group coverage over time was sometimes stable and sometimes highly variable: some groups varied by less than 5 percentage points, while others ranged by more than 50 percentage points; Tobacco Addiction had more than 90% study coverage and appeared stable, whereas Musculoskeletal varied from a little over 40% to 90%.

Methods

  • Population: all included studies in all Cochrane reviews from 2012 to 2016, extracted from the Cochrane Library by Cochrane group using Python; analyses used both publications and studies as units.
  • PubMed availability: Cochrane direct PubMed links were available for 49.1% of included publications and were verified by matching title and journal name; matching problems affected 1.2% of included publications, or 2.4% of links, and 335 publications (0.4%) with incorrect links were manually corrected. For remaining publications, title-and-year queries were generated automatically and manually verified. Data collection took place from January 2018 to December 2018.
  • Analysis: coverage rates were calculated overall and by review year and Cochrane group, with 95% CIs; the study examined PubMed, not Medline.

Limitations

  • Stated: the study measured whether included studies could be found in PubMed, not whether searchers could retrieve them; poor searching could find much less than what is available, so the reported coverage rates can be seen as an upper limit.
  • Stated: the study focused exclusively on PubMed, did not consider the difference between PubMed and Medline, and did not assess whether studies or publications might be retrievable in other key databases.
  • Stated: if a Cochrane group included many publications not indexed by PubMed, such as grey publications, lower observed PubMed coverage may reflect thorough grey-source searching rather than PubMed coverage per se.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2019.04.015

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 12 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Mean PubMed coverage of included publications varies greatly across Cochrane groups, ranging from less than 50% (Cystic Fibrosis and Genetic Disorders, Airways) to more than 90% (Metabolic and Endocrine Disorders, Childhood Cancer).

    “There are huge differences in coverage across as well as within specialties.”
    • Also stated in: § 3. Results · plain
      “the mean coverage of the included publications in PubMed ranges from less than 50 percent for Cystic Fibrosis and Genetic Disorders and Airways to more than 90 percent for Metabolic and Endocrine Disorders and Childhood Cancer.”
    Quote in the paperSubstantial claim
  • Findingown data

    PubMed coverage of the included publications in all Cochrane reviews published 2012-2016 is 70.9% (95% CI 68.40-73.30), i.e. 61,052 of 86,171 publications.

    “The results show that PubMed has a coverage of 70.9, 95% CI [68.40,73.30] of all the included publications and 82.8%, 95% CI [80.9, 84.7] of the included studies.”
    • Also stated in: § 3. Results · plain
      “The overall coverage from 2012 to 2016 in PubMed is 61,052 publications out of 86,171 corresponding to a coverage rate of 70.9, 95% CI [68.40,73.30].”
    • Also stated in: § 4. Discussion and conclusion · plain
      “The results show that PubMed has a coverage of 71 percent of all the included publications and 83 percent of all included studies in Cochrane reviews from 2012 to 2016.”
    Quote in the paperSubstantial claim
  • Findingown data

    The Musculoskeletal Group's PubMed coverage rate varies from a little over 40% to 90%.

    “It is another matter altogether with the Musculoskeletal Group, where the coverage rate varies from a little over 40 percent to 90 percent indicating that coverage for this group can be difficult to predict.”
    Quote in the paperSubstantial claim
  • Findingown data

    PubMed coverage of the included studies in all Cochrane reviews published 2012-2016 is 82.8% (95% CI 80.9-84.7), i.e. 45,483 of 54,903 studies.

    “The results show that PubMed has a coverage of 70.9, 95% CI [68.40,73.30] of all the included publications and 82.8%, 95% CI [80.9, 84.7] of the included studies.”
    • Also stated in: § 3. Results · plain
      “PubMed covers 45,483 out of 54,903 studies and thus has a coverage rate of 82.8%, 95% CI [80.9, 84.7].”
    • Also stated in: § 4. Discussion and conclusion · plain
      “The results show that PubMed has a coverage of 71 percent of all the included publications and 83 percent of all included studies in Cochrane reviews from 2012 to 2016.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    PubMed coverage within each Cochrane group differs depending on whether the unit of analysis is publications or studies.

    “Consequently, the coverage of PubMed within each group differs also depending on the unit of analysis (publications or studies).”
    • Also stated in: § 4. Discussion and conclusion · plain
      “the results of this study show that the coverage of PubMed within each group differs depending on the unit of analysis being publications or studies.”
    Quote in the paperSubstantial claim
  • May combine two claims. 11
  • Classified as probably not a substantial claim. 8

  • May combine two claims.

    Findingown data

    PubMed coverage varies within Cochrane groups over time, from less than 5 percentage points in some groups to over 50 percentage points in others.

    “Some groups vary very little over time (e.g. less than 5 percentage points) whereas other groups can range considerably (e.g. over 50 percentage points).”
    • Also stated in: § 5. Conclusion · hedged
      “Some groups vary very little over time (e.g. less than 5 percentage points) whereas other groups can range considerably (e.g. over 50 percentage points), possibly due to the fact that the specific subjects of the Cochrane reviews may vary a lot, even though they come from the same group.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Because Figure 1 excludes publications with no publication year, for which PubMed coverage is lower, the average coverage in Figure 1 (73.0%) is higher than in the entire dataset (70.9%).

    “Figure 1 excludes the publications which have no publication year, and as PubMed has a lower coverage of these (e.g.data sets and grey publications) the average coverage in this figure is higher than in the entire dataset (73.0%instead of 70.9%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    PubMed coverage of included studies differs across Cochrane groups, ranging from a little over 40% to more than 95%.

    “Again, the coverage of studies depicts large differences across groups (with the coverage rate ranging from a little over 40 percent to more than 95 percent).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The Tobacco Addiction Group has a PubMed coverage of included studies of more than 90%, and this coverage is stable over time.

    “The Tobacco Addiction Group has a PubMed coverage of included studies of more than 90 percent and coverage seems to be stable over time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    PubMed coverage is slightly higher for older publications, while remaining relatively stable for publications from the last two decades.

    “The figure shows that coverage is more is slightly higher for older publications, although the coverage is relatively stable for publications from the last two decades.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    PubMed may have relatively low coverage for Cochrane groups whose coverage is highly varying or low.

    “Databases used for searching topics within the groups with highly varying or low coverage should be chosen with care as PubMed may have a relatively low coverage.”
    • Also stated in: § 5. Conclusion · plain
      “Based on this study we conclude that coverage can be very difficult for some groups to predict and thus researchers should choose databases for searches within these areas with care.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisenone given

    Cochrane recommends Medline, so Medline coverage is of great importance.

    “Cochrane recommends Medline and thus Medline coverage is of great importance, however, PubMed has a higher sensitivity than Medline (46) and many researchers as well as clinicians use PubMed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The effect of adding another database to those already searched in a systematic review is greater for meta-analyses with 10 or more included studies.

    “An effect of adding another database to the already searched databases in a systematic review is found to be greater in the case of meta-analyses with 10+ included studies (20).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] Halladay, 2015, Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions, J Clin Epidemiol source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The literature on database coverage generally concludes that searching PubMed may not be sufficient and that other databases can contribute relevant publications.

    “The literature on database coverage generally concludes that searching PubMed may not be sufficient and that other databases can contribute with relevant publications (6,(27)(28)(29).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [6] Sampson, 2003, Should meta-analysts search Embase in addition to Medline?, J Clin Epidemiol source

      Not checked in the free run.

    • cites [27] Bahaadinbeigy, 2010, MEDLINE versus EMBASE and CINAHL for telemedicine searches, Telemed J E Health source

      Not checked in the free run.

    • cites [28] Slobogean, 2009, MEDLINE, EMBASE, and Cochrane index most primary studies but not abstracts included in orthopedic meta-analyses, J Clin Epidemiol source

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Existing studies of coverage typically restrict to one or a few topics, which limits their generalizability.

    “Summing up, the existing studies typically restrict to one or a few topics which seems to limit generalizability.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Incorrect links with no correct match available in PubMed, covering 335 publications (0.4%), were manually corrected in the dataset.

    “This latter group of links, covering 335 publications (0.4%), was subsequently manually corrected in the dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Premisecitation

    Some studies argue that the contributions from other databases are modest at best.

    “However, some studies argue that the contributions from other databases are modest at best (15,23).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [15] Bayliss, 2014, Where and how to search for information on the effectiveness of public health interventions - a case study for prevention of cardiovascular disease, Health Info Libr J source

      Not checked in the free run.

    • cites [23] Levay, 2015, The contributions of MEDLINE, other bibliographic databases and various search techniques to NICE public health guidance, Evid Based Libr Info Pract source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    All Cochrane PubMed links were verified by matching title and journal name, which revealed matching problems in 1.2% of the included publications (2.4% of the links).

    “However, as the links may not necessarily direct the reader to the correct publication in PubMed all links were verified matching title and journal name.This revealed matching problems in 1.2% of the included publications which amounts to 2.4% of the links.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Existing studies of coverage indicate considerable variation depending on topic, and coverage of Medline ranges from 73 to 85%.

    “The results of these studies of coverage indicate that considerable variation is found depending on topic (23) and coverage of Medline ranges from 73 to 85%.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [23] Levay, 2015, The contributions of MEDLINE, other bibliographic databases and various search techniques to NICE public health guidance, Evid Based Libr Info Pract source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    A search of Medline or PubMed alone is generally not considered adequate, and the number of searched sources is generally increasing.

    “However, a search of Medline or PubMed alone is generally not considered adequate, and the number of searched sources is generally increasing (13).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [13] Lam, 2016, Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014, J Med Libr Assoc source

      Not checked in the free run.

15 of 15

  • The paper relies on this without support.

    PubMed is one of the most commonly used search tools in Biomedical and Life Sciences.

    “PubMed is one of the most commonly used search tools in Biomedical and Life Sciences.”

  • The paper relies on this without support.

    PubMed contains over 29 million references, the vast majority (more than 25 million) of which come from Medline.

    “PubMed contains over 29 million references coming from various sources but the vast majority of the references (> 25 million) are from Medline.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Figure 1 includes 2016 despite stated 2015 endpoint

    Minor

    The text says Figure 1 shows publication years through 2015, but the figure's year labels extend to 2016.

1 of 1

The citation lists haven’t been fetched yet

References 45 · Cited by 44

Authors
Frandsen, T. F., Eriksen, M. B., Hammer, D. M. G., & Christensen, J. B.
Affiliations
  • University of Southern Denmark (DK)
  • Goethe University Frankfurt (DE)
  • Hospital South West Jutland (DK)
Corresponding author
Frandsen, Tove Faber
ORCID
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2019-08
Volume
112
Pages
59-66
ISSN
0895-4356 (ISSN-L)
DOI
10.1016/j.jclinepi.2019.04.015
OpenAlex
W2942937778
Semantic Scholar
a14894aca0b5454cb1f307d66a28cecbb953187d
Cited by
44
References
45
Citations by year
2026: 1 · 2025: 7 · 2024: 11 · 2023: 8 · 2022: 3 · 2021: 8 · 2020: 3 · 2019: 3
FWCI
2.84
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Clinical Epidemiology (publishedVersion)
  • PubMed (publishedVersion)
  • University of Southern Denmark Research Portal (University of Southern Denmark) (submittedVersion, cc-by-nc-nd)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
PubMed, database coverage, literature searching, included studies, bibliographic database searching, specialty differences, temporal trends, search strategies
MeSH terms
Systematic Reviews as Topic, Data Management, Humans, Publications, Confidence Intervals, Databases, Bibliographic, Information Storage and Retrieval, Biomedical Research, PubMed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review