CoChat
Sign Up
Cited by 168Meta-analysis and systematic reviewsscientometrics and bibliometrics researchInformation Retrieval and Search Behavior

Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study

Open access
Ewald, Hannah, Klerings, Irma, Wagner, Gernot, Heise, Thomas L., Stratil, Jan M., Lhachimi, Stefan K., Hemkens, Lars G., Gartlehner, Gerald +2 more·2022·Journal of Clinical Epidemiology
DOI 10.1016/j.jclinepi.2022.05.022
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND AND OBJECTIVES: Assessing changes in coverage, recall, review, conclusions and references not found when searching fewer databases. METHODS: In randomly selected 60 Cochrane reviews, we checked included study publications' coverage (indexation) and recall (findability) using different search approaches with MEDLINE, Embase, and CENTRAL and related them to authors' conclusions and certainty. We assessed characteristics of unfound references. RESULTS: Overall 1989/2080 included references, were indexed in ≥1 database (coverage = 96%). In reviews where using one of our search approaches would not change conclusions and certainty (n = 44-54), median coverage and recall were highest (range 87.9%-100.0% and 78.2%-93.3%, respectively). Here, searching ≥2 databases reached >95% coverage and ≥87.9% recall. In reviews with unchanged conclusions but less certainty (n = 2-8): 63.3%-79.3% coverage and 45.0%-75.0% recall. In reviews with opposite conclusions (n = 1-3): 63.3%-96.6% and 52.1%-78.7%. In reviews where a conclusion was no longer possible (n = 3-7): 60.6%-86.0% and 20.0%-53.8%. The 265 references that were indexed but unfound were more often abstractless (30% vs. 11%) and older (28% vs. 17% published before 1991) than found references. CONCLUSION: Searching ≥2 databases improves coverage and recall and decreases the risk of missing eligible studies. If researchers suspect that relevant articles are difficult to find, supplementary search methods should be used.

AI snapshot

Written from the paper’s full text.

TL;DR

In a metaresearch study of 60 Cochrane reviews, searching two or more databases improved coverage and recall and reduced the risk of inappropriate conclusions, but even MEDLINE + Embase + CENTRAL did not achieve total recall. Many references remained unfound, especially older references and those without abstracts.

Key findings

  • Of 2,080 included references, 1,989 (95.6%) were indexed in at least one of MEDLINE, Embase, or CENTRAL, but only 1,724 (83%) could be found by searching M + E + C. Median coverage and recall were highest for M + E + C (98.5% and 90.0%, respectively); single-database median coverage was highest in CENTRAL (91.3%) and single-database median recall was highest in MEDLINE (75.5%).
  • In reviews where conclusions and certainty did not change (n = 44–54 depending on search approach), median coverage ranged from 87.9% to 100.0% and median recall from 78.2% to 93.3%; searching two databases reached >95% coverage and 87.9% recall. In reviews where no conclusion was possible (n = 3–7), median coverage was 60.6%–86.0% and median recall 20.0%–53.8%; in reviews with an opposite conclusion (n = 1–3), coverage was 63.3%–96.6% and recall 52.1%–78.7%.
  • Overall, 356/2,080 (17%) references were not found by any of the seven limited database search approaches. After removing 30 duplicates, of 326 individual not-found references, 91 (28%) were not indexed; among the 235 indexed but not found, 70% had an abstract, 99% were in English, 74% were on cardiovascular topics, and 62% involved nutrition or physical activity interventions. Not-found references were older (28% vs. 17% published before 1991) and more often had no abstract (30% vs. 11%) than found references.
  • Many indexed references not found by M + E + C had no abstract; the paper states that almost a third of studies indexed but not found by searching MEDLINE, Embase, and CENTRAL had no abstract.

Methods

  • This was a sub-study of a larger methodological project. The authors randomly selected 60 Cochrane reviews that had a replicable search strategy, clear conclusions, and a meta-analysis. The gold standard-set was all references in the “References to studies included in this review” sections (2,080 references).
  • They assessed seven limited search approaches: MEDLINE-only, Embase-only, CENTRAL-only, M + E, M + C, E + C, and M + E + C. For each, they calculated database coverage and recall and related these to the original authors’ conclusion categories: same conclusion/same certainty, same conclusion/less certainty, opposite conclusion, and no conclusion possible. Reference list checking was not considered in this sub-study.
  • Data management and analysis used EndNote X8 and Excel 2016. They performed descriptive statistics (mean, median, IQR) and examined unfound references for indexation status, abstract availability, publication date, language, intervention type, and topic; they compared abstract availability and publication date before/after 1991 between found and not-found indexed references.

Limitations

  • The paper states that the relationship between recall and conclusions is not straightforward: unfound references may have belonged to narrative syntheses rather than meta-analyses, affecting recall but not conclusions. It also states that recall was calculated independently of coverage, so a search finding all indexed references could still have low recall if the review contained many nonindexed references.
  • The paper states that limiting indexation checks to MEDLINE-only and Embase-only may underestimate coverage and recall compared with actual search interfaces; using Cochrane reviews may overestimate CENTRAL coverage; and updated reviews may include references added after the original publication. It also states that search strategies were assumed comprehensive and only checked for formal errors, not comprehensiveness or overall quality, so low recall could reflect suboptimal or deliberately specific searches.
  • Evident from the text: the sample was limited to 60 Cochrane reviews, and 2,014/2,080 references (96.8%) were RCTs or quasi-randomized trials, so findings may not generalize to other review types or study designs; as an observational metaresearch study, it cannot establish causality.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2022.05.022

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 15 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In reviews where the conclusion and certainty did not change (n=44 to 54 of 60), CENTRAL and each combination of at least two databases reached over 95% median coverage.

    “In reviews where the conclusion and certainty did not change (depending on search approach n 5 44 to 54 of 60), CENTRAL and each combination of at least two databases reached over 95% median coverage (although a mean coverage of at least 95% was only reached with C þ E and M þ E þ C).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
  • Findingown data

    Median coverage of the four reviews where conclusion was no longer possible or changed to opposite ranged between 64.4% and 96.6%.

    “Median coverage of these reviews ranged between 64.4% and 96.6%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
  • Findingown data

    CENTRAL and Embase median recall was higher for reviews on pharmacological interventions.

    “CENTRAL and Embase median recall was higher for reviews on pharmacological interventions (Table 2).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
    • Rests on an assumption: The intervention type classification (pharmacological vs nonpharmacological) is valid and meaningful.
  • Findingown data

    The 91 not indexed references were mainly grey literature (36/91 conference abstracts, 14/91 study registers, 12/91 theses, and 9/91 other types) and some journal articles (20/91, 22%).

    “These were mainly grey literature (36/91 conference abstracts, 14/91 study registers, 12/ 91 theses, and 9/91 other types) and some journal articles (20/91, 22%).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
  • Findingown data

    Overall, 356 of the 2,080 (17%) references were not found when using any of the seven search approaches in a limited number of databases.

    “Overall, 356 of the 2,080 (17%) references were not found when using any of the seven search approaches in a limited number of databases.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
  • May combine two claims. 26
  • The quote only partly states this claim. 2
  • Classified as probably not a substantial claim. 15

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In reviews where conclusions and certainty did not change, searching 2 databases reached >95% coverage and 87.9% recall.

    “Here, searching 2 databases reached O95% coverage and 87.9% recall.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    Depending on the approach, there were always three to seven Cochrane reviews where a conclusion was no longer possible (median recall 20.0%-53.8%) and one to three reviews where the opposite conclusion was drawn (median recall 52.1%-78.7%).

    “Depending on which approach was used (single database search, two databases, or combination of all three), there were always three to seven Cochrane reviews where a conclusion was no longer possible (median recall ranged from 20.0% to 53.8%) and one to three reviews where the opposite conclusion was drawn (median recall ranged from 52.1% to 78.7%).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Median recall per review ranged between 62.4% and 100% when limiting to M+E+C and conclusion did not change.

    “Median recall per review ranged between 62.4% and 100% (Table 4).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    The combination of databases increased median coverage, with the highest coverage attained with M+E+C (median 98.5%, IQR 91.9%-100.0%).

    “The combination of databases increased the median coverage.The highest coverage could be attained with M þ E þ C (Median 98.5%; IQR 91.9%-100.0%)(Table 2).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    The highest median single-database recall was in MEDLINE (median 75.5%, IQR 56.8%-87.0%), which seemed balanced between reviews of pharmacological and nonpharmacological interventions.

    “The highest median single-database recall was in MEDLINE (median: 75.5%, IQR: 56.8%-87.0%,Table 2) which seemed balanced between reviews of pharmacological and nonpharmacological interventions.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Of the remaining 235 individual, not found, indexed references, the majority had an abstract (70%) and was written in English (99%).

    “Of the remaining 235 individual, not found, indexed references, the majority had an abstract (70%), and was written in English (99%).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    Of the 1,969 references, 1,706 were journal articles and 263 grey literature articles, of which 238 (14%) and 90 (34%) were not found, respectively.

    “Of the 1,969 references, 1,706 were journal articles and 263 grey literature articles of which 238 (14%) and 90 (34%) were not found, respectively.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    Median coverage in CENTRAL and Embase was higher for reviews on pharmacological than nonpharmacological interventions, while the opposite was true for MEDLINE.

    “The median coverage in CENTRAL and EMBASE was higher for reviews on pharmacological than nonpharmacological interventions although this was the opposite for MEDLINE (Table 2).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
    • Rests on an assumption: The intervention type classification (pharmacological vs nonpharmacological) is valid and meaningful.

  • May combine two claims.

    Findingown data

    In comparison to references that could be found with at least one limited database search approach, references that were not found were on average older (28% vs. 17% published before 1991) and more often had no abstract available (30% vs. 11%).

    “In comparison to the references that could be found with at least one of the limited database search approaches, the references that were not found were on average older (28% vs. 17% published before 1991) and more often had no abstract available (30% vs. 11%) (Appendix 5).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingcitation

    When limiting information sources to M+E+C, a conclusion was no longer possible in three reviews and changed to the opposite in one review.

    “When limiting the information sources to M þ E þ C, a conclusion was no longer possible or changed to the opposite of the original review in three [14e16] and one [17] reviews, respectively.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites 14e16

      Not checked in the free run.

    • cites [17] Blessberger, 2014, Perioperative beta-blockers for preventing surgery-related mortality and morbidity, Cochrane Database Syst Rev

      Not checked in the free run.

    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    Three of these four reviews included at least one reference that was not indexed (overall 11 of 111, 10%), but coverage differed greatly between individual reviews (54%-100%).

    “Three of these four reviews included at least one reference that was not indexed (overall 11 of 111, 10%), but coverage differed greatly between individual reviews (54%-100%; Appendix 5).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    The medical topics of the 235 not found indexed references were mainly cardiovascular diseases (74%) and the intervention types were nutrition or physical activity (62%).

    “The medical topics were mainly cardiovascular diseases (74%) and the intervention types were nutrition or physical activity (62%) (Table 5).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    The 90/263 grey literature references not found when limiting to M+E+C and conclusion unchanged were 15/60 study register entries (25%), 52/177 conference abstracts (29%), 11/14 theses (79%), 3/3 books (100%), 4/4 pharma documents (100%), and 5/5 other document types (100%).

    “The 90/263 grey literature references that could not be found when limiting the information sources to M þ E þ C and the conclusion remained unchanged were 15/60 study register entries (25%), 52/177 conference abstracts (29%), 11/14 theses (79%), 3/3 books (100%), 4/4 pharma documents (100%), and 5/5 other document types (100%).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    In two instances, all references that were indexed were also found (coverage = recall), but the conclusion was still not possible.

    “In two instances [14,25], all references that were indexed were also found (coverage 5 recall), but the conclusion was still not possible.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [14] Geretsegger, 2014, Music therapy for people with autism spectrum disorder. Cochrane Database of Systematic Reviews, Cochrane Database Syst Rev

      Not checked in the free run.

    • cites [25] Wong, 2015, Blood pressure lowering efficacy of dual alpha and beta blockers for primary hypertension, Cochrane Database Syst Rev

      Not checked in the free run.

    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    References of Cochrane reviews where the conclusion was no longer possible tended to have lower median database coverage than the others, although reviews where conclusion and certainty remained the same had a markedly higher median coverage than those where changes occurred.

    “We found that references of Cochrane reviews where the conclusion was no longer possible tended to have lower median database coverage than the others, although Cochrane reviews where the conclusion and its certainty remained the same had a markedly higher median coverage than those where changes occurred.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingreasoning

    Even the combination of the three most frequently used databases (MEDLINE, Embase and CENTRAL) did not suffice to achieve total recall or to avoid any change in the conclusions of systematic reviews.

    “Yet, even the combination of the three most frequently used databases Medline, Embase, and CENTRAL did not suffice to achieve total recall or avoid any change in the conclusions of systematic reviews.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
    • Rests on an assumption: The 60 Cochrane reviews and the references they included are representative of systematic reviews and their included studies more generally.

  • May combine two claims.

    Findingown data

    Searching in more databases improved recall, but not all indexed references could be found.

    “As expected, searching in more databases improved recall, but not all indexed references could be found.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • May combine two claims.

    Findingown data

    Median coverage of a single database ranged between 83% and 91%, and median recall between 71% and 76%.

    “In comparison, median coverage of a single database ranged between 83% and 91%, and median recall between 71% and 76%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

  • Classified as probably not a substantial claim.

    Premisecitation

    Removing studies that were not indexed in MEDLINE from meta-analyses had only little impact on effect estimates.

    “Removing studies that were not indexed in MEDLINE from the meta-analyses had only little impact on the effect estimates [3,4,6].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [3] Hartling, 2016, The contribution of databases to the results of systematic reviews: a cross-sectional study, BMC Med Res Methodol source

      Not checked in the free run.

    • cites [4] Halladay, 2015, Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions, J Clin Epidemiol source

      Not checked in the free run.

    • cites [6] Marshall, 2019, Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study, J Clin Epidemiol source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Recent methods studies found that the majority of relevant references included in Cochrane reviews were indexed in MEDLINE.

    “Recent methods studies evaluating research studies of medical or psychological topics [3e5] found that the majority of relevant references included in Cochrane reviews were available (indexed) in MEDLINE (i.e., database coverage).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites 3e5

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Bramer et al. found high database coverage in MEDLINE and Embase, but in some cases even the cumulative recall of MEDLINE and Embase was below 50%.

    “Bramer et al. [8] assessed systematic search results in various medical topics, from therapeutic effectiveness and diagnostic accuracy to ethics and public health.They found high database coverage in MEDLINE and Embase, but reported that in some cases even the cumulative recall of MEDLINE and Embase was below 50%.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [8] Bramer, 2016, Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study, Syst Rev source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Searching at least two databases improves coverage and recall and decreases the chances of making an inappropriate conclusion in a review.

    “Searching at least two databases improves coverage and recall and decreases the chances of making an inappropriate conclusion in a review.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: A change in a Cochrane review's conclusion when searching fewer databases indicates an inappropriate conclusion.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The search strategies of the included Cochrane reviews were checked only for formal errors (syntax, spelling, block building), not for comprehensiveness of search terms or overall quality.

    “Hence, we only checked the search strategies for formal errors (i.e., syntax, spelling, block building), not for comprehensiveness of search terms or overall quality.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.

  • May combine two claims.

    Premisecitation

    Most Cochrane reviews' search strategies are designed by trained information specialists who follow the Cochrane handbook and MECIR standards.

    “Third, most Cochrane reviews' search strategies are designed by trained information specialists who follow the Cochrane handbook [1] and MECIR standards [2].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [1] Higgins

      Not checked in the free run.

    • cites [2] Higgins, 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The overall database coverage and recall observed in this study are similar to findings of previous research.

    “The overall database coverage and recall observed in our study are similar to findings of previous research [7e9].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites 7e9

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    It may not always be necessary to find all the relevant references to draw a 'correct' conclusion.

    “It may not always be necessary to find all the relevant references do draw a ''correct'' conclusion.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The potential overestimation of CENTRAL coverage was countered by verifying that the references were available before the date of the latest search.

    “We sought to counter this issue by verifying that the references were available before the date of the latest search.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The relationship between recall and conclusions is not straightforward.

    “First, the relationship between recall and conclusions is not straightforward.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    In determining whether records were indexed in the databases, the analysis was limited to MEDLINE-only and Embase-only indexing so that the results would be independent of the search interface.

    “Second, when determining if records were indexed in the databases, we limited results to MEDLINE-only and Embase-only, to ensure the results were independent of the search interface.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    A low recall may relate to suboptimal searches, but it is also possible that a good-quality search was deliberately designed to be specific, so that a lower recall was a deliberate choice.

    “Although a low recall could relate to suboptimal searches [32], it is also possible that a good quality search was designed to be specific and hence, a lower recall would have been a deliberate choice.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] Franco, 2018, Identification of problems in search strategies in Cochrane Reviews, Res Synth Methods source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    As most databases only allow searching in the metadata of a study rather than the full text, a reference without an abstract is harder to find and more dependent on the use of subject headings such as MeSH and Emtree.

    “As most databases only allow searching in the metadata of a study (especially in title, abstract, author keywords and subject headings) rather than the full text, a reference without an abstract is harder to find and more dependent on the use of subject headings such as MeSH and Emtree.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Recall was calculated independently of database coverage, meaning a search strategy that found all indexed references would have low recall if the review contained many nonindexed references.

    “In addition, we calculated the recall independently of the database coverage.This means that even a search strategy that found all indexed references would have a low recall if the review contained many nonindexed references.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Because some search interfaces index additional documents beyond MEDLINE-only or Embase-only records, the study may have underestimated the coverage and recall of searching a limited number of databases.

    “However, this might differ from the actual search experience as some interfaces index additional documents, e.g., PubMed includes MEDLINE-in-Process materials and nonMEDLINE records, and Embase.comincludes most MeSH-indexed MEDLINE records.Hence, we may have effectively underestimated the coverage and recall of searching in a limited number of databases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Using references from Cochrane reviews may overestimate the database coverage of CENTRAL, because references to RCTs included in Cochrane reviews are regularly added to CENTRAL.

    “Conversely, by using references from Cochrane reviews, we might overestimate the database coverage of CENTRAL because references to RCTs included in Cochrane reviews are regularly added to this database.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    It is possible that references not found by the searches belonged to studies included in the narrative syntheses, not the meta-analyses, and in that case they affected recall but not conclusions.

    “As Cochrane reviews list all included study publications, it is possible that references not found by the searches belonged to studies included in the narrative syntheses, not the metaanalyses.In that case, they affected the recall of the searches but not the conclusions of the review.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Although searching two databases may be enough for some systematic reviews, others may require a more thorough approach like MECIR.

    “Although searching two databases may be enough for some systematic reviews, others may require a more thorough approach like MECIR.”
    Quote in the paperNot a substantial claim
36 of 36

  • The paper relies on this without support.

    The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.

    “From each review, we used all references reported in the ''References to studies included in this review'' section as a gold standard-set to assess 14 different search approaches using a limited number of sources (i.e., the 14 possible combinations of MEDLINE, Embase, CENTRAL with and without screening reference lists).”
    “For this sub-study, we used the same gold standard-set as mentioned above.”

  • The paper relies on this without support.

    The 60 randomly selected Cochrane reviews are representative of Cochrane reviews generally.

    “We randomly selected 60 Cochrane reviews that included a replicable search strategy, had clear conclusions, and reported a meta-analysis [12].”

  • The paper relies on this without support.

    A change in a Cochrane review's conclusion when searching fewer databases indicates an inappropriate conclusion.

    “Searching at least two databases improves coverage and recall and decreases the chances of making an inappropriate conclusion in a review.”

  • The paper relies on this without support.

    The database indexation status at the time of the original search can be accurately determined using database-specific identifiers.

    “For each reference, we noted the availability in the target databases using the database-specific identifiers.We used these identifiers to determine if available studies were found by the original search strategies.”

  • The paper relies on this without support.

    The intervention type classification (pharmacological vs nonpharmacological) is valid and meaningful.

    “We determined the clinical topic of each review and categorized all reviews according to the type of intervention (pharmacological or nonpharmacological).”

  • The paper relies on this without support.

    The original conclusions of the Cochrane reviews are a valid benchmark for assessing whether limited searches change conclusions.

    “Then, we asked the original authors whether the resulting summary of findings tables would alter their conclusions.”

  • The authors state this as an assumption.

    The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.

    “We assumed that these would be comprehensive and of high quality to serve as gold-standard [30].”

  • The paper relies on this without support.

    The 60 Cochrane reviews and the references they included are representative of systematic reviews and their included studies more generally.

    “Characteristics of references of studies included by the 60 Cochrane reviews”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The guidance is uniquely based on empirical evidence about rapid review methods, supplemented by expert input and feedback from a formal evaluation of how it was used in practice.

    “Although these other guides are also evidence informed and we encourage their use, our guide is uniquely grounded in empirical evidence evaluating methods of rapid reviews, 12 further enriched by expert input [13][14][15][16] and feedback from a formal evaluation of its real world use.7”

    From Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness (2024) · cites this paper as [12]

Cites

Flagged first, then by citations

3 of the 32 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. Geretsegger, 2014, Music therapy for people with autism spectrum disorder. Cochrane Database of Systematic Reviews, Cochrane Database Syst Rev, 2014, CD004381

Cited by

Sorted by citations

Related

Authors
Ewald, H., Klerings, I., Wagner, G., Heise, T. L., Stratil, J. M., Lhachimi, S. K., Hemkens, L. G., Gartlehner, G., Armijo-Olivo, S., & Nussbaumer-Streit, B.
Affiliations
  • University of Basel (CH)
  • Universität für Weiterbildung Krems (AT)
  • University of Bremen (DE)
  • Ludwig-Maximilians-Universität München (DE)
  • University of Alberta (CA)
Corresponding author
Ewald, Hannah
ORCID
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2022-09
Volume
149
Pages
154-164
ISSN
0895-4356 (ISSN-L)
DOI
10.1016/j.jclinepi.2022.05.022
OpenAlex
W4281701339
Semantic Scholar
bfbec399290bbad3b129739c10adc93ca968aad8
Cited by
168
References
32
Citations by year
2026: 61 · 2025: 54 · 2024: 44 · 2023: 9
FWCI
16.15
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Open access LMU (Ludwid Maxmilian's Universitat Munchen) (submittedVersion, cc-by-nc-nd)
  • Informationszentrum Lebenswissenschaften (ZB MED) (submittedVersion, cc-by-nc-nd)
  • Library Osnabrück University (Hochschule Osnabrück) (submittedVersion, cc-by-nc-nd)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
database searching, search recall, missing studies, bibliographic database indexing, MEDLINE, Embase, Central
MeSH terms
Abstracting and Indexing, Humans, Databases, Bibliographic, Databases, Factual, MEDLINE, Information Storage and Retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review