The DOI matches its Crossref record; 29 of the 32 references it cites were found in a registry or an index; none of the 15 sources its claims cite is checked for support yet.
Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study
Open accessAbstract
BACKGROUND AND OBJECTIVES: Assessing changes in coverage, recall, review, conclusions and references not found when searching fewer databases. METHODS: In randomly selected 60 Cochrane reviews, we checked included study publications' coverage (indexation) and recall (findability) using different search approaches with MEDLINE, Embase, and CENTRAL and related them to authors' conclusions and certainty. We assessed characteristics of unfound references. RESULTS: Overall 1989/2080 included references, were indexed in ≥1 database (coverage = 96%). In reviews where using one of our search approaches would not change conclusions and certainty (n = 44-54), median coverage and recall were highest (range 87.9%-100.0% and 78.2%-93.3%, respectively). Here, searching ≥2 databases reached >95% coverage and ≥87.9% recall. In reviews with unchanged conclusions but less certainty (n = 2-8): 63.3%-79.3% coverage and 45.0%-75.0% recall. In reviews with opposite conclusions (n = 1-3): 63.3%-96.6% and 52.1%-78.7%. In reviews where a conclusion was no longer possible (n = 3-7): 60.6%-86.0% and 20.0%-53.8%. The 265 references that were indexed but unfound were more often abstractless (30% vs. 11%) and older (28% vs. 17% published before 1991) than found references. CONCLUSION: Searching ≥2 databases improves coverage and recall and decreases the risk of missing eligible studies. If researchers suspect that relevant articles are difficult to find, supplementary search methods should be used.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
In a metaresearch study of 60 Cochrane reviews, searching two or more databases improved coverage and recall and reduced the risk of inappropriate conclusions, but even MEDLINE + Embase + CENTRAL did not achieve total recall. Many references remained unfound, especially older references and those without abstracts.
Key findings
- Of 2,080 included references, 1,989 (95.6%) were indexed in at least one of MEDLINE, Embase, or CENTRAL, but only 1,724 (83%) could be found by searching M + E + C. Median coverage and recall were highest for M + E + C (98.5% and 90.0%, respectively); single-database median coverage was highest in CENTRAL (91.3%) and single-database median recall was highest in MEDLINE (75.5%).
- In reviews where conclusions and certainty did not change (n = 44–54 depending on search approach), median coverage ranged from 87.9% to 100.0% and median recall from 78.2% to 93.3%; searching two databases reached >95% coverage and 87.9% recall. In reviews where no conclusion was possible (n = 3–7), median coverage was 60.6%–86.0% and median recall 20.0%–53.8%; in reviews with an opposite conclusion (n = 1–3), coverage was 63.3%–96.6% and recall 52.1%–78.7%.
- Overall, 356/2,080 (17%) references were not found by any of the seven limited database search approaches. After removing 30 duplicates, of 326 individual not-found references, 91 (28%) were not indexed; among the 235 indexed but not found, 70% had an abstract, 99% were in English, 74% were on cardiovascular topics, and 62% involved nutrition or physical activity interventions. Not-found references were older (28% vs. 17% published before 1991) and more often had no abstract (30% vs. 11%) than found references.
- Many indexed references not found by M + E + C had no abstract; the paper states that almost a third of studies indexed but not found by searching MEDLINE, Embase, and CENTRAL had no abstract.
Methods
- This was a sub-study of a larger methodological project. The authors randomly selected 60 Cochrane reviews that had a replicable search strategy, clear conclusions, and a meta-analysis. The gold standard-set was all references in the “References to studies included in this review” sections (2,080 references).
- They assessed seven limited search approaches: MEDLINE-only, Embase-only, CENTRAL-only, M + E, M + C, E + C, and M + E + C. For each, they calculated database coverage and recall and related these to the original authors’ conclusion categories: same conclusion/same certainty, same conclusion/less certainty, opposite conclusion, and no conclusion possible. Reference list checking was not considered in this sub-study.
- Data management and analysis used EndNote X8 and Excel 2016. They performed descriptive statistics (mean, median, IQR) and examined unfound references for indexation status, abstract availability, publication date, language, intervention type, and topic; they compared abstract availability and publication date before/after 1991 between found and not-found indexed references.
Limitations
- The paper states that the relationship between recall and conclusions is not straightforward: unfound references may have belonged to narrative syntheses rather than meta-analyses, affecting recall but not conclusions. It also states that recall was calculated independently of coverage, so a search finding all indexed references could still have low recall if the review contained many nonindexed references.
- The paper states that limiting indexation checks to MEDLINE-only and Embase-only may underestimate coverage and recall compared with actual search interfaces; using Cochrane reviews may overestimate CENTRAL coverage; and updated reviews may include references added after the original publication. It also states that search strategies were assumed comprehensive and only checked for formal errors, not comprehensiveness or overall quality, so low recall could reflect suboptimal or deliberately specific searches.
- Evident from the text: the sample was limited to 60 Cochrane reviews, and 2,014/2,080 references (96.8%) were RCTs or quasi-randomized trials, so findings may not generalize to other review types or study designs; as an observational metaresearch study, it cannot establish causality.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
49 Own result · 15 Own interpretation · 10 Not checked yet · 6 Own method · 2 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In reviews where the conclusion and certainty did not change (depending on search approach n 5 44 to 54 of 60), CENTRAL and each combination of at least two databases reached over 95% median coverage (although a mean coverage of at least 95% was only reached with C þ E and M þ E þ C).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.4. Coverage in relation to reviews' conclusions
plainn = 44- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Median coverage of these reviews ranged between 64.4% and 96.6%.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.4. Coverage in relation to reviews' conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CENTRAL and Embase median recall was higher for reviews on pharmacological interventions (Table 2).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.3. Recall
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
- Rests on an assumption: The intervention type classification (pharmacological vs nonpharmacological) is valid and meaningful.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These were mainly grey literature (36/91 conference abstracts, 14/91 study registers, 12/ 91 theses, and 9/91 other types) and some journal articles (20/91, 22%).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.6. Characteristics of references not found
plainn = 326- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, 356 of the 2,080 (17%) references were not found when using any of the seven search approaches in a limited number of databases.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.6. Characteristics of references not found
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the 326 individual references not found, 91 (28%) were not indexed in MED-LINE, Embase or CENTRAL.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.6. Characteristics of references not found
plainn = 326- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In reviews where the conclusion did not change but the authors were less certain (n 5 2 to 8 of 60), median coverage was much lower ranging from 63.3% to 78.3% from single database searches to M þ E þ C (Table 3).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.4. Coverage in relation to reviews' conclusions
plaincorrelationaln = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For nearly half of the reviews under investigation (n 5 29; 16 pharmacological, 13 nonpharmacological), there was at least one database or combination with a coverage of 100% (Appendix 3).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.2. Coverage
plainn = 29- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In reviews where a conclusion could no longer be drawn (n 5 3 to 7 of 60 reviews), we found the lowest median coverage which ranged from 60.6% (C) to 86.0% (M þ E).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.4. Coverage in relation to reviews' conclusions
plaincorrelationaln = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In reviews where using one of our search approaches would not change conclusions and certainty (n 5 44-54), median coverage and recall were highest (range 87.9%-100.0%and 78.2%-93.3%,respectively).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > Abstract
plaincorrelationaln = 44- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall 1989/2080 included references, were indexed in 1 database (coverage 5 96%).”
- Also stated in: § 4.2. Coverage · plain
“Of the 2,080 references in the gold standard-set, 1,989 (95.6%) were indexed in at least one of the three databases (median coverage 5 98.5%; Table 2).”
- Also stated in: § 5.1. Overall results · plain
“The majority of references included in a random sample of 60 Cochrane reviews were indexed in at least one of the databases MEDLINE, Embase, and CENTRAL.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
- Also stated in: § 4.2. Coverage · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many studies, almost a third, which were indexed but not found by searching MEDLINE, Embase and CENTRAL had no abstract.”
- Also stated in: § 5.1. Overall results · plain
“A third of the references in our study that were indexed in at least one database but were not found had no abstract.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > What this adds to what is known
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 5.1. Overall results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In reviews where a conclusion was no longer possible (n 5 3-7): 60.6%-86.0%and 20.0%-53.8%.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > Abstract
plainn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When limiting the information sources to M þ E þ C, the conclusion did not change (i.e., with the same or less certainty) in 56 reviews including 1,969 references.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.4. Coverage in relation to reviews' conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Analysis of recall by conclusion (Table 4) showed that search approaches that led to the same conclusion with the same certainty had a markedly higher median recall than those where any change occurred (''same direction but less certainty'', ''opposite conclusion'' or ''no conclusion possible'').”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.5. Recall in relation to reviews' conclusions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The highest median recall was 90.0% in M þ E þ C (IQR: 77.7%-100.0%;pharmacological interventions: n 5 30, median 94.1%, IQR 82.4%-100.0%;nonpharmacological interventions: n 5 30, median 87.0%, IQR 64.0%-97.7%)(Table 2).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.3. Recall
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Half of these reviews (28/56; 50%) included at least one reference that was not indexed and hence not found (80/1,969, 4%; coverage 5 78.2%-78.5%;Table 3).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.4. Coverage in relation to reviews' conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In reviews with opposite conclusions (n 5 1-3): 63.3%-96.6%and 52.1%-78.7%.”
- Also stated in: § 4.4. Coverage in relation to reviews' conclusions · plain
“In reviews where searching in a limited number of databases led to an opposite conclusion (n 5 1 to 3 of 60), median coverage ranged between 63.3% (M) and 96.6% 3). (C þ E, M þ E, M þ E þ C) (Table”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > Abstract
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
- Also stated in: § 4.4. Coverage in relation to reviews' conclusions · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all 60 Cochrane reviews, recall was markedly lower than coverage.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.3. Recall
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The category ''no conclusion possible'' had the lowest median recall across all limited database search approaches (median ranged from 20.0% to 53.8%).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.5. Recall in relation to reviews' conclusions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 18 Cochrane reviews (11 pharmacological, 7 nonpharmacological), at least one search strategy or combination had a recall of 100% (Appendix 3).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.3. Recall
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When limiting the number of databases, single-database coverage across all 60 Cochrane reviews was highest in CEN-TRAL (median: 91.3%, IQR: 75.0%-100.0%).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.2. Coverage
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was a gradual increase in recall with more databases included across evaluated search approaches.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.3. Recall
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In reviews with unchanged conclusions but less certainty (n 5 2-8): 63.3%-79.3%coverage and 45.0%-75.0%recall.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > Abstract
plainn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the 2,080 references in the gold standard-set, 1,724 (83%) could be found searching in M þ E þ C (Table 2).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.3. Recall
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the 111 study references (101 journal articles and 10 grey literature articles) included in the four reviews where a conclusion was no longer possible or changed to the opposite of the original review, 28 were not found (Appendix 4).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.5. Recall in relation to reviews' conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A sixth of these references had not been found (328/ 1,969, 17%).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.5. Recall in relation to reviews' conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found a median coverage of 94%-99% for every combination of at least two databases.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Almost a fifth of all references were not found when using any of the seven search approaches in a limited number of databases.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Cumulative search results of two or more databases improve coverage and recall and decrease the chances of making an inappropriate conclusion in a review.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 6. Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
- Rests on an assumption: The 60 Cochrane reviews and the references they included are representative of systematic reviews and their included studies more generally.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Characteristics of the 326 individual study references not found when limiting database searches to MEDLINE, Embase and/or CENTRAL”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 6. Conclusion
plainn = 326- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, on an individual level, there is no consistent pattern of low coverage and conclusion (Appendix 4).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“M þ E þ C | 1,989 | 95.6 | 98.5 | (91.9-100.0) | 1,724 | 82.9 | 90.0 | (77.7-100.0)”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 6. Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The median recall was 85%-90% for every combination of at least two databases.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“M þ E þ C | 1,989 | 95.6 | 98.5 | (91.9-100.0) | 1,724 | 82.9 | 90.0 | (77.7-100.0)”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 6. Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, in some cases even combined searches of MEDLINE, Embase, and CENTRAL to identify research may not be sufficient to draw any conclusion.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > What this adds to what is known
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A systematic review not always needs to find all eligible studies to arrive at the same conclusion with the same certainty as compared to including all eligible studies.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > What this adds to what is known
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, these results may be overestimating the impact of searching only one database as they have not assessed the recall of the indexed studies.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 1. Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Combining multiple information sources often improves recall compared to single-database searches [7e10].”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 1. Introduction
What the sources say
- Not checked yetcites 7e10
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In most cases, searching in a limited number of databases will lead to the same conclusions in comprehensive systematic reviews of RCTs.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > What is the implication, what should change now?
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: A change in a Cochrane review's conclusion when searching fewer databases indicates an inappropriate conclusion.
In the paper
“As our previous study has shown, combining the database search with reference list checking would already have helped to accurately determine conclusions [11].”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
What the sources say
- Not checked yetcites [11] Nussbaumer-Streit, 2018, Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study, J Clin Epidemiol source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Database coverage alone cannot predict whether relevant references will be found (5recall) by a search strategy.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 6. Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For some research topics a small number of database searches combined with supplementary search methods may be more useful than searching many bibliographic databases.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 6. Conclusion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
- Rests on an assumption: The 60 Cochrane reviews and the references they included are representative of systematic reviews and their included studies more generally.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In such cases it would have been necessary to find the nonindexed references.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Cumulative search results of two or more databases improve coverage and recall and decrease the chances of making an inappropriate conclusion in a review.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 6. Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
- Rests on an assumption: The 60 Cochrane reviews and the references they included are representative of systematic reviews and their included studies more generally.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this does not account for updated reviews, where some references might have been added after the publication of the original review and then picked up by the update searches.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 26
- The quote only partly states this claim. 2
- Probably not a substantial claim. 15
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here, searching 2 databases reached O95% coverage and 87.9% recall.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Depending on which approach was used (single database search, two databases, or combination of all three), there were always three to seven Cochrane reviews where a conclusion was no longer possible (median recall ranged from 20.0% to 53.8%) and one to three reviews where the opposite conclusion was drawn (median recall ranged from 52.1% to 78.7%).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.5. Recall in relation to reviews' conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Median recall per review ranged between 62.4% and 100% (Table 4).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.5. Recall in relation to reviews' conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The combination of databases increased the median coverage.The highest coverage could be attained with M þ E þ C (Median 98.5%; IQR 91.9%-100.0%)(Table 2).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.2. Coverage
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The highest median single-database recall was in MEDLINE (median: 75.5%, IQR: 56.8%-87.0%,Table 2) which seemed balanced between reviews of pharmacological and nonpharmacological interventions.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.3. Recall
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the remaining 235 individual, not found, indexed references, the majority had an abstract (70%), and was written in English (99%).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.6. Characteristics of references not found
plainn = 235- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the 1,969 references, 1,706 were journal articles and 263 grey literature articles of which 238 (14%) and 90 (34%) were not found, respectively.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.5. Recall in relation to reviews' conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The median coverage in CENTRAL and EMBASE was higher for reviews on pharmacological than nonpharmacological interventions although this was the opposite for MEDLINE (Table 2).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.2. Coverage
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
- Rests on an assumption: The intervention type classification (pharmacological vs nonpharmacological) is valid and meaningful.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In comparison to the references that could be found with at least one of the limited database search approaches, the references that were not found were on average older (28% vs. 17% published before 1991) and more often had no abstract available (30% vs. 11%) (Appendix 5).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.6. Characteristics of references not found
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
None of the sources cited for this claim have been checked yet.
In the paper
“When limiting the information sources to M þ E þ C, a conclusion was no longer possible or changed to the opposite of the original review in three [14e16] and one [17] reviews, respectively.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.4. Coverage in relation to reviews' conclusions
What the sources say
- Not checked yetcites 14e16
- Not checked yetcites [17] Blessberger, 2014, Perioperative beta-blockers for preventing surgery-related mortality and morbidity, Cochrane Database Syst Rev
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Three of these four reviews included at least one reference that was not indexed (overall 11 of 111, 10%), but coverage differed greatly between individual reviews (54%-100%; Appendix 5).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.4. Coverage in relation to reviews' conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The medical topics were mainly cardiovascular diseases (74%) and the intervention types were nutrition or physical activity (62%) (Table 5).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.6. Characteristics of references not found
plainn = 235- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The 90/263 grey literature references that could not be found when limiting the information sources to M þ E þ C and the conclusion remained unchanged were 15/60 study register entries (25%), 52/177 conference abstracts (29%), 11/14 theses (79%), 3/3 books (100%), 4/4 pharma documents (100%), and 5/5 other document types (100%).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 4.5. Recall in relation to reviews' conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
None of the sources cited for this claim have been checked yet.
In the paper
“In two instances [14,25], all references that were indexed were also found (coverage 5 recall), but the conclusion was still not possible.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
What the sources say
- Not checked yetcites [14] Geretsegger, 2014, Music therapy for people with autism spectrum disorder. Cochrane Database of Systematic Reviews, Cochrane Database Syst Rev
- Not checked yetcites [25] Wong, 2015, Blood pressure lowering efficacy of dual alpha and beta blockers for primary hypertension, Cochrane Database Syst Rev
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that references of Cochrane reviews where the conclusion was no longer possible tended to have lower median database coverage than the others, although Cochrane reviews where the conclusion and its certainty remained the same had a markedly higher median coverage than those where changes occurred.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Yet, even the combination of the three most frequently used databases Medline, Embase, and CENTRAL did not suffice to achieve total recall or avoid any change in the conclusions of systematic reviews.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 6. Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
- Rests on an assumption: The 60 Cochrane reviews and the references they included are representative of systematic reviews and their included studies more generally.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As expected, searching in more databases improved recall, but not all indexed references could be found.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In comparison, median coverage of a single database ranged between 83% and 91%, and median recall between 71% and 76%.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
None of the sources cited for this claim have been checked yet.
In the paper
“Removing studies that were not indexed in MEDLINE from the meta-analyses had only little impact on the effect estimates [3,4,6].”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 1. Introduction
What the sources say
- Not checked yetcites [3] Hartling, 2016, The contribution of databases to the results of systematic reviews: a cross-sectional study, BMC Med Res Methodol source
- Not checked yetcites [4] Halladay, 2015, Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions, J Clin Epidemiol source
- Not checked yetcites [6] Marshall, 2019, Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study, J Clin Epidemiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Recent methods studies evaluating research studies of medical or psychological topics [3e5] found that the majority of relevant references included in Cochrane reviews were available (indexed) in MEDLINE (i.e., database coverage).”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 1. Introduction
What the sources say
- Not checked yetcites 3e5
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Bramer et al. [8] assessed systematic search results in various medical topics, from therapeutic effectiveness and diagnostic accuracy to ethics and public health.They found high database coverage in MEDLINE and Embase, but reported that in some cases even the cumulative recall of MEDLINE and Embase was below 50%.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 1. Introduction
What the sources say
- Not checked yetcites [8] Bramer, 2016, Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study, Syst Rev source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Searching at least two databases improves coverage and recall and decreases the chances of making an inappropriate conclusion in a review.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > Key findings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: A change in a Cochrane review's conclusion when searching fewer databases indicates an inappropriate conclusion.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Hence, we only checked the search strategies for formal errors (i.e., syntax, spelling, block building), not for comprehensiveness of search terms or overall quality.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
None of the sources cited for this claim have been checked yet.
In the paper
“Third, most Cochrane reviews' search strategies are designed by trained information specialists who follow the Cochrane handbook [1] and MECIR standards [2].”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
What the sources say
- Not checked yetcites [1] Higgins
- Not checked yetcites [2] Higgins, 2022
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The overall database coverage and recall observed in our study are similar to findings of previous research [7e9].”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
What the sources say
- Not checked yetcites 7e9
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It may not always be necessary to find all the relevant references do draw a ''correct'' conclusion.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We sought to counter this issue by verifying that the references were available before the date of the latest search.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“First, the relationship between recall and conclusions is not straightforward.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, when determining if records were indexed in the databases, we limited results to MEDLINE-only and Embase-only, to ensure the results were independent of the search interface.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Although a low recall could relate to suboptimal searches [32], it is also possible that a good quality search was designed to be specific and hence, a lower recall would have been a deliberate choice.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
What the sources say
- Not checked yetcites [32] Franco, 2018, Identification of problems in search strategies in Cochrane Reviews, Res Synth Methods source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As most databases only allow searching in the metadata of a study (especially in title, abstract, author keywords and subject headings) rather than the full text, a reference without an abstract is harder to find and more dependent on the use of subject headings such as MeSH and Emtree.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we calculated the recall independently of the database coverage.This means that even a search strategy that found all indexed references would have a low recall if the review contained many nonindexed references.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this might differ from the actual search experience as some interfaces index additional documents, e.g., PubMed includes MEDLINE-in-Process materials and nonMEDLINE records, and Embase.comincludes most MeSH-indexed MEDLINE records.Hence, we may have effectively underestimated the coverage and recall of searching in a limited number of databases.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, by using references from Cochrane reviews, we might overestimate the database coverage of CENTRAL because references to RCTs included in Cochrane reviews are regularly added to this database.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As Cochrane reviews list all included study publications, it is possible that references not found by the searches belonged to studies included in the narrative syntheses, not the metaanalyses.In that case, they affected the recall of the searches but not the conclusions of the review.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.2. Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although searching two databases may be enough for some systematic reviews, others may require a more thorough approach like MECIR.”
§ Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study > 5.1. Overall results
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
The paper relies on this without support.
The references included in the 60 Cochrane reviews (the gold standard-set) represent all relevant studies for those reviews.
“From each review, we used all references reported in the ''References to studies included in this review'' section as a gold standard-set to assess 14 different search approaches using a limited number of sources (i.e., the 14 possible combinations of MEDLINE, Embase, CENTRAL with and without screening reference lists).”
“For this sub-study, we used the same gold standard-set as mentioned above.”
The paper relies on this without support.
The 60 randomly selected Cochrane reviews are representative of Cochrane reviews generally.
“We randomly selected 60 Cochrane reviews that included a replicable search strategy, had clear conclusions, and reported a meta-analysis [12].”
The paper relies on this without support.
A change in a Cochrane review's conclusion when searching fewer databases indicates an inappropriate conclusion.
“Searching at least two databases improves coverage and recall and decreases the chances of making an inappropriate conclusion in a review.”
The paper relies on this without support.
The database indexation status at the time of the original search can be accurately determined using database-specific identifiers.
“For each reference, we noted the availability in the target databases using the database-specific identifiers.We used these identifiers to determine if available studies were found by the original search strategies.”
The paper relies on this without support.
The intervention type classification (pharmacological vs nonpharmacological) is valid and meaningful.
“We determined the clinical topic of each review and categorized all reviews according to the type of intervention (pharmacological or nonpharmacological).”
The paper relies on this without support.
The original conclusions of the Cochrane reviews are a valid benchmark for assessing whether limited searches change conclusions.
“Then, we asked the original authors whether the resulting summary of findings tables would alter their conclusions.”
The authors state this as an assumption.
The search strategies of the included Cochrane reviews are comprehensive and of high quality, so that the references they include can serve as a gold standard.
“We assumed that these would be comprehensive and of high quality to serve as gold-standard [30].”
The paper relies on this without support.
The 60 Cochrane reviews and the references they included are representative of systematic reviews and their included studies more generally.
“Characteristics of references of studies included by the 60 Cochrane reviews”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Although these other guides are also evidence informed and we encourage their use, our guide is uniquely grounded in empirical evidence evaluating methods of rapid reviews, 12 further enriched by expert input [13][14][15][16] and feedback from a formal evaluation of its real world use.7”
From Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness (2024) · cites this paper as [12]
Citations
Cites
PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement
McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.01.021 · Cited by 5,700
Bramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0644-y · Cited by 2,114
A systematic approach to searching: an efficient and complete method to develop literature searches
Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018) · Journal of the Medical Library Association · doi:10.5195/jmla.2018.283 · Cited by 863
Reduction in saturated fat intake for cardiovascular disease
Hooper, L., Martin, N., Abdelhamid, A., & Davey Smith, G. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011737 · Cited by 704
Searching for studies: a guide to information retrieval for Campbell systematic reviews
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. K., Hammerstrøm, K., & Sathe, N. (2017) · Campbell Systematic Reviews · doi:10.4073/cmg.2016.1 · Cited by 371
The Challenges of Searching for and Retrieving Qualitative Studies
Barroso, J., Gollop, C. J., Sandelowski, M., Meynell, J., Pearce, P. F., & Collins, L. J. (2003) · Western Journal of Nursing Research · doi:10.1177/0193945902250034 · Cited by 272
Rivastigmine for Alzheimer's disease
Birks, J., & Evans, J. G. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001191.pub3 · Cited by 247
Bramer, W. M., Giustini, D., & Kramer, B. M. R. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0215-7 · Cited by 202
Effects of total fat intake on body weight
Hooper, L., Abdelhamid, A. S., Bunn, D. K., Brown, T. J., Summerbell, C., & Skeaff, C. M. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011834 · Cited by 170
Halladay, C. W., Trikalinos, T. A., Schmid, I. T., Schmid, C. H., & Dahabreh, I. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.12.017 · Cited by 166
Research works printed but not linked: 1
- Geretsegger, 2014, Music therapy for people with autism spectrum disorder. Cochrane Database of Systematic Reviews, Cochrane Database Syst Rev, 2014, CD004381
Cited by
Sorted by citations
Garritty, C., Hamel, C., Trivella, M., Gartlehner, G., Nussbaumer-Streit, B., Devane, D., Kamel, C., Griebler, U., & King, V. J. (2024) · BMJ · doi:10.1136/bmj-2023-076335 · Cited by 260
Self-citationEeuwijk, J., Ferreira, G. S., Yarzabal, J., & Holle, M. R.-D. R. V. B. (2024) · Infectious Disease and Therapy · doi:10.1007/s40121-024-00919-0 · Cited by 52
Affengruber, L., van der Maten, M. M., Spiero, I., Nußbaumer-Streit, B., Mahmić-Kaknjo, M., Ellen, M., Gooßen, K., Kantorová, L., Hooft, L., Riva, N., et al. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02320-4 · Cited by 63
Self-citationMarengo, A., & Pagano, A. (2023) · Electronics · doi:10.3390/electronics12143006 · Cited by 52
Iyngkaran, P., Appuhamilage, P. Y., Patabandige, G., Kandage, P. S. S. P., Usmani, W., & Hanna, F. S. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21030339 · Cited by 51
Metadata
- Authors
- Ewald, H., Klerings, I., Wagner, G., Heise, T. L., Stratil, J. M., Lhachimi, S. K., Hemkens, L. G., Gartlehner, G., Armijo-Olivo, S., & Nussbaumer-Streit, B.
- Affiliations
- University of Basel (CH)
- Universität für Weiterbildung Krems (AT)
- University of Bremen (DE)
- Ludwig-Maximilians-Universität München (DE)
- University of Alberta (CA)
- Corresponding author
- Ewald, Hannah
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2022-09
- Volume
- 149
- Pages
- 154-164
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W4281701339
- Semantic Scholar
- bfbec399290bbad3b129739c10adc93ca968aad8
- Cited by
- 168
- References
- 32
- Citations by year
- 2026: 61 · 2025: 54 · 2024: 44 · 2023: 9
- FWCI
- 16.15
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Open access LMU (Ludwid Maxmilian's Universitat Munchen) (submittedVersion, cc-by-nc-nd)
- Informationszentrum Lebenswissenschaften (ZB MED) (submittedVersion, cc-by-nc-nd)
- Library Osnabrück University (Hochschule Osnabrück) (submittedVersion, cc-by-nc-nd)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- database searching, search recall, missing studies, bibliographic database indexing, MEDLINE, Embase, Central
- MeSH terms
- Abstracting and Indexing, Humans, Databases, Bibliographic, Databases, Factual, MEDLINE, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
