A Comparison of the Quality of Cochrane Reviews and Systematic Reviews Published in Paper-Based Journals
The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
This study set out to compare Cochrane reviews and reviews published in paper-based journals. Two assessment tools were used to collect the data, a 23-itemchecklist developed by Sacks and a nine-itemscale developed by Oxman. Cochrane reviews were found to be better at reporting some items and paper-based reviews at reporting others. The overall quality was found to be low. This represents a serious situation because clinicians, health policy makers, and consumers are often told that systematic reviews represent “the best available evidence.” In the period since this study, the Cochrane Collaboration has taken steps to improve the quality of its reviews through, for example, more thorough prepublication refereeing, developments in the training and support offered to reviewers, and improvements in the system for postpublication peer review. In addition, the use of evidence-based criteria (i.e., the QUOROM statement) for reporting systematic reviews may help further to improve their quality.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bias in treatment assignment in controlled clinical trials
Chalmers, T. C., Celano, P., Sacks, H. S., & Smith, H. (1983) · New England Journal of Medicine · doi:10.1056/nejm198312013092204 · Cited by 864
Jadad, A. R., Cook, D. J., Jones, A., Klassen, T. P., Tugwell, P., Moher, M., & Moher, D. (1998) · JAMA · doi:10.1001/jama.280.3.278 · Cited by 534
Self-citationDoes blinding of readers affect the results of meta-analyses?
Berlin, J. A. (1997) · The Lancet · doi:10.1016/s0140-6736(05)62352-5 · Cited by 478
What contributions do languages other than English make on the results of meta-analyses?
Moher, D., Pham, Klassen, T. P., Schulz, K. F., Berlin, J. A., Jadad, A. R., & Liberati, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00188-8 · Cited by 453
Self-citationAssessing the Quality of Randomized Controlled Trials:Current Issues and Future Directions
Moher, D., Jadad, A. R., & Tugwell, P. (1996) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300009570 · Cited by 436 · cited 1×
“The systematic reviews from paper-based journals came from a database of known systematic reviews of randomized trials developed by Moher et al. (1999). This database had been compiled using an extensive search strategy and an electronic search of MEDLINE (OVID) for the period from January 1, 1966, to December 31, 199…”
MethodSelf-citationQuality of Cochrane reviews: Assessment of sample from 1998.
Olsen, O., Middleton, P., Ezzo, J., Gøtzsche, P. C., Hadhazy, V., Herxheimer, A., Kleijnen, J., & McIntosh, H. (2001) · BMJ · doi:10.1136/bmj.323.7317.829 · Cited by 159 · cited 1×
“Olsen et al. (2001) studied the 53 Cochrane reviews that were published for the first time at the end of 1998.”
Background
Research works printed but not linked: 18
- 10.1001/jama.1995.03530240052041 · DOI · Paper page
- Clarke, M., 2001, The Cochrane Library
- DARE, 2001, The Cochrane Library
- 10.1016/0895-4356(95)00062-3 · DOI · Paper page
- 10.1136/bmj.320.7234.537 · DOI · Paper page
- 10.1200/JCO.1986.4.6.942 · DOI · Paper page
- Moher, D., 1997, Non-random reflections on health services research, 250
- 10.1016/S0140-6736(99)04149-5 · DOI · Paper page
- 10.1001/jama.1995.03530240072044 · doi:10.1001/jama.1995.03530240072044 · Paper page
- 10.1136/bmj.309.6955.648 · DOI · Paper page
- 10.1016/0895-4356(91)90160-B · DOI · Paper page
- 10.1016/0895-4356(91)90205-N · DOI · Paper page
- 10.1056/NEJM198702193160806 · DOI · Paper page
- Sacks, H. S., 1996, Mount Sinai Journal of Medicine, 63, 216
- Shea, B. (1999). Assessing the quality of reporting meta-analyses of randomized controlled trials . Unpublished thesis, University of Ottawa.
- Shea, B., 2001, Systematic reviews, 122, 10.1002/9780470693926.ch7 · DOI · Paper page
- 10.1093/fampra/10.3.337 · DOI
- 10.1016/S0883-0355(96)80001-2 · DOI
Cited by
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133
“Cochrane systematic reviews are internationally recognised as among the best sources, if not the best source, of reliable up to date information on health care.”
BackgroundThe appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052
Epidemiology and reporting characteristics of systematic reviews.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040078 · Cited by 891
“Cochrane and non-Cochrane reviews provides the most discouraging results and suggests little improvement in the quality of reporting of non-Cochrane reviews over time [4, 7 ].”
BackgroundResultSelf-citationPanic, N., Leoncini, E., de Belvis, G., Ricciardi, W., & Boccia, S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0083138 · Cited by 883
“Further, although the PRISMA statement was primarily developed to improve the quality of reporting of systematic reviews and meta-analyses [7], we assumed that improving the quality of reporting can also influence their ideation and conduction.”
Method
Metadata
- Authors
- Shea, B., Moher, D., Graham, I., Pham, B., & Tugwell, P.
- Venue
- Evaluation & the Health Professions
- Publisher
- SAGE Publications
- Published
- 2002-03
- Volume
- 25
- Issue
- 1
- Pages
- 116-129
- ISSN
- 0163-2787 (ISSN-L)
- 1552-3918
- OpenAlex
- W2133192384
- Cited by
- 110
- References
- 6
- FWCI
- 2.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Comparison of the Quality of Cochrane Reviews and Systematic Reviews Published in Paper-Based Journals. https://app.cochat.ai/papers/10.1177/0163278702025001008
- The paper
- Shea, B., Moher, D., Graham, I., Pham, B., & Tugwell, P. (2002). A Comparison of the Quality of Cochrane Reviews and Systematic Reviews Published in Paper-Based Journals. Evaluation & the Health Professions. https://doi.org/10.1177/0163278702025001008
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.