Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies
Open accessAbstract
BACKGROUND: Assessing the risk of bias in individual studies in a systematic review can be done using individual components or by summarizing the study quality in an overall score. METHODS: We examined the instructions to authors of the 50 Cochrane Review Groups that focus on clinical interventions for recommendations on methodological quality assessment of studies. RESULTS: Forty-one of the review groups (82%) recommended quality assessment using components and nine using a scale. All groups recommending components recommended to assess concealment of allocation, compared to only two of the groups recommending scales (P < 0.0001). Thirty-five groups (70%) recommended assessment of sequence generation and 21 groups (42%) recommended assessment of intention-to-treat analysis. Only 28 groups (56%) had specific recommendations for using the quality assessment of studies analytically in reviews, with sensitivity analysis, quality as an inclusion threshold and subgroup analysis being the most commonly recommended methods. The scales recommended had problems in the individual items and some of the groups recommending components recommended items not related to bias in their quality assessment. CONCLUSION: We found that recommendations by some groups were not based on empirical evidence and many groups had no recommendations on how to use the quality assessment in reviews. We suggest that all Cochrane Review Groups refer to the Cochrane Handbook for Systematic Reviews of Interventions, which is evidence-based, in their instructions to authors and that their own guidelines are kept to a minimum and describe only how methodological topics that are specific to their fields should be handled.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Risk of bias for the 17 selected studies was assessed by 2 teams of reviewers using the Cochrane Collaboration tool, and discrepancies were discussed between the teams.
“Risk of bias for the 17 selected studies was assessed by the 2 teams of reviewers using the Cochrane Collaboration tool [41-43], and any discrepancies between reviewers were discussed between the 2 teams.”
From Tailored web-based interventions for pain: Systematic review and meta-analysis (2017) · cites this paper as [41]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
(2014) · Medical Entomology and Zoology · Cited by 17,633 · see citing works
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582 · see citing works
Writing Group for the Women's Health Initiative Investigators (2002) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.288.3.321 · Cited by 15,889 · see citing works
Zhang, Q., Wang, C., Zheng, M., Li, Y., Li, J., Zhang, L., Shang, X., & Yan, C. (2014) · Cerebrovascular Diseases · doi:10.1159/000369778 · Cited by 11,609 · see citing works
Grading quality of evidence and strength of recommendations
(2004) · BMJ · doi:10.1136/bmj.328.7454.1490 · Cited by 8,477 · see citing works
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · see citing works
Dong, Y., Hao, L., Shi, Z., Wang, G., Zhang, Z., & Han, C. (2013) · Urologia Internationalis · doi:10.1159/000351405 · Cited by 4,556 · see citing works
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560 · see citing works
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · see citing works
Gluud, L. L., Villumsen, J., & Gluud, C. N. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,262 · see citing works
Updated Method Guidelines for Systematic Reviews in the Cochrane Collaboration Back Review Group
van Tulder, M., Furlan, A., Bombardier, C., & Bouter, L. (2003) · Spine · doi:10.1097/01.brs.0000065484.95996.af · Cited by 2,180 · see citing works
Charbonneau, D. H. (2005) · Europe PMC (PubMed Central) · Cited by 2,041 · see citing works
Postmenopausal Estrogen Therapy and Cardiovascular Disease
Stampfer, M. J., Colditz, G. A., Willett, W. C., Manson, J. E., ROSNER, B. A., Speizer, F. E., & Hennekens, C. H. (1991) · New England Journal of Medicine · doi:10.1056/nejm199109123251102 · Cited by 2,000 · see citing works
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · see citing works
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · see citing works
Self-citationStampfer, M. J., & Colditz, G. A. (1991) · Preventive Medicine · doi:10.1016/0091-7435(91)90006-p · Cited by 1,611 · see citing works
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308 · see citing works
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181 · see citing works
Clinical Epidemiology: The Architecture of Clinical Research.
(1985) · Annals of Internal Medicine · doi:10.7326/0003-4819-103-4-649_2 · Cited by 954 · see citing works
Postmenopausal Estrogen Therapy and Cardiovascular Disease
MacLean, D. B., & Bostom, A. G. (1992) · New England Journal of Medicine · doi:10.1056/nejm199203053261012 · Cited by 703 · see citing works
Assessing the Quality of Randomized Trials
Clark, H. D., Wells, G. A., Huët, C., McAlister, F. A., Salmi, L., Fergusson, D. A., & Laupacis, A. (1999) · Controlled Clinical Trials · doi:10.1016/s0197-2456(99)00026-4 · Cited by 649 · see citing works
UPTON, G. J. G. (1992) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2982890 · Cited by 636 · see citing works
Melander, H., Ahlqvist-Rastad, J., Meijer, G., & Beermann, B. (2003) · BMJ · doi:10.1136/bmj.326.7400.1171 · Cited by 626 · see citing works
Jadad, A. R., Cook, D. J., Jones, A., Klassen, T. P., Tugwell, P., Moher, M., & Moher, D. (1998) · JAMA · doi:10.1001/jama.280.3.278 · Cited by 534 · see citing works
Schulz, K. F. (1994) · JAMA · doi:10.1001/jama.1994.03520020051014 · Cited by 514 · see citing works
Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal, J., Hrobjartsson, A., Jorgensen, K., Hilden, J., Altman, D., & Gotzsche, P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym087 · Cited by 472 · see citing works
Randomisation to protect against selection bias in healthcare trials
Odgaard-Jensen, J., Vist, G. E., Timmer, A., Kunz, R., Akl, E. A., Schünemann, H., Briel, M., Nordmann, A. J., Pregno, S., & Oxman, A. D. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000012.pub3 · Cited by 439 · see citing works
Jørgensen, A. W., Hilden, J., & Gøtzsche, P. C. (2006) · BMJ · doi:10.1136/bmj.38973.444699.0b · Cited by 439 · see citing works
Self-citationAssessing the quality of research.
Glasziou, P., Vandenbroucke, J., & Chalmers, I. (2004) · BMJ · doi:10.1136/bmj.328.7430.39 · Cited by 391 · see citing works
Moher, D., Cook, Jadad, Tugwell, P., Moher, Jones, Pham, & Klassen (1999) · Health Technology Assessment · doi:10.3310/hta3120 · Cited by 364 · see citing works
Gøtzsche, P. C. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90017-2 · Cited by 339 · see citing works
Self-citationComparability of Randomised Groups
Altman, D. G. (1985) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2987510 · Cited by 329 · see citing works
Uses and abuses of meta-analysis
Egger, M., Smith, G. D., & Sterne, J. A. C. (2001) · Clinical Medicine · doi:10.7861/clinmedicine.1-6-478 · Cited by 258 · see citing works
Emerson, J. D., Burdick, E., Hoaglin, D. C., Mosteller, F. C., & Chalmers, T. C. (1990) · Controlled Clinical Trials · doi:10.1016/0197-2456(90)90175-2 · Cited by 241 · see citing works
Moja, L., Telaro, E., D’Amico, R., Moschetti, I., Coe, L., & Liberati, A. (2005) · BMJ · doi:10.1136/bmj.38414.515938.8f · Cited by 238 · see citing works
Intention to treat – who should use ITT?
Lewis, J., & Machin, D. (1993) · British Journal of Cancer · doi:10.1038/bjc.1993.402 · Cited by 236 · see citing works
Investigating patient exclusion bias in meta-analysis
Tierney, J. F. (2004) · International Journal of Epidemiology · doi:10.1093/ije/dyh300 · Cited by 221 · see citing works
(2003) · Wiley eBooks · doi:10.1002/9781444315844 · Cited by 205 · see citing works
Randomisation to protect against selection bias in healthcare trials
Kunz, R., Vist, G. E., & Oxman, A. D. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000012.pub2 · Cited by 188 · see citing works
Development of a quality assessment instrument for trials of treatments for depression and neurosis
Moncrieff, J., Churchill, R., Drummond, D. C., & McGuire, H. (2001) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.108 · Cited by 185 · see citing works
Greenland, S. (1994) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117250 · Cited by 150 · see citing works
Somatostatin analogues for acute bleeding oesophageal varices
Gøtzsche, P. C., & Hróbjartsson, A. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000193.pub3 · Cited by 113 · see citing works
Self-citationSchulz, K. F. (1994) · JAMA · doi:10.1001/jama.272.2.125 · Cited by 93 · see citing works
Bhandari, M., Richards, R. R., Sprague, S., & Schemitsch, E. H. (2001) · Controlled Clinical Trials · doi:10.1016/s0197-2456(01)00147-7 · Cited by 34 · see citing works
Somatostatin analogues for acute bleeding oesophageal varices
Pc, G., & Hróbjartsson, A. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000193.pub2 · Cited by 22 · see citing works
Systematic reviews evaluating effects of health care interventions : issues of synthesis and bias
Deeks, J. J. (2007) · UvA-DARE (University of Amsterdam) · Cited by 1 · see citing works
[Selective reporting--greater problem than selective publishing?].
Melander, H. (2005) · PubMed · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Cochrane Review Groups (CRGs) (Accessed March 26, 2007).
Cited by
Sorted by citations
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
Industry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697 · see citing works
Self-citationArmijo‐Olivo, S., Stiles, C. R., Hagen, N. A., Biondo, P. D., & Cummings, G. G. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2010.01516.x · Cited by 1,539 · see citing works
Smoking cessation reduces postoperative complications: A systematic review and meta-analysis
Mills, E., Eyawo, O., Lockhart, I., Kelly, S., Wu, P., & Ebbert, J. O. (2011) · The American Journal of Medicine · doi:10.1016/j.amjmed.2010.09.013 · Cited by 599 · see citing works
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486 · see citing works
Savović, J., Weeks, L., Sterne, J. A., Turner, L., Altman, D. G., Moher, D., & Higgins, J. P. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-37 · Cited by 340 · see citing works
Page, M. J., McKenzie, J. E., & Higgins, J. P. T. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019703 · Cited by 278 · see citing works
Lim, S. M., Shiau, C. W. C., Cheng, L. J., & Lau, Y. (2022) · Behavior Therapy · doi:10.1016/j.beth.2021.09.007 · Cited by 163 · see citing works
Armijo-Olivo, S., Ospina, M., da Costa, B. R., Egger, M., Saltaji, H., Fuentes, J., Ha, C., & Cummings, G. G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0096920 · Cited by 124 · see citing works
A proposed method of bias adjustment for meta-analyses of published observational studies
Thompson, S., Ekelund, U., Jebb, S., Lindroos, A. K., Mander, A., Sharp, S., Turner, R., & Wilks, D. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyq248 · Cited by 101 · see citing works
Tailored web-based interventions for pain: Systematic review and meta-analysis
Martorella, G., Boitor, M., Berube, M., Fredericks, S., Le May, S., & Gélinas, C. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8826 · Cited by 88 · see citing works
Haag, G., Diener, H.-C., May, A., Meyer, C., Morck, H., Straube, A., Wessely, P., & Evers, S. (2011) · The Journal of Headache and Pain · doi:10.1007/s10194-010-0266-4 · Cited by 78 · see citing works
Hopewell, S., Boutron, I., Altman, D. G., & Ravaud, P. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003342 · Cited by 66 · see citing works
Campos, B., Vinder, V., Passos, R. B. F., Coutinho, E. S. F., Vieira, N. C. P., Leal, K. B., Mendlowicz, M. V., Figueira, I., Luz, M. P., Marques-Portela, C., et al. (2022) · Journal of Psychopharmacology · doi:10.1177/02698811221080464 · Cited by 31 · see citing works
A systematic review of the evidence for hyporesponsivity in ASD
Watts, S. J., Rodgers, J., & Riby, D. (2016) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-016-0084-y · Cited by 21 · see citing works
Heidari‐Soureshjani, R., Nasrabadi, A. N., Zakerimoghadam, M., Mohammadi, T., & Rasti, A. (2023) · Health Science Reports · doi:10.1002/hsr2.1536 · Cited by 10 · see citing works
The effect of tailored Web-based interventions on pain in adults: a systematic review protocol
Martorella, G., Gélinas, C., Bérubé, M., Boitor, M., Fredericks, S., & LeMay, S. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0233-5 · Cited by 5 · see citing works
- Authors
- Lundh, A., & Gøtzsche, P. C.
- Affiliations
- Rigshospitalet (DK)
- Corresponding author
- Gøtzsche, Peter C
- Venue
- BMC Medical Research Methodology
- Published
- 2008-04-21
- Volume
- 8
- Issue
- 1
- Pages
- 22-22
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC2375895
- OpenAlex
- W2019660808
- Cited by
- 352
- References
- 48
- Citations by year
- 2026: 10 · 2025: 34 · 2024: 34 · 2023: 34 · 2022: 25 · 2021: 21 · 2020: 17 · 2019: 24 · 2018: 19 · 2017: 21 · 2016: 22 · 2015: 15 · 2014: 23 · 2013: 15 · 2012: 9
- FWCI
- 10.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Funders
- Rigshospitalet
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- risk of bias assessment, methodological quality assessment, Cochrane Review Groups, author guidelines, quality assessment scale, sequence generation, intention-to-treat analysis, sensitivity analysis, subgroup analysis
- MeSH terms
- Clinical Trials as Topic, Quality Control, Research Design, Review Literature as Topic, Meta-Analysis as Topic, Reproducibility of Results, Bias, Evidence-Based Medicine, Advisory Committees
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
