Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
(1963) · Medical Entomology and Zoology · Cited by 15,085
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
Items 101–200 of 1,024
Combined oral contraceptives: venous thrombosis
de Bastos, M., Stegeman, B. H., Rosendaal, F. R., Van Hylckama Vlieg, A., Helmerhorst, F. M., Stijnen, T., & Dekkers, O. M. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010813.pub2 · Cited by 399
Lo, J., Chan, L., & Flynn, S. (2021) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2020.04.001 · Cited by 403
Lee, C., Gong, Y., Brok, J., Boxall, E. H., & Gluud, C. (2006) · BMJ · doi:10.1136/bmj.38719.435833.7c · Cited by 402
Outcomes of maternal weight gain.
Viswanathan, M., Siega‐Riz, A. M., Moos, M. K., Deierlein, A. L., Mumford, S. L., Knaack, J., Thieda, P., Lux, L. J., & Lohr, K. N. (2008) · PubMed · Cited by 500
Zhang, H., Gao, Y., Jiang, F., Fu, M., Yuan, Y., Guo, Y., Zhu, Z., Lin, M., Liu, Q., Tian, Z., et al. (2015) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.14792 · Cited by 389
Determinants of health-related quality of life in Parkinson’s disease: A systematic review
Soh, S.-E., Morris, M. E., & McGinley, J. L. (2011) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2010.08.012 · Cited by 386
“Quality extraction tools from other reviews on similar topics [3,4,7] and literature on developing quality criteria [8,9] were also examined to identify relevant quality themes.”
BackgroundDimberg, L., Arnrup, K., & Bondemark, L. (2015) · The European Journal of Orthodontics · doi:10.1093/ejo/cju046 · Cited by 375
Mackenbach, J. D., Rutter, H., Compernolle, S., Glonti, K., Oppert, J.-M., Charreire, H., De Bourdeaudhuij, I., Brug, J., Nijpels, G., & Lakerveld, J. (2014) · BMC Public Health · doi:10.1186/1471-2458-14-233 · Cited by 381
“Assessment of the methodological quality For quality assessment, we adapted the Quality Assessment Tool for Quantitative Studies (as developed by the Effective Public Health Practice Project),[23] based on recommendations from a number of authors.[24-27] This contains 19 items in eight key domains for assessment of st…”
MethodKagan, K. O., Wright, D., Baker, A., Sahota, D., & Nicolaides, K. H. (2008) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.5331 · Cited by 378
Pereira, C., & Castilho, E. (2009) · American Journal of Epidemiology · doi:10.1093/aje/kwp016 · Cited by 375
Etiology of molar incisor hypomineralization – A systematic review
Silva, M. J., Scurrah, K. J., Craig, J. M., Manton, D. J., & Kilpatrick, N. (2016) · Community Dentistry and Oral Epidemiology · doi:10.1111/cdoe.12229 · Cited by 366
Samoocha, D., Bruinvels, D. J., Elbers, N. A., Anema, J. R., & van der Beek, A. J. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1286 · Cited by 374
Ospina, M. B., Krebs Seida, J., Clark, B., Karkhaneh, M., Hartling, L., Tjosvold, L., Vandermeer, B., & Smith, V. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003755 · Cited by 366
“There is some evidence that results of RCTs and non-RCTs sometimes, but not always, differ,[130] and that non-RCT can be more prone to bias and overestimate treatment effects [131,132].”
BackgroundThe prevention and treatment of complicated grief: a meta-analysis
Wittouck, C., Van Autreve, S., De Jaegere, E., Portzky, G., & van Heeringen, K. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2010.09.005 · Cited by 361
Gartlehner, G., Hansen, R. A., Morgan, L. C., Thaler, K., Lux, L., Van Noord, M., Mager, U., Thieda, P., Gaynes, B. N., Wilkins, T., et al. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-11-201112060-00009 · Cited by 355
Fletcher, B., Gheorghe, A., Moore, D., Wilson, S., & Damery, S. (2012) · BMJ Open · doi:10.1136/bmjopen-2011-000496 · Cited by 355
“This instrument was chosen as it enables different study designs to be assessed using the same tool and was identified as one of only six judged to be suitable for systematic reviews assessing multiple study designs.(18) Using the EPHPP tool, studies were assessed against six criteria: selection bias, design, confound…”
MethodQuansah, R., Armah, F. A., Essumang, D. K., Luginaah, I., Clarke, E., Marfoh, K., Cobbina, S. J., Nketiah-Amponsah, E., Namujju, P. B., Obiri, S., et al. (2015) · Environmental Health Perspectives · doi:10.1289/ehp.1307894 · Cited by 349
“Finally, some items of the Newcastle–Ottawa Scale such as representativeness of study cohort with respect to community and duration of follow-up do not belong to the risk of bias tools (Deeks et al. 2003; National Research Council 2004; Sanderson et al. 2007).”
MethodIndications for Rotator Cuff Repair: A Systematic Review
Oh, L. S., Wolf, B. R., Hall, M. P., Levy, B. A., & Marx, R. G. (2007) · Clinical Orthopaedics and Related Research · doi:10.1097/blo.0b013e31802fc175 · Cited by 355
Tassone, F., Pan, R., Amiri, K., Taylor, A. K., & Hagerman, P. J. (2008) · The Journal of Molecular Diagnostics · doi:10.2353/jmoldx.2008.070073 · Cited by 350
Tang, B. M. P., Craig, J. C., Eslick, G. D., Seppelt, I., & McLean, A. S. (2009) · Critical Care Medicine · doi:10.1097/ccm.0b013e31819fb507 · Cited by 351
“Cohort studies were assessed using criteria based on the Health Technology Assessment Program guidelines, which provided evidence-based recommendations on the assessment of nonrandomized trials (18).”
MethodKoch, S., Kunz, T., Lykou, S., & Cruz, R. (2014) · The Arts in Psychotherapy · doi:10.1016/j.aip.2013.10.004 · Cited by 349
“On the other hand, Deeks et al. (2003) analyzed eight reviews and concluded that the effects”
BackgroundThe challenges faced in the design, conduct and analysis of surgical randomised controlled trials
Cook, J. A. (2009) · Trials · doi:10.1186/1745-6215-10-9 · Cited by 351
“Comparisons of randomised and non-randomised studies have shown that the results can be divergent, in direction as well as magnitude [10].”
ResultRepeat Transurethral Resection in Non–muscle-invasive Bladder Cancer: A Systematic Review
Cumberbatch, M. G. K., Foerster, B., Catto, J. W. F., Kamat, A. M., Kassouf, W., Jubber, I., Shariat, S. F., Sylvester, R. J., & Gontero, P. (2018) · European Urology · doi:10.1016/j.eururo.2018.02.014 · Cited by 343
Residents’ and Attending Physicians’ Handoffs: A Systematic Review of the Literature
Riesenberg, L. A., Leitzsch, J., Massucci, J. L., Jaeger, J., Rosenfeld, J. C., Patow, C., Padmore, J. S., & Karpovich, K. P. (2009) · Academic Medicine · doi:10.1097/acm.0b013e3181bf51a6 · Cited by 345
A meta‐analysis of the association of physical activity with reduced risk of colorectal cancer
Samad, A. K. A., Taylor, R. S., Marshall, T., & Chapman, M. A. S. (2005) · Colorectal Disease · doi:10.1111/j.1463-1318.2005.00747.x · Cited by 347
Laparoscopic versus open hepatic resections for benign and malignant neoplasms—a meta-analysis
Simillis, C., Constantinides, V. A., Tekkis, P. P., Darzi, A., Lovegrove, R., Jiao, L., & Antoniou, A. (2007) · Surgery · doi:10.1016/j.surg.2006.06.035 · Cited by 343
A glossary for evidence based public health
Rychetnik, L., Hawe, P., Waters, E., Barratt, A., & Frommer, M. (2004) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2003.011585 · Cited by 341
Self-Management Skills in Chronic Disease Management: What Role Does Health Literacy Have?
Mackey, L. M., Doody, C., Werner, E. L., & Fullen, B. (2016) · Medical Decision Making · doi:10.1177/0272989x16638330 · Cited by 339
The Association Between Chronic Low Back Pain and Sleep: A Systematic Review
Kelly, G. A., Blake, C., Power, C. K., O'Keeffe, D., & Fullen, B. M. (2011) · The Clinical Journal of Pain · doi:10.1097/ajp.0b013e3181f3bdd5 · Cited by 330
Knee osteoarthritis phenotypes and their relevance for outcomes: a systematic review
Deveza, L. A., Melo, L., Yamato, T. P., Mills, K., Ravi, V., & Hunter, D. J. (2017) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2017.08.009 · Cited by 331
Flexible working conditions and their effects on employee health and wellbeing
Joyce, K., Pabayo, R., Critchley, J. A., & Bambra, C. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008009.pub2 · Cited by 325
Cugelman, B., Thelwall, M., & Dawes, P. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1367 · Cited by 323
Roy, J.-S., Braën, C., Leblond, J., Desmeules, F., Dionne, C. E., MacDermid, J. C., Bureau, N. J., & Frémont, P. (2015) · British Journal of Sports Medicine · doi:10.1136/bjsports-2014-094148 · Cited by 319
Effectiveness of exercise-referral schemes to promote physical activity in adults: systematic review
Williams, N. H., Hendry, M., France, B., Lewis, R., & Wilkinson, C. (2007) · British Journal of General Practice · doi:10.3399/096016407782604866 · Cited by 321
Aboud, F. E., Tredoux, C., Tropp, L. R., Brown, C. S., Niens, U., & Noor, N. M. (2012) · Developmental Review · doi:10.1016/j.dr.2012.05.001 · Cited by 316
“The nine qualities of evaluation were taken from Deeks et al. (2003) and Downs and Black (1998) to addressinternalandexternalvalidity.”
A systematic review of the effectiveness of occupational health and safety training
Robson, L. S., Stephenson, C. M., Schulte, P. A., Amick, B. C. I., Irvin, E. L., Eggerth, D. E., Chan, S., Bielecky, A. R., Wang, A. M., Heidotting, T. L., et al. (2012) · Scandinavian Journal of Work, Environment & Health · doi:10.5271/sjweh.3259 · Cited by 296
“First, sixteen items, based on established instruments (28, 29) adapted to the training literature and refined through pretesting, were used to assess specific biases.”
MethodWhen and how can real world data analyses substitute for randomized controlled trials?
Franklin, J. M., & Schneeweiss, S. (2017) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.857 · Cited by 316
Bias Modelling in Evidence Synthesis
Turner, R. M., Spiegelhalter, D. J., Smith, G. C. S., & Thompson, S. G. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00547.x · Cited by 321
“It is clear that differences between populations will cause both internal and external bias with respect to the target question, but unclear how to determine the extent of each bias.”
BackgroundMethodWilcox, M. E., Chong, C. A. K. Y., Niven, D. J., Rubenfeld, G. D., Rowan, K. M., Wunsch, H., & Fan, E. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e318292313a · Cited by 316
Issues in the reporting of epidemiological studies: a survey of recent practice.
Pocock, S. J., Collier, T. J., Dandreo, K. J., de Stavola, B. L., Goldman, M. B., Kalish, L. A., Kasten, L. E., & McCormack, V. A. (2004) · BMJ · doi:10.1136/bmj.38250.571088.55 · Cited by 313
Screening for breast cancer with mammography
Gøtzsche, P. C., & Nielsen, M. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001877.pub4 · Cited by 311
Music therapy in dementia: a narrative synthesis systematic review
McDermott, O., Crellin, N., Ridder, H. M., & Orrell, M. (2013) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.3895 · Cited by 310
“Quality assessment of the included studies Two quality assessment tools were chosen on the basis of the favourable reviews in the ‘HTA Evaluating nonrandomised interventions’ (Deeks et al., 2003) and the Centre for Reviews and Dissemination’s guidance for systematic reviews (CRD, 2009).”
MethodRahbari, N. N., Aigner, M., Thorlund, K., Mollberg, N., Motschall, E., Jensen, K., Diener, M. K., Büchler, M. W., Koch, M., & Weitz, J. (2010) · Gastroenterology · doi:10.1053/j.gastro.2010.01.008 · Cited by 308
Bond, M., Mealing, S., Anderson, R., Elston, J., Weiner, G., Taylor, R., Hoyle, M., Liu, Z., Price, A., & Stein, K. (2009) · Health Technology Assessment · doi:10.3310/hta13440 · Cited by 310
Including Non‐Randomized Studies
Reeves, B. C., Deeks, J. J., Higgins, J. P., & Wells, G. A. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch13 · Cited by 288
Self-citationMarkopoulou, P., Papanikolaou, E., Analytis, A., Zoumakis, E., & Siahanidou, T. (2019) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2019.02.041 · Cited by 300
A meta-analysis of endoscopic submucosal dissection and EMR for early gastric cancer
Lian, J., Chen, S., Zhang, Y., & Qiu, F. (2012) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2012.06.014 · Cited by 302
Donahue, K. E., Gartlehner, G., Jonas, D. E., Lux, L. J., Thieda, P., Jonas, B. L., Hansen, R. A., Morgan, L. C., & Lohr, K. N. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-2-200801150-00192 · Cited by 299
Soysal, A., Millington, K. A., Bakir, M., Dosanjh, D., Aslan, Y., Deeks, J. J., Efe, S., Staveley, I., Ewer, K., & Lalvani, A. (2005) · The Lancet · doi:10.1016/s0140-6736(05)67534-4 · Cited by 291
Self-citationP53 abnormalities and outcomes in colorectal cancer: a systematic review
Munro, A. J., Lain, S., & Lane, D. P. (2005) · British Journal of Cancer · doi:10.1038/sj.bjc.6602358 · Cited by 289
“The problems with the design and interpretation of studies dealing with the assessment of prognostic factors have been well rehearsed (Altman and Lyman, 1998; Altman, 2001; Deeks et al, 2003; Riley et al, 2003a, b; Riley et al, 2004).”
BackgroundDynamic Resistance Training as Stand‐Alone Antihypertensive Lifestyle Therapy: A Meta‐Analysis
MacDonald, H. V., Johnson, B. T., Huedo‐Medina, T. B., Livingston, J., Forsyth, K. C., Kraemer, W. J., Farinatti, P. T. V., & Pescatello, L. S. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.116.003231 · Cited by 273
Wallis, C. J. D., Saskin, R., Choo, R., Herschorn, S., Kodama, R. T., Satkunasivam, R., Shah, P. S., Danjoux, C., & Nam, R. K. (2016) · European Urology · doi:10.1016/j.eururo.2015.11.010 · Cited by 282
Thompson, L., & McCabe, R. (2012) · BMC Psychiatry · doi:10.1186/1471-244x-12-87 · Cited by 278
“A review of QA for non-randomised trials demonstrated six tools applicable to systematic reviews [60], from which Down’s et al.”
MethodLever Taylor, B., Cavanagh, K., & Strauss, C. (2016) · PLOS ONE · doi:10.1371/journal.pone.0155720 · Cited by 284
“However, while a number of quality assessment tools exist, and have been used for evaluating uncontrolled studies, most omit key quality domains and all have weaknesses [43].”
Background(2011) · The University of Bath Online Publications Store (The University of Bath) · Cited by 284
Circulating C reactive protein in osteoarthritis: a systematic review and meta-analysis
Jin, X., Beguerie, J. R., Zhang, W., Blizzard, L., Otahal, P., Jones, G., & Ding, C. (2015) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2013-204494 · Cited by 282
Effects of clinical pathways in the joint replacement: a meta-analysis
Barbieri, A., Vanhaecht, K., Van Herck, P., Sermeus, W., Faggiano, F., Marchisio, S., & Panella, M. (2009) · BMC Medicine · doi:10.1186/1741-7015-7-32 · Cited by 284
Murray, J., Farrington, D. P., & Eisner, M. P. (2009) · Journal of Experimental Criminology · doi:10.1007/s11292-008-9066-0 · Cited by 280
Gait modification strategies for altering medial knee joint load: A systematic review
Simic, M., Hinman, R. S., Wrigley, T. V., Bennell, K. L., & Hunt, M. A. (2011) · Arthritis Care & Research · doi:10.1002/acr.20380 · Cited by 278
Golley, R. K., Hendrie, G. A., Slater, A., & Corsini, N. (2011) · Obesity Reviews · doi:10.1111/j.1467-789x.2010.00745.x · Cited by 279
“In an assessment of 213 quality assessment tools, this tool was identified as useful for systematic reviews that evaluate randomized and non-randomized intervention studies (12).”
MethodChong, E. W.-T. (2008) · Archives of Ophthalmology · doi:10.1001/archopht.126.6.826 · Cited by 279
Designing and undertaking randomised implementation trials: guide for researchers
Wolfenden, L., Foy, R., Presseau, J., Grimshaw, J. M., Ivers, N. M., Powell, B. J., Taljaard, M., Wiggers, J., Sutherland, R., Nathan, N., et al. (2021) · BMJ · doi:10.1136/bmj.m3721 · Cited by 270
How to do a good systematic review of effects in international development: a tool kit
Waddington, H., White, H., Snilstveit, B., Hombrados, J. G., Vojtkova, M., Davies, P., Bhavsar, A., Eyers, J., Koehlmoos, T. P., Petticrew, M., et al. (2012) · Journal of Development Effectiveness · doi:10.1080/19439342.2012.711765 · Cited by 278
“All errors are the responsibility of the authors.”
BackgroundSelf-citationStrategies for detecting colon cancer and/or dysplasia in patients with inflammatory bowel disease
Collins, P. D., Mpofu, C., Watson, A. J., & Rhodes, J. M. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000279.pub3 · Cited by 280
Brinjikji, W., Duffy, S., Burrows, A., Hacke, W., Liebeskind, D., Majoie, C. B. L. M., Dippel, D. W. J., Siddiqui, A. H., Khatri, P., Baxter, B., et al. (2017) · Journal of NeuroInterventional Surgery · doi:10.1136/neurintsurg-2016-012391 · Cited by 274
Sagar-Ouriaghli, I., Godfrey, E., Bridge, L., Meade, L., & Brown, J. S. L. (2019) · American Journal of Men’s Health · doi:10.1177/1557988319857009 · Cited by 274
Self-monitoring of blood glucose in type 2 diabetes: systematic review
Clar, C., Barnard, K., Cummins, E., Royle, P., & Waugh, N. (2010) · Health Technology Assessment · doi:10.3310/hta14120 · Cited by 274
Noble, M., Tregear, S. J., Treadwell, J. R., & Schoelles, K. (2008) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2007.03.015 · Cited by 277
Wick, K., Leeger-Aschmann, C. S., Monn, N. D., Radtke, T., Ott, L. V., Rebholz, C. E., Cruz, S., Gerber, N., Schmutz, E. A., Puder, J. J., et al. (2017) · Sports Medicine · doi:10.1007/s40279-017-0723-1 · Cited by 249
“4 Risk of Bias Assessment The reviewers assessed the risk of bias of each eligible study using a slightly adapted version of the established ‘Effective Public Health Practice Project Quality Assessment Tool for Quantitative Studies’ (EPHPP) that has been proven valid in assessing Public Health interventions [35] (ESM…”
MethodLenza, M., Buchbinder, R., Takwoingi, Y., Johnston, R. V., Hanchard, N. C., & Faloppa, F. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009020.pub2 · Cited by 267
Second malignancies after radiotherapy for prostate cancer: systematic review and meta-analysis
Wallis, C. J. D., Mahar, A. L., Choo, R., Herschorn, S., Kodama, R. T., Shah, P. S., Danjoux, C., Narod, S. A., & Nam, R. K. (2016) · BMJ · doi:10.1136/bmj.i851 · Cited by 271
Use of process measures to monitor the quality of clinical practice
Lilford, R. J., Brown, C. A., & Nicholl, J. (2007) · BMJ · doi:10.1136/bmj.39317.641296.ad · Cited by 271
Transarterial (chemo)embolisation for unresectable hepatocellular carcinoma
Oliveri, R. S., Wetterslev, J., & Gluud, C. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004787.pub2 · Cited by 268
A Quality-Effects Model for Meta-Analysis
Doi, S. A. R., & Thalib, L. (2008) · Epidemiology · doi:10.1097/ede.0b013e31815c24e7 · Cited by 271
(2013) · doi:10.1016/j.jclinepi.2011.05.008 · Cited by 264
“These tools vary in the criteria covered [9] and their overall approach.”
BackgroundInfluentialOrmel, H. L., van der Schoot, G. G. F., Sluiter, W. J., Jalving, M., Gietema, J. A., & Walenkamp, A. M. E. (2018) · Psycho-Oncology · doi:10.1002/pon.4612 · Cited by 261
A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261
“Furthermore, most of the CATs lack the depth necessary to comprehensively assess the research being analyzed, and few were designed to assess the quality of the research [2].”
BackgroundInfluentialAmjad, S., MacDonald, I., Chambers, T., Osornio‐Vargas, A., Chandra, S., Voaklander, D., & Ospina, M. B. (2019) · Paediatric and Perinatal Epidemiology · doi:10.1111/ppe.12529 · Cited by 212
Screening for breast cancer with mammography
Gøtzsche, P. C., & Nielsen, M. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001877.pub2 · Cited by 260
Lardas, M., Liew, M., van den Bergh, R. C., De Santis, M., Bellmunt, J., Van den Broeck, T., Cornford, P., Cumberbatch, M. G., Fossati, N., Gross, T., et al. (2017) · European Urology · doi:10.1016/j.eururo.2017.06.035 · Cited by 256
“This is a pragmatic approach informed by methodological literature pertaining to assessing RoB in NRCSs [8, 9].”
MethodImproving the Use of Research Evidence in Guideline Development
Oxman, A. D., Schünemann, H. J., Fretheim, A., Boyd, E. A., Bero, L. A., & Edejer, T. T.-T. (2007) · BIBSYS Brage (BIBSYS (Norway)) · Cited by 261
Brinjikji, W., Murad, M. H., Rabinstein, A. A., Cloft, H. J., Lanzino, G., & Kallmes, D. F. (2015) · American Journal of Neuroradiology · doi:10.3174/ajnr.a4159 · Cited by 257
“The highest quality studies are awarded up to 9 stars.(7)”
BackgroundWong, W. C., Cheung, C. S., & Hart, G. J. (2008) · Emerging Themes in Epidemiology · doi:10.1186/1742-7622-5-23 · Cited by 257
“A number of consensus statements have previously been prepared to encourage higher quality of reporting, including recommendations for reporting systematic reviews (QUOROM)[4], randomized trials (CONSORT)[5], studies of diagnostic tests (STARD)[6], meta-analyses of observational studies (MOOSE)[7] and observational ep…”
A mixture model of nuchal translucency thickness in screening for chromosomal defects
Wright, D., Kagan, K. O., Molina, F. S., Gazzoni, A., & Nicolaides, K. H. (2008) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.5299 · Cited by 257
A Systematic Review of Dextrose Prolotherapy for Chronic Musculoskeletal Pain
Hauser, R. A., Lackner, J. B., Steilen-Matias, D., & Harris, D. K. (2016) · Clinical Medicine Insights: Arthritis and Musculoskeletal Disorders · doi:10.4137/cmamd.s39160 · Cited by 247
Hersch, D., Perdue, L., Ambroz, T., & Boucher, J. L. (2014) · Preventing Chronic Disease · doi:10.5888/pcd11.140267 · Cited by 250
Autologous chondrocyte implantation in the knee: systematic review and economic evaluation
Mistry, H., Connock, M., Pink, J., Shyangdan, D., Clar, C., Royle, P., Court, R., Biant, L. C., Metcalfe, A., & Waugh, N. (2017) · Health Technology Assessment · doi:10.3310/hta21060 · Cited by 255
Systematic reviews of adverse effects: framework for a structured approach
Loke, Y. K., Price, D., Herxheimer, A., & the Cochrane Adverse Effects Methods Group (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-32 · Cited by 253
Walach, H., Falkenberg, T., Fønnebø, V., Lewith, G., & Jonas, W. B. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-29 · Cited by 249
“However, we must address the issue of variability and divergence in non-randomized studies and how this should be managed [69].”
Newman, P. A., Logie, C. H., Lacombe-Duncan, A., Baiden, P., Tepjan, S., Rubincam, C., Doukas, N., & Asey, F. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019206 · Cited by 252
“18 Reviewers followed the EPHPP rubric to assess whether each component had low, moderate or high risk of bias, with any disagreements resolved by consensus with the senior investigator (PAN).”
MethodChatterjee, S., Wetterslev, J., Sharma, A., Lichstein, E., & Mukherjee, D. (2013) · JAMA Internal Medicine · doi:10.1001/2013.jamainternmed.1001 · Cited by 248
O’Connor, S. R., Tully, M. A., Ryan, B., Bradley, J. M., Baxter, G. D., & McDonough, S. M. (2015) · BMC Research Notes · doi:10.1186/s13104-015-1181-1 · Cited by 250
“We selected the rating scale proposed by Downs and Black [16] for comparison as it is one of the most commonly used and well validated numerical rating scales [17].”
MethodFarrah, K., Young, K., Tunis, M. C., & Zhao, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1172-8 · Cited by 246
Hartley, S., Raphael, J., Lovell, K., & Berry, K. (2020) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2019.103490 · Cited by 245
Ismail, K., Maissi, E., Thomas, S., Chalder, T., Schmidt, U., Bartlett, J., Patel, A., Dickens, C., Creed, F., & Treasure, J. (2010) · Health Technology Assessment · doi:10.3310/hta14220 · Cited by 245
The Multiple Baseline Design for Evaluating Population-Based Research
Hawkins, N. G., Sanson-Fisher, R. W., Shakeshaft, A., D’Este, C., & Green, L. W. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.03.020 · Cited by 242
Prgomet, M., Li, L., Niazkhani, Z., Georgiou, A., & Westbrook, J. I. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw145 · Cited by 243
“The tool was selected because it can be used to assess the methodological quality of both randomized and nonrandomized studies, and has been judged suitable for use in systematic reviews.(20) Using the tool, studies are attributed a rating of strong, moderate, or weak based on 6 components: (1) selection bias, (2) stu…”
MethodSystematic Review and Meta-analysis of Adolescent Cognitive–Behavioral Sleep Interventions
Blake, M. J., Sheeber, L. B., Youssef, G. J., Raniti, M. B., & Allen, N. B. (2017) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-017-0234-5 · Cited by 242
“It has been recommended as a strong instrument for assessing the methodological quality of interventions (Deeks et al. 2003; Higgins and Green 2011).”
BackgroundTobin, M. C., Drager, K. D. R., & Richardson, L. F. (2014) · Research in Autism Spectrum Disorders · doi:10.1016/j.rasd.2013.12.002 · Cited by 236
Castelino, T., Fiore, J. F., Niculiseanu, P., Landry, T., Augustin, B., & Feldman, L. S. (2016) · Surgery · doi:10.1016/j.surg.2015.11.029 · Cited by 236
“A large-scale review assessing 194 tools to evaluate methodologic quality deemed the Downs and Black tool as appropriate for use in systematic reviews.(13) The Downs and Black tool consists of 27 items divided into 5 subscales: reporting (10 items), external validity (3 items), bias (7 items), confounding (6 items), a…”
Method
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.