Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
(1963) · Medical Entomology and Zoology · Cited by 15,085
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
Items 801–900 of 1,024
Williamson, I., Benge, S., Barton, S., Petrou, M., Letley, L., Fasey, N., Abangma, G., Dakin, H., & Little, P. (2009) · Health Technology Assessment · doi:10.3310/hta13370 · Cited by 69
Hay, A. D., Redmond, N. M., Costelloe, C., Montgomery, A. A., Fletcher, M. E., Hollinghurst, S. P., & Peters, T. (2009) · Health Technology Assessment · doi:10.3310/hta13270 · Cited by 73
Wahbeh, H., Senders, A., Neuendorf, R., & Cayton, J. (2014) · Journal of Evidence-Based Complementary & Alternative Medicine · doi:10.1177/2156587214525403 · Cited by 61
Clinical Tests for the Diagnosis of Rotator Cuff Disease
Longo, U. G., Berton, A., Ahrens, P. M., Maffulli, N., & Denaro, V. (2011) · Sports Medicine and Arthroscopy Review · doi:10.1097/jsa.0b013e3182250c8b · Cited by 66
Miller, L., McFadyen, A. K., Lord, A. C., Hunter, R., Paul, L., Rafferty, D., Bowers, R. J., & Mattison, P. (2017) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2016.12.007 · Cited by 77
Wang, Z., Lapinski, M. K., Quilliam, E. T., Jaykus, L., & Fraser, A. M. (2017) · American Journal of Infection Control · doi:10.1016/j.ajic.2017.01.018 · Cited by 80
Uhlig, A., Strauß, A., Hosseini, A. S. A., Lotz, J., Trojan, L., Schmid, M., & Uhlig, J. (2017) · European Urology Focus · doi:10.1016/j.euf.2017.08.007 · Cited by 66
Ajrawat, P., Dwyer, T., Whelan, D. B., Theodoropoulos, J. S., Murnaghan, M. L., Bhargava, M., Ogilvie‐Harris, D. J., & Chahal, J. (2019) · Clinical Journal of Sport Medicine · doi:10.1097/jsm.0000000000000765 · Cited by 75
“The Newcastle–Ottawa Scale (NOS) is a widely used scale that evaluates nonrandomized studies, which has been recommended by the Cochrane Collaboration and Deeks et al as a suitable tool used in a systematic review.(25,26) The NOS uses a star-based grading system that ranks studies out of 9 stars.”
MethodAssessing harmful effects in systematic Reviews
McIntosh, H. M., Woolacott, N. F., & Bagnall, A.-M. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-19 · Cited by 69
A systematic review of research methodology in telemedicine studies
Whitten, P., Johannessen, L. K., Soerensen, T., Gammon, D., & Mackert, M. (2007) · Journal of Telemedicine and Telecare · doi:10.1258/135763307781458976 · Cited by 67
Wagner, D., Thomssen, C., Haerting, J., & Unverzagt, S. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008941.pub2 · Cited by 85
Winter, J. R., Jackson, C., Lewis, J. E. A., Taylor, G. S., Thomas, O. G., & Stagg, H. R. (2020) · Journal of Global Health · doi:10.7189/jogh.10.010404 · Cited by 79
Psychological factors involved in adherence to sport injury rehabilitation: a systematic review
Goddard, K., Roberts, C.-M., Byron‐Daniel, J., & Woodford, L. (2020) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2020.1744179 · Cited by 88
Setting a standard for electricity pilot studies
Davis, A. L., Krishnamurti, T., Fischhoff, B., & de Bruin, W. B. (2013) · Energy Policy · doi:10.1016/j.enpol.2013.07.093 · Cited by 74
Lens extraction for chronic angle-closure glaucoma
Friedman, D. S., & Vedula, S. S. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005555.pub2 · Cited by 77
Fleeman, N., McLeod, C., Bagust, A., Beale, S., Boland, A., Dundar, Y., Jorgensen, A., Payne, K., Pirmohamed, M., Pushpakom, S. P., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14030 · Cited by 72
Safari, R., & Meier, M. R. (2015) · The Journal of Rehabilitation Research and Development · doi:10.1682/jrrd.2014.08.0183 · Cited by 77
Witberg, G., Zusman, O., Codner, P., Assali, A., & Kornowski, R. (2018) · Circulation Cardiovascular Interventions · doi:10.1161/circinterventions.117.006000 · Cited by 67
Mari, A., Kimura, S., Foerster, B., Abufaraj, M. M., D’Andrea, D., Gust, K. M., & Shariat, S. F. (2018) · Urologic Oncology Seminars and Original Investigations · doi:10.1016/j.urolonc.2018.03.018 · Cited by 65
Schepens, H., Puyenbroeck, J. V., & Maes, B. (2018) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.12559 · Cited by 84
“Methods of research are identified as quantitative, qualitative, or mixed; quantitative study designs as experimental or observational, and qualitative study designs by their approach, data collection or analysis (Deeks et al., 2003; Savin‐Baden & Major, 2013).”
MethodHow to Appraise the Studies: An Introduction to Assessing Study Quality
(2006) · Systematic Reviews in the Social Sciences · doi:10.1002/9780470754887.ch5 · Cited by 63
Self-citationPhysician anaesthetists versus non-physician providers of anaesthesia for surgical patients
Lewis, S. R., Nicholson, A., Smith, A. F., & Alderson, P. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010357.pub2 · Cited by 64
Hersi, M., Stevens, A., Quach, P., Hamel, C., Thavorn, K., Garritty, C., Skidmore, B., Vallenas, C., Norris, S. L., Egger, M., et al. (2015) · PLoS ONE · doi:10.1371/journal.pone.0140290 · Cited by 64
“Risk of bias assessments were not done due to the lack of validated instruments to assess the methodological quality of non-comparative designs [20].”
BackgroundPatel, Z. M., Thamboo, A. V., Rudmik, L. R., Nayak, J. V., Smith, T. L., & Hwang, P. H. (2016) · International Forum of Allergy & Rhinology · doi:10.1002/alr.21872 · Cited by 74
Tran, G. T., Cowling, P. D., Smith, T. O., Bury, J., Lucas, A., Barr, A., Kingsbury, S. R., & Conaghan, P. G. (2018) · Arthritis Care & Research · doi:10.1002/acr.23554 · Cited by 64
Effectiveness of antiarrhythmic drugs for shockable cardiac arrest: A systematic review
Ali, M. U., Fitzpatrick‐Lewis, D., Kenny, M., Raina, P. S., Atkins, D. L., Soar, J., Nolan, J. P., Ristagno, G., & Sherifali, D. (2018) · Resuscitation · doi:10.1016/j.resuscitation.2018.08.025 · Cited by 67
Way, C., Wayne, C., Grandpierre, V., Harrison, B. J., Travis, N., & Nasr, A. (2019) · Pediatric Surgery International · doi:10.1007/s00383-019-04527-9 · Cited by 71
Childhood and Adolescent Bullying Perpetration and Later Substance Use: A Meta-analysis
Vrijen, C., Wiertsema, M., Ackermans, M. A., van der Ploeg, R., & Kretschmer, T. (2021) · PEDIATRICS · doi:10.1542/peds.2020-034751 · Cited by 73
A Systematic Review of the Use of Rituximab for Desensitization in Renal Transplantation
Macklin, P. S., Morris, P. J., & Knight, S. R. (2014) · Transplantation · doi:10.1097/tp.0000000000000362 · Cited by 64
“appropriate for use in systematic reviews (90).”
Kramer, A. H., & Fletcher, J. J. (2009) · Neurocritical Care · doi:10.1007/s12028-009-9306-9 · Cited by 64
The incidence of complications associated with lip and/or tongue piercings: a systematic review
Hennequin‐Hoenderdos, N. L., Slot, D. E., & van der Weijden, G. A. (2015) · International Journal of Dental Hygiene · doi:10.1111/idh.12118 · Cited by 64
Robson, S., Kelly, T., Howel, Deverill, M., Hewison, J., Lie, M., Stamp, E., Armstrong, N., & May, C. (2009) · Health Technology Assessment · doi:10.3310/hta13530 · Cited by 62
Effectiveness of Nonpharmacologic Interventions in Systemic Sclerosis: A Systematic Review
Willems, L. M., Vriezekolk, J. E., Schouffoer, A. A., Poole, J. L., Stamm, T., Boström, C., Kwakkenbos, L., Vlieland, T. P. M. V., & van den Ende, C. H. M. (2015) · Arthritis Care & Research · doi:10.1002/acr.22595 · Cited by 67
“The Cochrane Musculoskeletal Group has rated the Downs and Black checklist in the top 6 quality assessment tools that are potentially useful for systematic reviews including nonrandomized studies (18–20).”
Galloway, H., & Newman, E. (2016) · ADHD Attention Deficit and Hyperactivity Disorders · doi:10.1007/s12402-016-0210-9 · Cited by 84
“However, randomised controlled trials (RCTs) were not excluded from the review.”
BackgroundHidradenitis suppurativa and diabetes mellitus: A systematic review and meta-analysis
Bui, T., Silva‐Hirschberg, C., Torres, J., & Armstrong, A. W. (2017) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2017.08.042 · Cited by 69
“To measure study quality, each reviewer applied the appropriate Newcastle-Ottawa scale to every included article (Table II).(28)”
MethodAudag, N., Goubau, C., Toussaint, M., & Reychler, G. (2019) · Therapeutic Advances in Chronic Disease · doi:10.1177/2040622318821622 · Cited by 69
Stress management interventions for intensive and critical care nurses: A systematic review
Alkhawaldeh, J. M. A., Soh, K. L., Mukhtar, F. B. M., Peng, O. C., & Anshasi, H. A. (2020) · Nursing in Critical Care · doi:10.1111/nicc.12489 · Cited by 63
South, J., Bagnall, A.-M., Hulme, C., Woodall, J., Longo, R., Dixey, R., Kinsella, K., Raine, G., Vinall-Collier, K., & Wright, J. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02350 · Cited by 63
“Previous research has identified the lack of useful validity checklists for non-randomised studies.(237) There is debate over the correctness of using any checklist for the assessment of qualitative research;(107) one of the reasons given is that decisions are subjective and there has been very low inter-rater reliabi…”
A systematic review of parent based programs to prevent or reduce alcohol consumption in adolescents
Hurley, E., Dietrich, T., & Rundle-Thiele, S. R. (2019) · BMC Public Health · doi:10.1186/s12889-019-7733-x · Cited by 73
Ling, C. N. Y., Lim, S. C., Jonas, J. B., & Sabanayagam, C. (2021) · International Journal of Obesity · doi:10.1038/s41366-021-00829-y · Cited by 70
Kaminski, A., Kamper, A., Thaler, K., Chapman, A., & Gartlehner, G. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008013.pub2 · Cited by 63
Can AMSTAR also be applied to systematic reviews of non-randomized studies?
Pieper, D., Mathes, T., & Eikermann, M. (2014) · BMC Research Notes · doi:10.1186/1756-0500-7-609 · Cited by 62
Effects of interventions on depression in heart failure: A systematic review
Woltz, P. C., Chapa, D. W., Friedmann, E., Son, H., Akintade, B. F., & Thomas, S. A. (2012) · Heart & Lung · doi:10.1016/j.hrtlng.2012.06.002 · Cited by 70
DANIELS, J. P., Gray, J. W., Pattison, H. M., Roberts, T. E., Edwards, E. A., Milner, P., Spicer, L., King, E., Hills, R. K., Gray, R. G., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13420 · Cited by 75
Surgical Treatment of Chronic Retrocalcaneal Bursitis
Wiegerinck, J. I., Kok, A., & van Dijk, C. N. (2012) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2011.09.019 · Cited by 65
“Nesse and Finsen38 (Acta Orthop Scand 1994) IV 23 (35) 21 (16-43); 16 Clinical findings NR Symptomatic Haglund heel (defined as pain and thickening lateral to AT) 36 (12-72) Open VAS”
BackgroundInfluentialA systematic review of the effectiveness of mass media campaigns for the management of low back pain
Suman, A., Armijo-Olivo, S. L., Deshpande, S., Marietta-Vasquez, J., Dennett, L., Miciak, M., Reneman, M. F., Werner, E. L., Straube, S., Buchbinder, R., et al. (2020) · Disability and Rehabilitation · doi:10.1080/09638288.2020.1743777 · Cited by 69
Lazcano-Ponce, E. C., Salazar-Martínez, E., Gutiérrez‐Castrellón, P., Ángeles-Llerenas, A., Hernández-Garduño, A. G., & Viramontes, J. L. (2004) · Salud Pública de México · doi:10.1590/s0036-36342004000600012 · Cited by 166
Sohanpal, R., Hooper, R., Hames, R., Priebe, S., & Taylor, S. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-66 · Cited by 51
Comparison of case note review methods for evaluating quality and safety in health care
Lock, C. A., Wilson, J. A., Steen, N., Eccles, M., Mason, H., Carrie, S., Clarke, R. W., Kubba, H., Raine, C. H., Zarod, A., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14130 · Cited by 63
Does injury compensation lead to worse health after whiplash? A systematic review
Spearing, N., Connelly, L. B., Gargett, S., & Sterling, M. M. (2012) · Pain · doi:10.1016/j.pain.2012.03.007 · Cited by 71
Zhu, P., Zhou, P., Sun, Y., Guo, Y., Mai, M., & Shaoyi, Z. (2015) · Journal of Cardiothoracic Surgery · doi:10.1186/s13019-015-0262-5 · Cited by 62
“Quality of non-RCTs was evaluated with the modified NOS (http://www.ohri.ca/programs/clinical_ epidemiology/oxford.asp), which addressed three items: patient selection, comparability of groups, and outcome assessment (Table 1).”
MethodInfluentialBach Flower Remedies for psychological problems and pain: a systematic review
Thaler, K. J., Kaminski, A., Chapman, A. L., Langley, T. E., & Gartlehner, G. (2009) · BMC Complementary and Alternative Medicine · doi:10.1186/1472-6882-9-16 · Cited by 62
Turner, L., Boutron, I., Hróbjartsson, A., Altman, D. G., & Moher, D. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-79 · Cited by 62
“Even 10 years ago there were 114 published tools for assessing risk of bias in non-randomised studies [115].”
MethodSelf-citationLaparoscopic versus open Hartmann’s reversal: a systematic review and meta-analysis
Celentano, V., Giglio, M. C., & Bucci, L. (2015) · International Journal of Colorectal Disease · doi:10.1007/s00384-015-2325-4 · Cited by 62
Mind-Body Interventions for Treatment of Phantom Limb Pain in Persons with Amputation
Moura, V. L., Faurot, K. R., Gaylord, S. A., Mann, J. D., Sill, M., Lynch, C., & Lee, M. Y. (2012) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0b013e3182466034 · Cited by 61
Traditional vs. Alternative Teacher Preparation Programs: A Meta-Analysis
Whitford, D. K., Zhang, D., & Katsiyannis, A. (2017) · Journal of Child and Family Studies · doi:10.1007/s10826-017-0932-0 · Cited by 90
“Quality Indicators Although multiple resources and recommendations are available to assess the quality of studies (e.g., Deeks et al. 2003), studies included in this meta-analysis were assessed for quality using the essential quality indicators for group experimental and quasi-experimental research…”
Márta, K., Lãzãrescu, A.-M., Farkas, N., Mátrai, P., Cazacu, I. M., Ottóffy, M., Habon, T., Erőss, B. M., Vincze, Á., Veres, G., et al. (2019) · Frontiers in Physiology · doi:10.3389/fphys.2019.00328 · Cited by 62
“the modified Newcastle–Ottawa scale as described earlier (Deeks et al., 2003; Mata et al., 2015; Rotenstein et al., 2016).”
MethodFurrer, M. A., Spycher, S. C. J., Büttiker, S. M., Gross, T., Bosshard, P., Thalmann, G. N., Schneider, M. P., & Roth, B. (2019) · European Urology Oncology · doi:10.1016/j.euo.2019.08.013 · Cited by 61
(2017) · Europe PMC (PubMed Central) · Cited by 61
Malik, N., Facer-Irwin, E., Dickson, H., Bird, A., & MacManus, D. (2021) · Trauma Violence & Abuse · doi:10.1177/15248380211043890 · Cited by 78
Tacrolimus-induced nephrotoxicity and genetic variability: A review
de Wildt, S. N. (2012) · Annals of Transplantation · doi:10.12659/aot.883229 · Cited by 64
Coleman, T., Agboola, S., Leonardi-Bee, J., Taylor, M., McEwen, A., & McNeill, A. (2010) · Health Technology Assessment · doi:10.3310/hta14490 · Cited by 62
Herbal Treatments of Asthma: A Systematic Review
Singh, B. B., Khorsan, R., Vinjamury, S. P., Der‐Martirosian, C., Kizhakkeveettil, A., & Anderson, T. M. (2007) · Journal of Asthma · doi:10.1080/02770900701247202 · Cited by 64
“Sixteen (16) were RCTs and twenty one (21) QEDs of which 25 total were trials involving the use of Ayurvedic and Siddha traditional Indian systems of medicine (See Tables 4–7).”
BackgroundLowther, H., & Newman, E. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.06.051 · Cited by 68
“A further itemwas added into the ‘confounding’ subscale to account for the absence of an item assessing the measurement of baseline comparability, as suggested in a review of quality criteria for non-randomised research (Deeks et al., 2003).”
BackgroundJameson, S. S., Baker, P. N., Charman, S. C., Deehan, D. J., Reed, M. R., GREGG, P. J., & van der Meulen, J. (2012) · Journal of Bone and Joint Surgery - British Volume · doi:10.1302/0301-620x.94b7.29129 · Cited by 61
Bilandzic, A., Fitzpatrick, T., Rosella, L., & Henry, D. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1001987 · Cited by 60
“A wide variety of checklists, judgment ratings, and scales for observational studies have been proposed [10,11], including the Newcastle–Ottawa Scale (NOS) [12], the Downs and Black checklist [13], and the Scottish Risk of Bias Assessment in Systematic Reviews of Non-Randomized Studies”
MethodShin, J. Y., & Yi, H. S. (2016) · Burns · doi:10.1016/j.burns.2016.03.012 · Cited by 61
Noninvasive prenatal testing for fetal aneuploidy and single gene disorders
Skrzypek, H., & Hui, L. (2017) · Best Practice & Research Clinical Obstetrics & Gynaecology · doi:10.1016/j.bpobgyn.2017.02.007 · Cited by 75
The association between pain and sleep in fibromyalgia
Keskindag, B., & Karaaziz, M. (2017) · Saudi Medical Journal · doi:10.15537/smj.2017.5.17864 · Cited by 60
Mori, K., Janisch, F., Mostafaei, H., Lysenko, I., Kimura, S., Egawa, S., & Shariat, S. F. (2020) · Urologic Oncology Seminars and Original Investigations · doi:10.1016/j.urolonc.2020.01.015 · Cited by 69
Nelson, K. M., Perry, N. S., Horvath, K. J., & Smith, L. R. (2020) · Translational Behavioral Medicine · doi:10.1093/tbm/ibaa007 · Cited by 64
Evidence-Based Interventional Pain Management: Principles, Problems, Potential and Applications
Manchikanti, L., Boswell, M. V., & Giordano, J. (2007) · Pain Physician · doi:10.36076/ppj.2007/10/329 · Cited by 65
Toews, I., Anglemyer, A., Nyirenda, J. L., Alsaid, D., Balduzzi, S., Grummich, K., Schwingshackl, L., & Bero, L. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub3 · Cited by 60
Gupta, P., Lança, C., Gan, A. T. L., Soh, P., Thakur, S., Tao, Y., Neelam, K., Man, R. E. K., Fenwick, E., & Lamoureux, E. L. (2019) · Scientific Reports · doi:10.1038/s41598-019-49162-5 · Cited by 74
Mozaffari, H., Hosseini, Z., Lafrenière, J., & Conklin, A. (2021) · Obesity Reviews · doi:10.1111/obr.13174 · Cited by 67
“Because the Cochrane method for quality assessment (designed for appraising clinical practice) considers observational study evidence as low quality, we employed the Effective Public Health Practice Project tool and the Newcastle-Ottawa scale—both identified by the UK's HTA Programme as “best” tools for evaluating obs…”
MethodZakhem, G. A., Pulavarty, A. N., Carucci, J. A., & Stevenson, M. L. (2022) · JAMA Dermatology · doi:10.1001/jamadermatol.2022.5508 · Cited by 62
A systematic review: The role and impact of the physician assistant in the emergency department
Doan, Q. H., Sabhaney, V. J., Kissoon, N. “., Sheps, S. B., & Singer, J. (2011) · Emergency Medicine Australasia · doi:10.1111/j.1742-6723.2010.01368.x · Cited by 71
“Each study was assessed independently by two reviewers using the ‘Quality Assessment Tool for Quantitative Studies Method’ developed by Thomas,(5) which has been found to be suitable for use in a systematic review of nonrandomized studies.(6)”
MethodNon-occupational postexposure prophylaxis for HIV: a systematic review
Bryant, J., Baxter, L., & Hird, S. (2009) · Health Technology Assessment · doi:10.3310/hta13140 · Cited by 68
Hamstring Tendon Regeneration After Harvesting: A Systematic Review
Suijkerbuijk, M. A. M., Reijman, M., Lodewijks, S. J. M., Punt, J., & Meuffels, D. E. (2014) · The American Journal of Sports Medicine · doi:10.1177/0363546514562169 · Cited by 66
“We assessed the risk of bias of studies using a quality assessment list (Table 1), based on modified questions of existing quality assessment tools.(6,7,29) The purposes of this systematic review were of a different nature.”
MethodPhysical activity in spondyloarthritis: a systematic review
O’Dwyer, T., O'Shea, F. D., & Wilson, F. (2014) · Rheumatology International · doi:10.1007/s00296-014-3141-9 · Cited by 67
“Numerous appraisal tools are described in the literature, yet no gold standard exists for evaluating risk of bias [38, 39].”
BackgroundClegg, A. J., Scott, D. A., Loveman, E., Colquitt, J. L., Hutchinson, J., Royle, P., & Bryant, J. (2005) · Health Technology Assessment · doi:10.3310/hta9450 · Cited by 61
Colson, N., Naug, H. L., Nikbakht, E., Zhang, P., & McCormack, J. (2015) · Zeitschrift für Ernährungswissenschaft · doi:10.1007/s00394-015-1076-x · Cited by 66
Macri, E. M., Stefanik, J. J., Karim, K., & Crossley, K. M. (2016) · Arthritis Care & Research · doi:10.1002/acr.22842 · Cited by 68
“Our tool was adapted from Downs and Black (14) and influenced by previously published modifications (15,16), recent systematic reviews of quality assessment tools (17,18) and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines for reporting of observational studies (19).”
MethodThompson, T., Gallop, K., Correll, C. U., Carvalho, A. F., Veronese, N., Wright, E., & Stubbs, B. (2017) · Ageing Research Reviews · doi:10.1016/j.arr.2017.01.005 · Cited by 67
Sujenthiran, A., Elshout, P. J., Veskimäe, E., MacLennan, S., Yuan, Y., Serafetinidis, E., Sharma, D. M., Kitrey, N. D., Djakovic, N., Lumen, N., et al. (2017) · European Urology Focus · doi:10.1016/j.euf.2017.04.011 · Cited by 60
“This was a pragmatic approach based on methodological literature [22,23] and included an additional domain to assess the risk of confounding bias.”
MethodFennelly, O., Blake, C. M., Desmeules, F., Stokes, D., & Cunningham, C. (2017) · Musculoskeletal Care · doi:10.1002/msc.1200 · Cited by 69
“It has been reported to have content and construct validity (Thomas, Ciliska, Dobbins, & Micucci, 2004), excellent inter‐rater reliability (Armijo‐Olivo, Stiles, Hagen, Biondo, & Cummings, 2012), and was rated by experts as one of the best tools to use when assessing the quality of studies for systematic reviews (Deek…”
BackgroundEffectiveness of Interventions to Promote Sustainable Employability: A Systematic Review
Hazelzet, E., Picco, E., Houkes, I., Bosma, H., & de Rijk, A. E. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16111985 · Cited by 60
“It is suitable for use in a systematic literature review and has previously been used in other studies [15,16].”
MethodMori, K., Janisch, F., Parizi, M. K., Mostafaei, H., Lysenko, I., Enikeev, D. V., Kimura, S., Egawa, S., & Shariat, S. F. (2019) · International Journal of Clinical Oncology · doi:10.1007/s10147-019-01578-9 · Cited by 63
“1 3 Handbook for systematic reviews of interventions for included non-randomized studies [20, 21].”
BackgroundLazarus, H. M., Pérez, W. S., Klein, J. P., Kollman, C., Bate-Boyle, B., Bredeson, C., Gale, R. P., Geller, R. B., Keating, A., Litzow, M. R., et al. (2006) · British Journal of Haematology · doi:10.1111/j.1365-2141.2005.05947.x · Cited by 60
“Even if the data were complete, adjustments for confounding cannot guarantee removal of bias (Deeks et al, 2003).”
BackgroundDonald, F., Kilpatrick, K., Reid, K., Carter, N., Martin-Misener, R., Bryant-Lukosius, D., Harbman, P., Kaasalainen, S., Marshall, D. A., Charbonneau-Smith, R., et al. (2014) · Nursing Research and Practice · doi:10.1155/2014/896587 · Cited by 59
Vigueras, N. M., Román, E. P., Valenza, M. C., Santiago, M. G., Martos, I. C., Rodríguez‐Torres, J., & Torres‐Sánchez, I. (2020) · Clinical Rehabilitation · doi:10.1177/0269215520976267 · Cited by 68
Pulman, J., & Buckley, E. J. (2013) · Topics in Stroke Rehabilitation · doi:10.1310/tsr2002-171 · Cited by 66
Screening for hyperglycaemia in pregnancy: a rapid update for the National Screening Committee
Waugh, N., Royle, P., Clar, C., Henderson, R., Cummins, E., Hadden, D., Lindsay, R., & Pearson, D. (2010) · Health Technology Assessment · doi:10.3310/hta14450 · Cited by 60
Coker, E., Ploeg, J., & Kaasalainen, S. (2014) · Research in Gerontological Nursing · doi:10.3928/19404921-20140110-01 · Cited by 59
Interferon for secondary progressive multiple sclerosis: a systematic review
Mantia, L. L., Vacchi, L., Rovaris, M., Di Pietrantonj, C., Ebers, G., Fredrikson, S., & Filippini, G. (2013) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp-2012-303291 · Cited by 61
“33 Such non-randomised studies are highly susceptible to bias, particularly because of differences between comparison groups (selection bias) and lack of study protocol (reporting bias) and their findings must be interpreted with caution.(35) The results of this review suggest that the effects of IFNb on relapse are i…”
BackgroundSethi, N. J., Nielsen, E. E., Safi, S., Feinberg, J. B., Gluud, C. N., & Jakobsen, J. C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0193924 · Cited by 66
Taylor, R. S., & Elston, J. (2009) · Health Technology Assessment · doi:10.3310/hta13080 · Cited by 63
Smoking cessation apps: a systematic review of format, outcomes, and features
Barroso-Hurtado, M., Suárez-Castro, D., Martínez-Vispo, C., Becoña, E., & López-Durán, A. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182111664 · Cited by 58
“This tool has been deemed suitable for a systematic review of effectiveness [30].”
MethodEfficacy of face-to-face versus self-guided treatments for disordered gambling: a meta-analysis
Goslar, M., Leibetseder, M., Muench, H. M., Hofmann, S. G., & Laireiter, A.-R. (2017) · Journal of Behavioral Addictions · doi:10.1556/2006.6.2017.034 · Cited by 58
“This tool has demonstrated content and construct validity (Thomas et al., 2004), and is recommended for use in systematic reviews and meta-analyses (Deeks et al., 2003).”
MethodLü, C., Yang, J. M., Yu, W.-L., Li, D., Xiang, Z., Lin, Y., & Yu, C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0132036 · Cited by 58
“Another two investigators assessed the quality of papers by applying the Strengthening the Reporting of Observational studies in Epidemiology (STROBE) checklist[13] or other tools[14, 15].”
Method
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.