Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
(1963) · Medical Entomology and Zoology · Cited by 15,085
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
Items 701–800 of 1,024
Is Implicit Motor Learning Preserved after Stroke? A Systematic Review with Meta-Analysis
Kal, E. C., Winters, M., van der Kamp, J., Houdijk, H., Groet, E., van Bennekom, C. A. M., & Scherder, E. (2016) · PLoS ONE · doi:10.1371/journal.pone.0166376 · Cited by 89
Macklin, P. S., Morris, P. J., & Knight, S. R. (2017) · Transplantation Reviews · doi:10.1016/j.trre.2017.01.002 · Cited by 77
“Risk of bias in individual studies The methodological quality of included studies was evaluated by the Downs and Black Quality Index [24], one of only six tools to evaluate the quality of non-randomized studies deemed appropriate for use in systematic reviews [25].”
MethodYoga and Quality of Life in Women with Primary Dysmenorrhea: A Systematic Review
McGovern, C. E., & Cheung, C. (2018) · Journal of Midwifery & Women s Health · doi:10.1111/jmwh.12729 · Cited by 82
Romeo, B., Hermand, M., Pétillion, A., Karila, L., & Benyamina, A. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.03.002 · Cited by 76
Randomised controlled trials for policy interventions: a review of reviews and meta-regression
Oliver, S., Bagnall, A., Thomas, J., Shepherd, J., Sowden, A., White, I., Dinnes, J., Rees, R., Colquitt, J., Oliver, K., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14160 · Cited by 76
Self-citationLuan, F.-J., Wan, Y., Mak, K. C., Ma, C.-J., & Wang, H. (2020) · European Spine Journal · doi:10.1007/s00586-020-06573-7 · Cited by 84
“Scores for low (0–3), moderate (4–6) and high-quality studies (7–9) were assigned [13].”
BackgroundBaradaran, A., Dehghanbanadaki, H., Naderpour, S., Pirkashani, L. M., Rajabi, A., Rashti, R., Riyahifar, S., & Moradi, Y. (2021) · Clinical Diabetes and Endocrinology · doi:10.1186/s40842-021-00131-w · Cited by 85
“the selected case–control studies [25].”
Auerbach, A. D., Rasic, M. A., Sehgal, N. J., Ide, B., Stone, B., & Maselli, J. H. (2007) · Archives of Internal Medicine · doi:10.1001/archinte.167.21.2338 · Cited by 83
Fast track surgery versus conventional recovery strategies for colorectal surgery
Spanjersberg, W. R., Reurings, J. C., & van Laarhoven, C. J. H. M. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007635 · Cited by 77
Drukker, M., Weltens, I., van Hooijdonk, C. F. M., Vandenberk, E., & Bak, M. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.675071 · Cited by 75
“The first systematic review using the NOS was published in 2003 in PubMed (Deeks et al., 2003).”
MethodSystematic review of interventions to improve the psychological well-being of general practitioners
Murray, M., Murray, L., & Donnelly, M. (2016) · BMC Family Practice · doi:10.1186/s12875-016-0431-1 · Cited by 75
“for use in non-randomised intervention studies [41]) deemed them to be ‘weak’ in quality.”
MethodFrom caution to urgency: the evolution of HIV testing and counselling in Africa
Baggaley, R. C., Hensen, B., Ajose, O., Grabbe, K., Wong, V. J., Schilsky, A., Lo, Y.-R., Lule, F., Granich, R., & Hargreaves, J. (2012) · Bulletin of the World Health Organization · doi:10.2471/blt.11.100818 · Cited by 75
Omer, S., Jijon, A. M., & Leonard, H. C. (2018) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13001 · Cited by 86
Strategies of left ventricular unloading during VA-ECMO support: a network meta-analysis
Baldetti, L., Gramegna, M., Beneduce, A., Melillo, F., Moroni, F., Calvo, F., Melisurgo, G., Ajello, S., Fominskiy, E. V., Pappalardo, F., et al. (2020) · International Journal of Cardiology · doi:10.1016/j.ijcard.2020.02.004 · Cited by 78
Watkins, K., Wood, H. A., Schneider, C. R., & Clifford, R. M. (2015) · Implementation Science · doi:10.1186/s13012-015-0337-7 · Cited by 83
“was one of only two tools recommended by Deeks in a review of 182 instruments for measuring risk of bias in cohort studies [50].”
MethodHoen, L. ’., Ecclestone, H., Blok, B. F. M., Karsenty, G., Phé, V., Bossier, R., Groen, J., Castro-Diaz, D. M., Padilla-Fernández, B. Y., POPOLO, G. D., et al. (2017) · Neurourology and Urodynamics · doi:10.1002/nau.23205 · Cited by 75
“Furthermore, for each of the seven identified potential confounders an assessment was made whether they were considered, balanced between the patients and controlled for during the analysis.(11) The risk of bias was considered to be high if the confounder had not been considered, was imbalanced between patients or not…”
BackgroundRied‐Larsen, M., Aarts, H. M., & Joyner, M. J. (2017) · Journal of Applied Physiology · doi:10.1152/japplphysiol.00415.2017 · Cited by 86
Diaz-Piedra, C., Di Stasi, L. L., Baldwin, C. M., Buela-Casal, G., & Catena, A. (2015) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2014.09.001 · Cited by 74
Feedback interventions for impaired self-awareness following brain injury: A systematic review
Schmidt, J., Lannin, N. A., Fleming, J., & Ownsworth, T. (2011) · Journal of Rehabilitation Medicine · doi:10.2340/16501977-0846 · Cited by 84
Suri, P., Fry, A. L., & Gellhorn, A. C. (2015) · PM&R · doi:10.1016/j.pmrj.2015.04.016 · Cited by 74
“The NOS has been shown previously to have face and criterion validity, and it is considered to be one of the most useful existing tools for assessment of the risk of bias in nonrandomized studies [15,16].”
MethodShaw, K., Charlton, J. M., Perry, C. K. L., de Vries, C. M., Redekopp, M. J., White, J. A., & Hunt, M. A. (2017) · British Journal of Sports Medicine · doi:10.1136/bjsports-2016-097108 · Cited by 74
Remiro‐Azócar, A., Heath, A., & Baio, G. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1511 · Cited by 81
Growth patterns during and after treatment in patients with pediatric ALL: A meta‐analysis
Zhang, F. F., Liu, S., Chung, M., & KELLY, M. J. (2015) · Pediatric Blood & Cancer · doi:10.1002/pbc.25519 · Cited by 87
“There is no standard scale to assess quality in observational studies [13,14].”
BackgroundTotal knee arthroplasty after high tibial osteotomy. A systematic review
van Raaij, T. M., Reijman, M., Furlan, A. D., & Verhaar, J. A. N. (2009) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-10-88 · Cited by 77
“The critical appraisal of observational studies tool (Deeks) [10] and the methodological index for non-randomized studies (MINORS) form [12] were used.”
FMR1 CGG expansions: Prevalence and sex ratios
Maenner, M. J., Baker, M. W., Broman, K. W., Tian, J., Fedler, J. K., Atkins, A. E., McPherson, E. W., Hong, J., Brilliant, M. H., & Seltzer, M. M. (2013) · American Journal of Medical Genetics Part B Neuropsychiatric Genetics · doi:10.1002/ajmg.b.32176 · Cited by 76
Priority setting for health technology assessment at CADTH
Husereau, D., Boucher, M., & Noorani, H. Z. (2010) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462310000383 · Cited by 76
Bin, S., Kim, H., Ahn, H.-S., Rim, D. S., & Lee, D. (2016) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2016.06.012 · Cited by 73
Fathers' Stress in a Neonatal Intensive Care Unit
Noergaard, B., Ammentorp, J., Garne, E., Fenger-Gron, J., & Kofoed, P.-E. (2018) · Advances in Neonatal Care · doi:10.1097/anc.0000000000000503 · Cited by 74
Goldberg, H., Ahmad, A. E., Chandrasekar, T., Klotz, L. H., Emberton, M., Haider, M. A., Taneja, S. S., Arora, K., Fleshner, N. E., Finelli, A., et al. (2019) · The Journal of Urology · doi:10.1097/ju.0000000000000595 · Cited by 73
Reinikainen, M., Mussalo, P., Hovilehto, S., Uusaro, A. V., Varpula, T., Kari, A., & Pettilä, V. (2012) · Acta Anaesthesiologica Scandinavica · doi:10.1111/j.1399-6576.2012.02669.x · Cited by 72
Interventions for the prevention of nutritional rickets in term born children
Lerch, C., & Meissner, T. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006164.pub2 · Cited by 69
Second trimester serum tests for Down's Syndrome screening
Alldred, S. K., Deeks, J. J., Guo, B., Neilson, J. P., & Alfirević, Ž. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009925 · Cited by 73
Self-citationHow effective is drug testing as a workplace safety strategy? A systematic review of the evidence
Pidd, K., & ROCHE, A. M. (2014) · Accident Analysis & Prevention · doi:10.1016/j.aap.2014.05.012 · Cited by 72
Screening for Pre-eclampsia–Lessons from Aneuploidy Screening
Cuckle, H. (2011) · Placenta · doi:10.1016/j.placenta.2010.07.015 · Cited by 75
Predictors of Primary Achilles Tendon Ruptures
Claessen, F. M. A. P., de Vos, R., Reijman, M., & Meuffels, D. E. (2014) · Sports Medicine · doi:10.1007/s40279-014-0200-z · Cited by 78
“We assessed the methodological quality of all studies using modified questions from existing quality assessment tools [15–17].”
MethodSurgical options for patients with shoulder pain
Chaudhury, S., Gwilym, S. E., Moser, J. S., & Carr, A. J. (2010) · Nature Reviews Rheumatology · doi:10.1038/nrrheum.2010.25 · Cited by 87
Voss, P. H., & Rehfuess, E. A. (2013) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2011-200940 · Cited by 72
“Deeks and colleagues assessed 194 QATs; 14 covered at least five internal validity domains and three core items for nonrandomised studies and were thus considered most reliable.(5) Among these, we selected the Effective Public Health Practice Project tool (EPHPPT; recommended by the Cochrane Public Health Group),(17)…”
MethodHabers, G. E. A., & Takken, T. (2011) · Rheumatology · doi:10.1093/rheumatology/ker292 · Cited by 72
“This tool is able to deal with any study design and was shown to be suitable for use in a systematic review [15].”
Hemmingsen, B., Sonne, D. P., Metzendorf, M.-I., & Richter, B. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012151.pub2 · Cited by 60
Trastuzumab containing regimens for early breast cancer
Moja, L., Brambilla, C., Compagnoni, A., & Pistotti, V. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006243 · Cited by 73
Mohamed, R. N., Basha, S., & Al-Thomali, Y. (2018) · The Angle Orthodontist · doi:10.2319/091717-624.1 · Cited by 71
Taylor, C., McHugh, C., Mockler, D., Minogue, C. M., Reilly, R. B., & Fleming, N. (2021) · PLoS ONE · doi:10.1371/journal.pone.0260166 · Cited by 82
“The D&B Checklist has also been recommended for use in assessing non-RCTs due to its psychometric properties [63, 64].”
MethodLaparoscopic surgery for ulcerative colitis – a meta‐analysis
Tan, J. J. Y., & Tjandra, J. J. (2006) · Colorectal Disease · doi:10.1111/j.1463-1318.2006.00971.x · Cited by 71
Norris, S. L., Atkins, D. C., Bruening, W., Fox, S. B., Johnson, E., Kane, R. L., Morton, S. C., Oremus, M., Ospina, M. B., Randhawa, G. S., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.027 · Cited by 73
Systematic review: Heliocobacter pylori infection and impaired drug absorption
LAHNER, E., ANNIBALE, B., & DELLE FAVE, G. (2009) · Alimentary Pharmacology & Therapeutics · doi:10.1111/j.1365-2036.2008.03906.x · Cited by 72
Loesch, D. Z., Godler, D. E., Evans, A. H., Bui, M., Gehling, F., Kotschet, K. E., Trost, N. M., Storey, E., Stimpson, P., Kinsella, G., et al. (2011) · Genetics in Medicine · doi:10.1097/gim.0b013e3182064362 · Cited by 77
Does This Patient Have an Exudative Pleural Effusion?
Wilcox, M. E., Chong, C. A. K. Y., Stanbrook, M. B., Tricco, A. C., Wong, C., & Straus, S. E. (2014) · JAMA · doi:10.1001/jama.2014.5552 · Cited by 71
Economic determinants of diet in older adults: systematic review
Conklin, A., Maguire, E. R., & Monsivais, P. (2013) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2013-202513 · Cited by 71
“Intervention studies were excluded unless measuring diet as a function of fi nancial incentives or change in another economic factor(s) (eg, reduced food price, subsidies, or coupons).”
Recio‐Saucedo, A., Gerty, S. M., Foster, C. L., Eccles, D., & Cutress, R. (2015) · The Breast · doi:10.1016/j.breast.2015.11.001 · Cited by 71
Beck‐Fruchter, R., Shalev, E., & Weiss, A. (2015) · Reproductive BioMedicine Online · doi:10.1016/j.rbmo.2015.12.001 · Cited by 73
Sethi, N. J., Feinberg, J. B., Nielsen, E. E., Safi, S., Gluud, C. N., & Jakobsen, J. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0186856 · Cited by 82
Depression and metabolic syndrome in the older population: A review of evidence
Repousi, N., Masana, M. F., Niubo, A. S., Haro, J. M., & Tyrovolas, S. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.04.102 · Cited by 83
Zampieri, F. G., Mendes, P. V., Ranzani, O., Taniguchi, L. U., Azevedo, L. C. P., Costa, E. L. V., & Park, M. (2013) · Journal of Critical Care · doi:10.1016/j.jcrc.2013.07.047 · Cited by 77
“Besides a high grade, all included case-control studies required 2 stars on the item “Comparability” of the Newcastle-Ottawa Scale.”
BackgroundHospital delivery volume, severe obstetrical morbidity, and failure to rescue
Friedman, A. M., Ananth, C. V., Huang, Y., D’Alton, M. E., & Wright, J. D. (2016) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2016.07.039 · Cited by 70
Glaser, B. E., & Bero, L. A. (2005) · Science and Engineering Ethics · doi:10.1007/s11948-005-0026-z · Cited by 85
Gartlehner, G., Hansen, R. A., Jonas, B., Thieda, P., & Lohr, K. N. (2007) · Clinical Rheumatology · doi:10.1007/s10067-007-0654-6 · Cited by 70
A Systematic Review of Yoga for Balance in a Healthy Population
Jeter, P. E., Nkodo, A.-F., Moonaz, S. H., & Dagnelie, G. (2014) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2013.0378 · Cited by 78
Treating Pit-and-Fissure Caries: A Systematic Review and Network Meta-analysis
Schwendicke, F., Jäger, A. M., Paris, S., Hsu, L. Y., & Tu, Y. (2015) · Journal of Dental Research · doi:10.1177/0022034515571184 · Cited by 72
“Including nonrandomized studies was performed to increase comprehensiveness of the review and has been shown to not generally introduce systematic bias (Deeks et al. 2003).”
BackgroundDaglas, R., Yücel, M., Cotton, S., Allott, K., Hetrick, S., & Berk, M. (2015) · International Journal of Bipolar Disorders · doi:10.1186/s40345-015-0024-2 · Cited by 73
Hills, A., Donaghue, C., Waters, J. J., Waters, K. S., Sullivan, C., Kulkarni, A., Docherty, Z., Mann, K., & Ogilvie, C. M. (2010) · Prenatal Diagnosis · doi:10.1002/pd.2503 · Cited by 70
Pyke, T. L., Osmotherly, P. G., & Baines, S. (2016) · Clinical Journal of Pain · doi:10.1097/ajp.0000000000000474 · Cited by 77
Resch, C., Rosema, S., Hurks, P. P. M., de Kloet, A. J., & Heugten, C. M. V. (2018) · Brain Injury · doi:10.1080/02699052.2018.1458335 · Cited by 82
“quality assessment tools and useful in systematic reviews (26).”
BackgroundBergmann, S., Li, B., Pilot, E., Chen, R., Wang, B., & Yang, J. (2020) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2020.136985 · Cited by 75
Iversen, L. H., Harling, H., Laurberg, S., & Wille‐Jørgensen, P. (2006) · Colorectal Disease · doi:10.1111/j.1463-1318.2006.01100.x · Cited by 70
Fernández, P., Vallejo, G., Livacic-Rojas, P. E., & Herrero, E. T. (2014) · Anales de Psicología · doi:10.6018/analesps.30.2.166911 · Cited by 147
Reeves, B. C., & Murphy, G. J. (2008) · Current Opinion in Anaesthesiology · doi:10.1097/aco.0b013e32830dd087 · Cited by 69
Bambra, C. L., Hillier-Brown, F. C., Cairns, J., Kasim, A., Moore, H. J., & Summerbell, C. (2015) · Public Health Research · doi:10.3310/phr03010 · Cited by 87
Challenges in Systematic Reviews of Therapeutic Devices and Procedures
Hartling, L., McAlister, F. A., Rowe, B. H., Ezekowitz, J., Friesen, C., & Klassen, T. P. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-142-12_part_2-200506211-00010 · Cited by 70
“On the basis of the existing evidence, Deeks and colleagues (27) could not make firm conclusions regarding the value of randomization because of the conflicting results and limitations in the studies reviewed.”
BackgroundMethodInfluentialBullard, M., Villa‐Roel, C., Guo, X., Holroyd, B., Innes, G., Schull, M., Vandermeer, B., Ospina, M., & Rowe, B. (2011) · Emergency Medicine Journal · doi:10.1136/emj.2010.103598 · Cited by 74
Rizvi, A., Seyedsaadat, S. M., Murad, M. H., Brinjikji, W., Fitzgerald, S. T., Kadirvel, R., Rabinstein, A. A., & Kallmes, D. F. (2018) · Journal of NeuroInterventional Surgery · doi:10.1136/neurintsurg-2018-013950 · Cited by 69
Barbonetti, A., Martorella, A., Minaldi, E., D’Andrea, S., Bardhi, D., Castellini, C., Francavilla, F., & Francavilla, S. (2019) · Frontiers in Endocrinology · doi:10.3389/fendo.2019.00164 · Cited by 69
Effects of active video games on children and adolescents: A systematic review with meta‐analysis
Oliveira, C. B., Pinto, R. Z., Saraiva, B. T. C., Tebar, W. R., Delfino, L. D., Franco, M. R., Silva, C. C. M., & Christofaro, D. G. D. (2020) · Scandinavian Journal of Medicine & Science in Sports · doi:10.1111/sms.13539 · Cited by 70
Mamikutty, R., Aly, A. S., & Marhazlinda, J. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18168623 · Cited by 75
“Hence, choosing the most appropriate tools for assessing the ROB for observational studies is not easy [6,7].”
BackgroundMethodReese, G., Purkayastha, S., Tilney, H. S., von Roon, A. C., Yamamoto, T., & Tekkis, P. P. (2006) · Colorectal Disease · doi:10.1111/j.1463-1318.2006.01114.x · Cited by 78
The Critical Role Of Observational Evidence In Comparative Effectiveness Research
Fleurence, R. L., Naci, H., & Jansen, J. P. (2010) · Health Affairs · doi:10.1377/hlthaff.2010.0630 · Cited by 68
Kai, J., Ulph, F., Cullinan, T., & Qureshi, N. (2009) · Health Technology Assessment · doi:10.3310/hta13570 · Cited by 72
Systematic review: body composition in children with inflammatory bowel disease
Thangarajah, D., Hyde, M. J., Konteti, V. K. S., Santhakumaran, S., Frost, G. S., & Fell, J. M. E. (2015) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.13218 · Cited by 68
Melanoma screening: A plan for improving early detection
Shellenberger, R. A., Nabhan, M., & Kakaraparthi, S. (2016) · Annals of Medicine · doi:10.3109/07853890.2016.1145795 · Cited by 72
“Cohort and case-controlled trials were evaluated based on the Newcastle-Ottawa Quality Assessment Tool (22) See Table 1 for the Quality Assessment Worksheet.”
MethodFry, C. E., Langley, K., & Shelton, K. H. (2016) · Child Neuropsychology · doi:10.1080/09297049.2016.1207758 · Cited by 88
“In their comprehensive review of quality assessment tools, Deeks et al. (2003) recommended only the NOS and 5 other tools for use in systematic reviews out of 194 tools identified, based on their coverage of core internal validity criteria.”
Arditi, C., Burnand, B., & Peytremann-Bridevaux, I. (2016) · BMC Health Services Research · doi:10.1186/s12913-016-1816-5 · Cited by 68
“Yet, reviews and metareviews have shown that non-randomised studies do not systematically lead to biased results and that there is little evidence for significant effect estimate differences between observational studies and RCTs [14, 19].”
BackgroundInfluentialHernández, V., Linares, E., Dunn, J., MacLennan, S., Lam, T. B. L., Yuan, Y., Compérat, É., Cowan, N. C., Gakis, G., Lebrét, T., et al. (2017) · Urologic Oncology Seminars and Original Investigations · doi:10.1016/j.urolonc.2017.04.013 · Cited by 76
“A modified version of the RoB assessment tool was used in assessing NRCSs with the addition of further domains to assess risk of confounding [12].”
MethodDee, M., Lennon, O. C., & O'Sullivan, C. M. (2018) · Disability and Rehabilitation · doi:10.1080/09638288.2018.1501617 · Cited by 81
“This tool was previously identified in a comprehensive review as one of the most appropriate quality assessment tools to use in systematic reviews that include non-randomised trials [26].”
MethodJaques, A. M., Halliday, J. L., Francis, I., Bonacquisto, L., Forbes, R., Cronin, A., & Sheffield, L. (2007) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2007.01349.x · Cited by 68
Uptake of prenatal diagnostic testing and the effectiveness of prenatal screening for Down syndrome
Jaques, A. M., Collins, V., Muggli, E., Amor, D. J., Francis, I., Sheffield, L. J., & Halliday, J. L. (2010) · Prenatal Diagnosis · doi:10.1002/pd.2509 · Cited by 68
Li, Y.-F., Zhou, K.-Y., Fang, J., Wang, C., Hua, Y.-M., & Mu, D.-Z. (2016) · World Journal of Pediatrics · doi:10.1007/s12519-016-0016-z · Cited by 67
Bang, C. S., & Song, I. H. (2017) · BMC Gastroenterology · doi:10.1186/s12876-017-0606-9 · Cited by 67
“Each parameter consists of subcategorized questions: selection (n = 4), comparability (n = 1), and exposure or outcome (n = 3) [6, 7].”
BackgroundMnich, C., Weyland, S., Jekauc, D., & Schipperijn, J. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16214266 · Cited by 65
“The EPHP tool has hown to be suitable for use in systematic reviews [45] and has fair inter-rater reliability and excel ent agre ment for the final rating [43].”
MethodBiomechanics of lower limb in badminton lunge: a systematic scoping review
Lam, W., Wong, D. W., & Lee, W. C. C. (2020) · PeerJ · doi:10.7717/peerj.10300 · Cited by 82
“due to ambiguity, as recommended in the literature (Deeks et al., 2003).”
BackgroundAn analysis of quality of systematic reviews on pharmacist health interventions
Melchiors, A. C., Correr, C. J., Venson, R., & Pontarolo, R. (2011) · International Journal of Clinical Pharmacy · doi:10.1007/s11096-011-9592-0 · Cited by 76
Cementless total hip arthroplasty in rheumatoid arthritis: a systematic review of the literature
Zwartelé, R. E., Witjes, S., Doets, H. C., Stijnen, T., & Pöll, R. G. (2011) · Archives of Orthopaedic and Trauma Surgery · doi:10.1007/s00402-011-1432-0 · Cited by 72
Cotchett, M., Landorf, K. B., & Munteanu, S. E. (2010) · Journal of Foot and Ankle Research · doi:10.1186/1757-1146-3-18 · Cited by 67
“Second, Item 25 was removed as it has been shown that case mix adjustment cannot reduce the extent of bias in non-randomised trials [19].”
BackgroundMethodDietary intake and diabetic retinopathy: A systematic review
Wong, M. Y. Z., Man, R. E. K., Fenwick, E., Gupta, P., Li, L., van Dam, R. M., Chong, M. F., & Lamoureux, E. L. (2018) · PLoS ONE · doi:10.1371/journal.pone.0186582 · Cited by 81
Laparoscopic vs. Open Surgery for Diverticular Disease: A Meta-Analysis of Nonrandomized Studies
Purkayastha, S., Constantinides, V. A., Tekkis, P. P., Athanasiou, T., Aziz, O., Tilney, H. S., Darzi, A. W., & Heriot, A. G. (2006) · Diseases of the Colon & Rectum · doi:10.1007/s10350-005-0316-1 · Cited by 68
The case for randomized controlled trials to assess the impact of clinical information systems
Liu, J. L. Y., & Wyatt, J. C. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.010306 · Cited by 66
Babu, A. S., Ramachandran, P., Maiya, A. G., Mohapatra, A. K., & Kamath, R. (2015) · Heart Lung and Circulation · doi:10.1016/j.hlc.2015.10.015 · Cited by 66
“[16,17] The scale has five sub-domains which include reporting, external validity, bias, confounding and power being scored out of 11, 3, 7, 6 and 5 respectively.”
BackgroundHyaluronic Acid Is an Effective Dermal Filler for Lip Augmentation: A Meta-Analysis
Czumbel, L. M., Farkasdi, S., Gede, N., Mikó, A., Csupor, D., Lukács, A., Gaál, V., Kiss, S., Hegyi, P., & Varga, G. (2021) · Frontiers in Surgery · doi:10.3389/fsurg.2021.681028 · Cited by 69
“Cohort studies were evaluated based on the Newcastle Ottawa Scale (NOS) for Cohort Studies (39).”
MethodSilva-Fernández, C. S., de la Calle, M., Arribas, S. M., Garrosa, E., & Ramiro‐Cortijo, D. (2023) · Nursing Reports · doi:10.3390/nursrep13040130 · Cited by 99
“This tool has been judged suitable to be used in systematic reviews of effectiveness [31].”
MethodPaulus, J. K., Dahabreh, I. J., Balk, E. M., Avendano, E. E., Lau, J., & Ip, S. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1101 · Cited by 62
“…focusing on historically controlled studies of cancer treatments (Diehl and Perry, 1986) andmore recent research on vascular disease populations (Deeks et al., 2003) also support the contention that such studies often favor recently introduced treatments because of underlying time trends in…”
MethodMichaels, J., Campbell, W., King, B., MacIntyre, J., Palfreyman, S., Shackley, P., & Stevenson, M. (2009) · Health Technology Assessment · doi:10.3310/hta13560 · Cited by 66
Kromer, T. O., de Bie, R. A., & Bastiaenen, C. H. G. (2010) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-11-114 · Cited by 91
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.