Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
(1963) · Medical Entomology and Zoology · Cited by 15,085
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
Items 601–700 of 1,024
Rehabilitation in Patients with Dementia Following Hip Fracture: A Systematic Review
Allen, J. P., Koziak, A., Buddingh, S., Liang, J., Buckingham, J., & Beaupré, L. A. (2011) · Physiotherapy Canada · doi:10.3138/ptc.2011-06bh · Cited by 90
Using quality assessment tools to critically appraise ageing research: a guide for clinicians
Harrison, J. K., Reid, J., Quinn, T., & Shenkin, S. D. (2016) · Age and Ageing · doi:10.1093/ageing/afw223 · Cited by 117
Mentalization‐based treatment and its evidence‐base status: A systematic literature review
Malda‐Castillo, J., Browne, C., & Perez‐Algorta, G. (2019) · Psychology and Psychotherapy: Theory, Research and Practice · doi:10.1111/papt.12195 · Cited by 89
The relationship between diabetic retinopathy and psychosocial functioning: a systematic review
Khoo, K., Man, R. E. K., Rees, G., Gupta, P., Lamoureux, E. L., & Fenwick, E. K. (2019) · Quality of Life Research · doi:10.1007/s11136-019-02165-1 · Cited by 89
“After synthesising the results from 22 studies rated as “high quality” by either the NOS or the modified Downs and Black checklist, the findings remained similar to those of the full systematic review (n = 42 studies).”
BackgroundInfluentialChase, D., Rosten, C., Turner, S., Hicks, N., & Milne, R. (2009) · Health Technology Assessment · doi:10.3310/hta13590 · Cited by 92
Peinemann, F., Tushabe, D. A., & Kleijnen, J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0085035 · Cited by 89
“Cheng 2012 [71] same no Choi 2012 [72] same no Clagett 1984 [73] same no Colditz 1989 [74] various yes Conaty 2004 [75] same no Concato 2000 [76] same no Deeks 2003 [77] various various Edwards 2012 [78] various no Flossman 2007 [79] same no Franklin 2000 [80] same no Furlan (Exam) 2008 [81] same various Furlan (Meth)…”
Romero, R. J., Erez, O., Maymon, E., Chaemsaithong, P., Xu, Z., Pacora, P., Chaiworapongsa, T., Done, B., Hassan, S. S., & Tarca, A. L. (2017) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2017.02.037 · Cited by 98
Treatments for internet addiction, sex addiction and compulsive buying: A meta-analysis
Goslar, M., Leibetseder, M. M., Münch, H. M., Hofmann, S. G., & Laireiter, A. (2020) · Journal of Behavioral Addictions · doi:10.1556/2006.2020.00005 · Cited by 115
“, 2004) and is recommended for systematic reviews and metaanalyses (Deeks et al., 2003).”
BackgroundMethodAhmad, M., Bani Mohammad, E., & Anshasi, H. A. (2020) · Pain Management Nursing · doi:10.1016/j.pmn.2020.04.002 · Cited by 89
European Academy of Neurology guidelines on the treatment of cluster headache
May, A., Evers, S., Goadsby, P. J., Leone, M., Manzoni, G. C., Pascual, J., Carvalho, V., Romoli, M., Aleksovska, K., Pozo‐Rosich, P., et al. (2023) · European Journal of Neurology · doi:10.1111/ene.15956 · Cited by 89
“Risk of bias was evaluated for all controlled trials published after the cutoff date of highquality systematic reviews using tools recommended by the Cochrane Handbook [20], Cochrane Risk of Bias tool for randomized controlled trials, and the Downs and Black Scale [21] for observational studies.”
MethodDucreux, M. P., Dahan, L., Smith, D. M., O’Toole, D., Lepère, C., Dromain, C., Vilgrain, V., Baudin, É., Lombard‐Bohas, C., Scoazec, J., et al. (2014) · European Journal of Cancer · doi:10.1016/j.ejca.2014.10.002 · Cited by 89
“The main limitation of our study is its non-randomised and non-comparative nature against a 5-FU/streptozocin arm, as randomisation is the sole way to avoid selection bias [32].”
Monje, A., Alcoforado, G. A. P., Padial‐Molina, M., Suárez, F., Lin, G., & Wang, H. (2014) · Journal of Periodontology · doi:10.1902/jop.2014.140135 · Cited by 92
Colquitt, J., Jones, J., Harris, P., Loveman, E., Bird, A., Clegg, A., Baguley, D., Proops, D., Mitchell, T., Sheehan, P., et al. (2011) · Health Technology Assessment · doi:10.3310/hta15260 · Cited by 90
Association Between Gout and All-Cause as well as Cardiovascular Mortality: A Systematic Review
Lottmann, K., Chen, X., & Schädlich, P. K. (2012) · Current Rheumatology Reports · doi:10.1007/s11926-011-0234-2 · Cited by 89
Kendrick, D., Barlow, J., Hampshire, A. J., Stewart-Brown, S. L., & Polnay, L. (2008) · Child Care Health and Development · doi:10.1111/j.1365-2214.2008.00849.x · Cited by 92
Grati, F. R., Barlocco, A., Grimi, B., Milani, S., Frascoli, G., Meco, A. M. D., Liuti, R., Trotta, A., Chinetti, S., Dulcetti, F., et al. (2010) · American Journal of Medical Genetics Part A · doi:10.1002/ajmg.a.33370 · Cited by 89
Clarke, M. J., & Loudon, K. (2011) · Trials · doi:10.1186/1745-6215-12-16 · Cited by 98
“As others have shown, the correction for potential confounders in non-randomised studies does not guarantee that selection biases will be overcome [19].”
BackgroundBassuk, E. L., DeCandia, C. J., Tsertsvadze, A., & Richard, M. K. (2014) · American Journal of Orthopsychiatry · doi:10.1037/ort0000020 · Cited by 107
“Overall mental health was measured using the Brief Symptoms Inventory (BSI; Derogatis, 1993); screening for Posttraumatic Stress Disorder was conducted using the PTSD Scale (Blake et al., 1995); substance abuse was measured using Alcohol, Smoking and Substance Involvement Screening Test (ASSIST;…”
MethodMcHutchison, C. A., Backhouse, E. V., Cvoro, V., Shenkin, S. D., & Wardlaw, J. M. (2017) · Epidemiology · doi:10.1097/ede.0000000000000675 · Cited by 86
Schneider, A., & Weigl, M. (2018) · PLOS ONE · doi:10.1371/journal.pone.0197375 · Cited by 86
Outcomes of intuitive eating interventions: a systematic review and meta-analysis
Babbott, K. M., Cavadino, A., Brenton-Peters, J., Consedine, N. S., & Roberts, M. (2022) · Eating Disorders · doi:10.1080/10640266.2022.2030124 · Cited by 102
“This checklist is highly regarded as a quality assessment tool and was selected for this research due to its usefulness in evaluating both randomised and non-randomised studies using the same stringent criteria (Deeks et al., 2003).”
MethodDoctors’ attitudes and beliefs regarding acute low back pain management: A systematic review
Fullen, B. M., Baxter, D. G., O'Donovan, B. G. G., Doody, C., Daly, L., & Hurley, D. A. (2008) · Pain · doi:10.1016/j.pain.2008.01.003 · Cited by 98
“tative studies, as a review of the literature [10] identified this tool as one of the most appropriate for assessing non-randomised studies as well as RCTs and it has been used in 29 previous systematic reviews [35].”
MethodChoice of fluids for resuscitation in children with severe infection and shock: systematic review
Akech, S., Ledermann, H., & Maitland, K. (2010) · BMJ · doi:10.1136/bmj.c4416 · Cited by 87
Sinonasal Adenoid Cystic Carcinoma
Husain, Q. M., Kanumuri, V. V., Svider, P. F., Radvansky, B. M., Boghani, Z., Liu, J. K., & Eloy, J. A. (2012) · Otolaryngology · doi:10.1177/0194599812464020 · Cited by 88
Inamdar, A. S., Croucher, R. E., Chokhandre, M. K., Mashyakhy, M. H., & Marinho, V. C. C. (2015) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntu255 · Cited by 94
Bhangu, A. A., Fitzgerald, J. E. F., Singh, P. M., Battersby, N. J., Marriott, P. J., & Pinkney, T. D. (2013) · Hernia · doi:10.1007/s10029-013-1119-2 · Cited by 92
“Assessing trial quality, risk of bias and outcome reporting is controversial, can be difficult and is often inadequately performed [19, 20].”
BackgroundEngebretsen, K. B., Grotle, M., Bautz‐Holter, E., Ekeberg, O. M., Juel, N. G., & Brox, J. I. (2010) · Physical Therapy · doi:10.2522/ptj.20090338 · Cited by 97
Physical Activity Levels of Children With Down Syndrome
Fox, B., Moffett, G. E., Kinnison, C., Brooks, G., & Case, L. E. (2018) · Pediatric Physical Therapy · doi:10.1097/pep.0000000000000556 · Cited by 87
Police crackdowns on illegal gun carrying: a systematic review of their impact on gun crime
Koper, C. S., & Mayo‐Wilson, E. (2006) · Journal of Experimental Criminology · doi:10.1007/s11292-006-9005-x · Cited by 90
Szpunar, M. J., & Parry, B. L. (2017) · Archives of Women's Mental Health · doi:10.1007/s00737-017-0787-9 · Cited by 85
Sulphonylurea monotherapy for patients with type 2 diabetes mellitus
Hemmingsen, B., Schroll, J. B., Lund, S. S., Wetterslev, J., Gluud, C., Vaag, A., Sonne, D. P., Lundstrøm, L. H., & Almdal, T. P. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009008.pub2 · Cited by 88
RetractedUptake of noninvasive prenatal testing at a large academic referral center
Larion, S., Warsof, S. L., Romary, L., Mlynarczyk, M., Peleg, D., & Abuhamad, A. (2014) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2014.06.038 · Cited by 86
A systematic review and meta-analysis of outpatient treatment for acute diverticulitis
van Dijk, S. T., Bos, K., de Boer, M. G. J., Draaisma, W. A., van Enst, W. A., Felt-Bersma, R. J. F., Klarenbeek, B. R., Otte, J. A., Puylaert, J. B. C. M., van Geloven, A. A. W., et al. (2018) · International Journal of Colorectal Disease · doi:10.1007/s00384-018-3015-9 · Cited by 90
Hawthorne effect on surgical studies
Demetriou, C., Hu, L., Smith, T. O., & Hing, C. B. (2019) · ANZ Journal of Surgery · doi:10.1111/ans.15475 · Cited by 101
Small Intestinal Bacterial Overgrowth in Functional Dyspepsia: A Systematic Review and Meta-Analysis
Gurusamy, S. R., Shah, A., Talley, N. J., Koloski, N. A., Jones, M. P., Walker, M. M., Morrison, M., & Holtmann, G. (2021) · The American Journal of Gastroenterology · doi:10.14309/ajg.0000000000001197 · Cited by 91
“In addition, the quality of the case-control included studies was assessed using the Newcastle-Ottawa scale (NOS) which judges the selection of the study groups, the comparability of the groups, and the ascertainment of the exposure of interest, to assign a maximum score of 9 stars (18).”
Khavandegar, A., Heidarzadeh, A., Angoorani, P., Hasani‐Ranjbar, S., Ejtahed, H., Larijani, B. A., & Qorbani, M. (2024) · BMC Medical Genomics · doi:10.1186/s12920-024-01861-3 · Cited by 102
Hummel, S., Simpson, E., Hemingway, P., Stevenson, M., & Rees, A. (2010) · Health Technology Assessment · doi:10.3310/hta14470 · Cited by 98
Flexible working conditions and their effects on employee health and wellbeing
Joyce, K., Critchley, J. A., & Bambra, C. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008009 · Cited by 83
FMR1 premutation and full mutation molecular mechanisms related to autism
Hagerman, R., Au, J., & Hagerman, P. (2011) · Journal of Neurodevelopmental Disorders · doi:10.1007/s11689-011-9084-5 · Cited by 85
Screening for pancreatic cancer in familial high-risk individuals: A systematic review
Lu, C. (2015) · World Journal of Gastroenterology · doi:10.3748/wjg.v21.i28.8678 · Cited by 86
Schultz, C. H., Fairley, R., Murphy, L. S.-L., & Doss, M. (2020) · Prehospital and Disaster Medicine · doi:10.1017/s1049023x1900520x · Cited by 93
Pilgrim, H., Lloyd-Jones, M., & Rees, A. (2009) · Health Technology Assessment · doi:10.3310/hta13100 · Cited by 102
Nava, F., Tramacere, I., Fittipaldo, V. A., Bruzzone, M. G., DiMeco, F., Fariselli, L., Finocchiaro, G., Pollo, B., Salmaggi, A., Silvani, A., et al. (2014) · Neuro-Oncology · doi:10.1093/neuonc/not316 · Cited by 89
“and detection bias.(45,46) However, applying a multivariable approach in design and analysis, as we did, to determine outcome predictors boosts confidence in the reliability of the estimates thus derived.”
MethodOutcomes of vertebroplasty compared with kyphoplasty: a systematic review and meta-analysis
Gu, C. N., Brinjikji, W., Evans, A. J., Murad, M. H., & Kallmes, D. F. (2015) · Journal of NeuroInterventional Surgery · doi:10.1136/neurintsurg-2015-011714 · Cited by 85
The Necessity of Randomized Clinical Trials
Jakobsen, J. (2013) · British Journal of Medicine and Medical Research · doi:10.9734/bjmmr/2013/3208 · Cited by 84
“It is only in retrospect it may be concluded that interventions have been validly assessed by observational studies [5], and evidence based on observational evidence will in most circumstances be uncertain [18-20].”
BackgroundResultInfluentialMari, A., Kimura, S., Foerster, B., Abufaraj, M. M., D’Andrea, D., Hassler, M. R., Minervini, A., Morgan Rouprêt, A., Babjuk, M. M., & Shariat, S. F. (2018) · British Journal of Urology · doi:10.1111/bju.14417 · Cited by 84
Tuleasca, C., Paddick, I., Hopewell, J. W., Jones, B., Millar, W. T., Hamdi, H., Porcheron, D., Levivier, M., & Régis, J. M. (2019) · World Neurosurgery · doi:10.1016/j.wneu.2019.10.021 · Cited by 84
Sgreccia, D., Manicardi, M., Malavasi, V. L., Vitolo, M., Valenti, A. C., Proietti, M., Lip, G. Y. H., & Boriani, G. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10173979 · Cited by 80
Cook, N. E., & Gorraiz, M. (2016) · Child and Adolescent Mental Health · doi:10.1111/camh.12112 · Cited by 83
Edwards, D. J., Noyes, J., Lowes, L., Spencer, L. H., & Gregory, J. W. (2014) · BMC Pediatrics · doi:10.1186/1471-2431-14-228 · Cited by 111
“Key aspects of quality for non randomised intervention studies in stream 1 were based on the work of Deek et al. [33] (see p39 of the Centre for Reviews and Dissemination, University of York guidance on undertaking reviews in health care [34] ).”
MethodA meta-analysis of case-control studies of the association of migraine and patent foramen ovale
Takagi, H., & Umemoto, T. (2015) · Journal of Cardiology · doi:10.1016/j.jjcc.2015.09.016 · Cited by 92
McElnay, P., Choong, A. M. T. L., Jordan, E., Song, F., & Lim, E. K. S. (2015) · Thorax · doi:10.1136/thoraxjnl-2014-206292 · Cited by 84
Self-citationWillmott, T. J., Pang, B., Rundle-Thiele, S., & Badejo, A. (2019) · Journal of Medical Internet Research · doi:10.2196/10265 · Cited by 81
Ward, R. J., Bristow, S. J., Kovshoff, H., Cortese, S., & Kreppner, J. (2020) · Journal of Attention Disorders · doi:10.1177/1087054720972801 · Cited by 114
“Secondly, there are only few well-developed tools to assess risk of bias in non-randomized studies (Deeks et al., 2003; Reed et al., 2005), particularly when a number of different study designs are included (Deeks et al.”
BackgroundMethodInfluentialSaxton, J., Rodda, S. N., Booth, N., Merkouris, S., & Dowling, N. A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248262 · Cited by 94
“As we only included RCTs, we also expected any baseline imbalances to be random, affecting intervention and control groups approximately equally.”
BackgroundInfluentialClomiphene citrate for male infertility: A systematic review and meta‐analysis
Huijben, M., Huijsmans, R., Lock, M., de Kemp, V. D., de Kort, L., & van Breda, H. V. (2023) · Andrology · doi:10.1111/andr.13388 · Cited by 64
Micronutrients (Other than iron) and Helicobacter pylori Infection: A Systematic Review
Lahner, E., Persechino, S., & Annibale, B. (2012) · Helicobacter · doi:10.1111/j.1523-5378.2011.00892.x · Cited by 87
Siersma, V., Als‐Nielsen, B., Chen, W., Hilden, J., Gluud, L. L., & Gluud, C. (2007) · Statistics in Medicine · doi:10.1002/sim.2752 · Cited by 84
Efficacy of ω-3 Fatty Acids in Preventing Age-Related Macular Degeneration
Hodge, W., Schachter, H., Barnes, D., Pan, Y., Lowcock, E. C., Zhang, L., Sampson, M., Morrison, A., Tran, K. T., Miguelez, M., et al. (2006) · Ophthalmology · doi:10.1016/j.ophtha.2006.02.043 · Cited by 91
Konialis, C. P., Hagnefelt, B., Sevastidou, S., Karapanou, S., Pispili, K., Markaki, A., & Pángalos, C. (2011) · Prenatal Diagnosis · doi:10.1002/pd.2750 · Cited by 82
First trimester serum tests for Down's syndrome screening
Alldred, S. K., Takwoingi, Y., Guo, B., Pennant, M. E., Deeks, J. J., Neilson, J. P., & Alfirević, Ž. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011975 · Cited by 87
Self-citationHeitmann, J., Bennik, E. C., van Hemel-Ruiter, M. E., & de Jong, P. J. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0822-6 · Cited by 94
The Aging Muscle in Experimental Bed Rest: A Systematic Review and Meta-Analysis
Girolamo, F. G. D., Fiotti, N., Milanović, Z., Situlin, R., Mearelli, F., Vinci, P., Šimunič, B., Pišot, R., Narici, M. V., & Biolo, G. (2021) · Frontiers in Nutrition · doi:10.3389/fnut.2021.633987 · Cited by 102
“Quality of the Included Studies Several tools are available to estimate the quality of randomized, controlled (56), and non-randomized trials or observational studies (57).”
Shah, A., Fairlie, T., Brown, G., Jones, M. P., Eslick, G. D., Duncanson, K., Thapar, N., Keely, S., Koloski, N. A., Shahi, M., et al. (2022) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2022.01.014 · Cited by 90
Guest, E., Zucchelli, F., Costa, B., Bhatia, R., Halliwell, E., & Harcourt, D. (2022) · Body Image · doi:10.1016/j.bodyim.2022.04.009 · Cited by 78
Stevens, B., Buettner, P., Watt, K., Clough, A., Brimblecombe, J., & Judd, J. (2015) · Maternal & Child Nutrition · doi:10.1111/mcn.12183 · Cited by 82
Thomas, R., Lee, S. C. L., Patole, S., & Rao, S. C. (2011) · British Journal of Neurosurgery · doi:10.3109/02688697.2011.603856 · Cited by 82
Clegg, A. J., Loveman, E., Gospodarevskaya, E., Harris, P. S., Bird, A., Bryant, J., Scott, D. A., Davidson, P., Little, P. S., & Coppin, R. (2010) · Health Technology Assessment · doi:10.3310/hta14280 · Cited by 87
The International Stroke Trial database
the International Stroke Trial Collaborative Group, Sandercock, P. A., Niewada, M., & Członkowska, A. (2011) · Trials · doi:10.1186/1745-6215-12-101 · Cited by 82
Potter, M. D. E., Brienesse, S. C., Walker, M. M., Boyle, A., & Talley, N. J. (2017) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.14039 · Cited by 82
Helicopter emergency medical services for adults with major trauma
Galvagno, S. M., Thomas, S. H., Stephens, C. T., Haut, E. R., Hirshon, J. M., Floccare, D. J., & Pronovost, P. J. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009228.pub2 · Cited by 82
Mador, R. L., & Shaw, N. T. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2009.01.002 · Cited by 84
Genetic causes of developmental disorders
Vorstman, J. A. S., & Ophoff, R. A. (2013) · Current Opinion in Neurology · doi:10.1097/wco.0b013e32835f1a30 · Cited by 80
Kim, K., Ozegovic, D., & Voaklander, D. C. (2012) · Injury Prevention · doi:10.1136/injuryprev-2011-040306 · Cited by 89
Ren, S. P., Newby, D., Li, S. C., Walkom, E. J., Miller, P., Hure, A. J., & Attia, J. (2015) · Open Heart · doi:10.1136/openhrt-2015-000247 · Cited by 81
“Each study was assessed using the validated Newcastle-Ottawa Quality Assessment Scale 13 for cohort studies and case – control studies.”
MethodMarks, D., Comans, T. A., Bisset, L., & Scuffham, P. (2016) · Physiotherapy · doi:10.1016/j.physio.2016.11.006 · Cited by 100
Chaplin, D., Cook, T. D., Zurovac, J., Coopersmith, J., Finucane, M. M., Vollmer, L., & Morris, R. E. (2018) · Journal of Policy Analysis and Management · doi:10.1002/pam.22051 · Cited by 131
Armstrong, E., Eggins, E., Reid, N., Harnett, P. H., & Dawe, S. (2017) · Journal of Experimental Criminology · doi:10.1007/s11292-017-9290-6 · Cited by 98
“The quality of quasi-experimental designs varies considerably and can inconsistently impact effect sizes (Deeks et al. 2003; Lipsey and Wilson 1993; Shadish and Heinsman 1997).”
A Systematic Review of COVID-19 and Kidney Transplantation
Mahalingasivam, V., Craik, A., Tomlinson, L. A., Ge, L., Hou, L., Wang, Q., Yang, K., Fogarty, D. G., & Keenan, C. (2020) · Kidney International Reports · doi:10.1016/j.ekir.2020.10.023 · Cited by 86
Lareyre, O., Gourlan, M., Stœbner-Delbarre, A., & Cousson‐Gélie, F. (2020) · Preventive Medicine · doi:10.1016/j.ypmed.2020.106327 · Cited by 91
Reynard, S., Dias, J., Mitić, M., Schrank, B., & Woodcock, K. A. (2022) · JMIR Serious Games · doi:10.2196/31456 · Cited by 110
Tew, G. A., Posso, M., Arundel, C., & McDaid, C. (2015) · Occupational Medicine · doi:10.1093/occmed/kqv044 · Cited by 91
Chronic Non-specific Low Back Pain and Motor Control During Gait
Koch, C., & Hänsel, F. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02236 · Cited by 99
Lund, C. A., Møller, A. M., Wetterslev, J., & Lundstrøm, L. H. (2014) · PLoS ONE · doi:10.1371/journal.pone.0099308 · Cited by 87
“Confounding by indication is well known to introduce bias in the results in any nonrandomized study involving interventions [25].”
BackgroundJameson, S. S., Charman, S. C., GREGG, P. J., Reed, M. R., & van der Meulen, J. (2011) · Journal of Bone and Joint Surgery - British Volume · doi:10.1302/0301-620x.93b11.27622 · Cited by 83
Premutation CGG-repeat expansion of the Fmr1 gene impairs mouse neocortical development
Cunningham, C. L., Martínez Cerdeño, V., Navarro Porras, E., Prakash, A. N., Angelastro, J. M., Willemsen, R., Hagerman, P. J., Pessah, I. N., Berman, R. F., & Noctor, S. C. (2011) · Human Molecular Genetics · doi:10.1093/hmg/ddq432 · Cited by 79
Effects of coffee consumption on serum uric acid: systematic review and meta-analysis
Park, K. Y., Kim, H. J., Ahn, H. S., Kim, S. H., Park, E. J., Yim, S.-Y., & Jun, J. (2016) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2016.01.003 · Cited by 79
The culture of designing hepato-biliary randomised trials
Gluud, C. N. (2006) · Journal of Hepatology · doi:10.1016/j.jhep.2005.12.006 · Cited by 79
“logistic regression analyses seem to dramatically increase rather than decrease the risks of over- and underestimation of intervention effects [69].”
BackgroundSequential Pathways of Testing After First-Trimester Screening for Trisomy 21
Platt, L. D., Greene, N., Johnson, A., Zachary, J. M., Thom, E. A., Krantz, D., Simpson, J. L., Silver, R. K., Snijders, R. J. M., Goetzl, L. M., et al. (2004) · Obstetrics and Gynecology · doi:10.1097/01.aog.0000139832.79658.b9 · Cited by 79
Bates, G., Begley, E., Tod, D., Jones, L., Leavey, C., & McVeigh, J. (2017) · Journal of Health Psychology · doi:10.1177/1359105317737607 · Cited by 89
“This tool is appropriate to use in systematic reviews of effectiveness interventions (Deeks et al., 2003) evaluated using a range of methodologies (Jackson and Waters, 2005).”
MethodLi, L., Wen, J., Wang, L., Li, Y., & Li, Y. (2010) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2010.02802.x · Cited by 86
Louw, M., & Deary, C. (2014) · Physical Therapy in Sport · doi:10.1016/j.ptsp.2013.07.002 · Cited by 58
Cost-effectiveness analysis of prenatal population-based fragile X carrier screening
Musci, T. J., & Caughey, A. B. (2005) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2005.02.052 · Cited by 78
Smith, L. E., Seltzer, M. M., & Greenberg, J. S. (2012) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-011-1422-7 · Cited by 77
Pharmacopuncture in Korea: A Systematic Review and Meta‐Analysis of Randomized Controlled Trials
Park, J., Lee, H., Shin, B., Lee, M. S., Kim, B., & Kim, J. (2016) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2016/4683121 · Cited by 83
Heron, S. R., Woby, S. R., & Thompson, D. P. (2016) · Physiotherapy · doi:10.1016/j.physio.2016.09.001 · Cited by 109
Coughtrey, A. E., & Pistrang, N. (2018) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x16686547 · Cited by 78
“It has evidence of content and construct validity 20,21 and has been judged suitable for systematic reviews of intervention effectiveness 22 .”
de Valk, E. J., Moen, M. H., Winters, M., Bakker, E. W. P., Tamminga, R., & van der Hoeven, H. (2013) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2013.07.273 · Cited by 83
“,(17) 2010 Unilateral ACL reconstruction; participants undergoing bilateral ACL reconstruction were excluded Multicenter Orthopaedic Outcomes Network: 393 patients Follow-up: 2 yr Age, 23 yr (17-35) yr; M/ F, 222/171; TTS, not specified Graft type: BPTB, ST/G, iliotibial band, quadriceps tendon, Achilles tendon Preinj…”
BackgroundKagan, K. O., Wright, D. E., Etchegaray, A., Zhou, Y., & Nicolaides, K. H. (2009) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.6370 · Cited by 77
MacLeod, S., Elliott, L., & Brown, R. (2010) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2010.09.005 · Cited by 96
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.