Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
(1963) · Medical Entomology and Zoology · Cited by 15,085
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
Items 201–300 of 1,024
Yank, V., Tuohy, C. V., Logan, A. C., Bravata, D. M., Staudenmayer, K., Eisenhut, R., Sundaram, V., McMahon, D., Olkin, I., McDonald, K. M., et al. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-154-8-201104190-00004 · Cited by 240
Breslin, G., Shannon, S., Haughey, T., Donnelly, P., & Leavey, G. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0568-6 · Cited by 239
“For non-randomised studies, we used the Quality Assessment Tool for Quantitative Studies (QATSQ) [55] which is recommended for use in systematic reviews [56].”
MethodMindfulness-based interventions during pregnancy: a systematic review and meta-analysis
Dhillon, A., Sparkes, E., & Duarte, R. V. (2017) · Mindfulness · doi:10.1007/s12671-017-0726-x · Cited by 239
McDaid, C., Griffin, S., Weatherly, H., Durée, K., van der Burgt, M., van Hout, S., Akers, J., Davies, R., Sculpher, M., & Westwood, M. (2009) · Health Technology Assessment · doi:10.3310/hta13040 · Cited by 241
Meta‐analysis examining the epidemiology of clozapine‐associated neutropenia
Myles, N., Myles, H., Xia, S., Large, M., Kisely, S., Galletly, C., Bird, R., & Siskind, D. (2018) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12898 · Cited by 232
“As such, we developed a four-point rating scale derived from relevant outcomes assessed in the Newcastle-Ottawa rating scale (7).”
MethodLonjon, G., Boutron, I., Trinquart, L., Ahmad, N., Aim, F., Nizard, R., & Ravaud, P. (2014) · Annals of Surgery · doi:10.1097/sla.0000000000000256 · Cited by 235
Henssler, J., Stock, F., van Bohemen, J., Walter, H., Heinz, A., & Brandt, L. (2021) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-020-01196-x · Cited by 230
Bewick, B. M., Trusler, K., Barkham, M., Hill, A. J., Cahill, J., & Mulhern, B. (2008) · Preventive Medicine · doi:10.1016/j.ypmed.2008.01.005 · Cited by 233
“This scale is regarded as being reliable and easy to use and particularly suitable for the evaluation of non-randomized intervention studies in systematic reviews (Deeks et al., 2003).”
MethodGartlehner, G., Gaynes, B. N., Hansen, R. A., Thieda, P., DeVeaugh-Geiss, A., Krebs, E. E., Moore, C. G., Morgan, L., & Lohr, K. N. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-149-10-200811180-00008 · Cited by 237
Manchikanti, L. (2010) · Pain Physician · doi:10.36076/ppj.2010/13/e215 · Cited by 245
Social Media Use in Chronic Disease: A Systematic Review and Novel Taxonomy
Patel, R., Chang, T., Greysen, S. R., & Chopra, V. (2015) · The American Journal of Medicine · doi:10.1016/j.amjmed.2015.06.015 · Cited by 230
Diabetes and the risk of tuberculosis: a neglected threat to public health?
Stevenson, C. R., Critchley, J. A., Forouhi, N. G., Roglic, G., Williams, B. G., Dye, C., & Unwin, N. C. (2007) · Chronic Illness · doi:10.1177/1742395307081502 · Cited by 226
Cigarette smoking and thyroid eye disease: a systematic review
Thornton, J., Kelly, S. P., Harrison, R. A., & Edwards, R. (2007) · Eye · doi:10.1038/sj.eye.6702603 · Cited by 228
Laparoscopy versus laparotomy for the management of early stage cervical cancer
Wang, Y.-Z., Deng, L., Xu, H.-C., Zhang, Y., & Liang, Z.-Q. (2015) · BMC Cancer · doi:10.1186/s12885-015-1818-4 · Cited by 227
Atun, R., de Jongh, T., Secci, F., Ohiri, K., & Adeyi, O. (2010) · Health Policy and Planning · doi:10.1093/heapol/czp053 · Cited by 228
Interventions for paracetamol (acetaminophen) overdose
Brok, J., Buckley, N., & Gluud, C. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003328.pub2 · Cited by 228
Laparoscopic Versus Open Live Donor Nephrectomy in Renal Transplantation: A Meta-Analysis
Nanidis, T. G., Antcliffe, D., Kokkinos, C., Borysiewicz, C. A., Darzi, A. W., Tekkis, P. P., & Papalois, V. E. (2008) · Annals of Surgery · doi:10.1097/sla.0b013e318153fd13 · Cited by 225
A systematic review of enhanced cognitive behavioral therapy (CBT‐E) for eating disorders
Atwood, M. E., & Friedman, A. (2020) · International Journal of Eating Disorders · doi:10.1002/eat.23206 · Cited by 217
Bonell, C. P., Hargreaves, J., Cousens, S., Ross, D., Hayes, R., Petticrew, M., & Kirkwood, B. R. (2011) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2008.082602 · Cited by 223
Self-citationChong, E. W.-T., Wong, T. Y., Kreis, A. J., Simpson, J. A., & Guymer, R. H. (2007) · BMJ · doi:10.1136/bmj.39350.500428.47 · Cited by 225
The effect of telemedicine in critically ill patients: systematic review and meta-analysis
Wilcox, M. E., & Adhikari, N. K. (2012) · Critical Care · doi:10.1186/cc11429 · Cited by 227
Meyer, S., Karttunen, A. H., Thijs, V., Feys, H., & Verheyden, G. (2014) · Physical Therapy · doi:10.2522/ptj.20130271 · Cited by 219
Frost, D. W., Cook, D. J., Heyland, D. K., & Fowler, R. A. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e31820eacf2 · Cited by 223
“We used the component approach to assess study quality according to reporting guidelines suggested by the STROBE group (9) and assessed features specific to certain to study designs, such as surveys (10), cohort studies (11, 12), and controlled clinical trials (13) (Appendix Table 1).”
MethodRecommendations for assessing the risk of bias in systematic reviews of health-care interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.004 · Cited by 221
Capuzzo, M., Volta, C. A., Tassinati, T., Moreno, R. P., Valentin, A., Guidet, B., Iapichino, G., Martin, C., Perneger, T., Combescure, C., et al. (2014) · Critical Care · doi:10.1186/s13054-014-0551-8 · Cited by 217
Conflict of Interest in Clinical Practice Guideline Development: A Systematic Review
Norris, S. L., Holmer, H. K., Ogden, L. A., & Burda, B. U. (2011) · PLoS ONE · doi:10.1371/journal.pone.0025153 · Cited by 218
Fitzpatrick-Lewis, D., Ganann, R., Krishnaratne, S., Ciliska, D., Kouyoumdjian, F., & Hwang, S. W. (2011) · BMC Public Health · doi:10.1186/1471-2458-11-638 · Cited by 216
“The tool is based on previously established guidelines [9,10], has been examined by experts in the field, and has received excellent ratings [11].”
MethodEvidence at a glance: error matrix approach for overviewing available evidence
Keus, F., Wetterslev, J., Gluud, C., & van Laarhoven, C. J. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-90 · Cited by 219
Bambra, C., Egan, M., Thomas, S., Petticrew, M., & Whitehead, M. (2007) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2006.054999 · Cited by 220
Self-citationScreening for Hepatocellular Carcinoma in Chronic Liver Disease: A Systematic Review
Kansagara, D., Papak, J., Pasha, A. S., O’Neil, M., Freeman, M., Relevo, R., Quiñones, A., Motu’apuaka, M., & Jou, J. H. (2014) · Annals of Internal Medicine · doi:10.7326/m14-0558 · Cited by 217
Trastuzumab-containing regimens for metastatic breast cancer
Balduzzi, S., Mantarro, S., Guarneri, V., Tagliabue, L., Pistotti, V., Moja, L., & D'Amico, R. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006242.pub2 · Cited by 218
Self-citationLiang, C.-S., Li, D.-J., Yang, F.-C., Tseng, P.-T., Carvalho, A. F., Stubbs, B., Thompson, T., Mueller, C., Shin, J. I., Radua, J., et al. (2021) · The Lancet Healthy Longevity · doi:10.1016/s2666-7568(21)00140-9 · Cited by 213
Meta‐analysis of heterogeneously reported trials assessing change from baseline
Abrams, K. R., Gillies, C. L., & Lambert, P. C. (2005) · Statistics in Medicine · doi:10.1002/sim.2423 · Cited by 215
“allowing for bias in observational studies, although empirical evidence regarding both the magnitude and uncertainty of such bias adjustment factors is currently lacking [48].”
BackgroundJohnson, B. T., & Hennessy, E. A. (2019) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.05.035 · Cited by 201
“Even as early as 2003, there were nearly 200 tools available to evaluate non-randomized designs (Deeks et al., 2003), let alone quality inventories for RCTs.”
BackgroundDeprived neighborhoods and adverse perinatal outcome: a systematic review and meta‐analysis
Vos, A. A., Posthumus, A. G., Bonsel, G. J., Steegers, E. A. P., & Denktaş, S. (2014) · Acta Obstetricia et Gynecologica Scandinavica · doi:10.1111/aogs.12430 · Cited by 187
“The scale was scientifically evaluated and is regarded as suitable to use for quality assessment of observational studies (30).”
MethodChildhood obesity and adult cardiovascular disease risk: a systematic review
Dunford, L. J., Langley-Evans, S. C., & McMullen, S. (2010) · International Journal of Obesity · doi:10.1038/ijo.2009.61 · Cited by 215
Ergina, P. L., Barkun, J. S., McCulloch, P., Cook, J. A., Altman, D. G., & On behalf of the IDEAL group (2013) · BMJ · doi:10.1136/bmj.f3011 · Cited by 210
Self-citationPimpin, L., Wu, J. H. Y., Haskelberg, H., Del Gobbo, L., & Mozaffarian, D. (2016) · PLOS ONE · doi:10.1371/journal.pone.0158118 · Cited by 209
“Quality assessment We adapted the Newcastle-Ottawa quality scale(NOS) [16] to assess study quality, based on five criteria evaluating the reporting and appropriateness/representativeness of participant inclusion and exclusion criteria (combining the first two items of the NOS Selection scale), participant attrition (N…”
MethodKennedy, C. E., Fonner, V. A., Armstrong, K. A., Denison, J. A., Yeh, P. T., O’Reilly, K. R., & Sweat, M. D. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0925-0 · Cited by 212
Simillis, C., Purkayastha, S., Yamamoto, T., Strong, S. A., Darzi, A. W., & Tekkis, P. P. (2007) · Diseases of the Colon & Rectum · doi:10.1007/s10350-007-9011-8 · Cited by 210
Ahmed Ali, U., Keus, F., Heikens, J. T., Bemelman, W. A., Berdah, S. V., Gooszen, H. G., & van Laarhoven, C. J. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006267.pub2 · Cited by 212
Thompson, C. S., & Hakim, A. M. (2009) · Stroke · doi:10.1161/strokeaha.108.542266 · Cited by 208
Dashorst, P., Mooren, T. M., Kleber, R. J., de Jong, P. J., & Huntjens, R. J. C. (2019) · European Journal of Psychotraumatology · doi:10.1080/20008198.2019.1654065 · Cited by 207
A systematic review of enhanced recovery care after colorectal surgery in elderly patients
Bagnall, N. M., Malietzis, G., Kennedy, R. H., Athanasiou, T., Faiz, O., & Darzi, A. (2014) · Colorectal Disease · doi:10.1111/codi.12718 · Cited by 204
Zhong, Q., Peng, H.-L., Zhao, X., Zhang, L., & Hwang, W.-T. (2015) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-14-1816 · Cited by 203
“Hwang), using the Newcastle–Ottawa Scale (28) and Altman framework (29).”
MethodEkeloef, S., Alamili, M., Devereaux, P. J., & Gögenur, I. (2016) · British Journal of Anaesthesia · doi:10.1093/bja/aew321 · Cited by 205
Fernandez-Carvajal, I., Walichiewicz, P., Xiaosen, X., Pan, R., Hagerman, P. J., & Tassone, F. (2009) · The Journal of Molecular Diagnostics · doi:10.2353/jmoldx.2009.080173 · Cited by 204
Systematic review and meta-analyses: fever in pregnancy and health impacts in the offspring
Dreier, J. W., Andersen, A.-M. N., & Berg-Beckhoff, G. (2014) · PEDIATRICS · doi:10.1542/peds.2013-3205 · Cited by 204
Marquet, K., Liesenborgs, A., Bergs, J., Vleugels, A., & Claes, N. (2015) · Critical Care · doi:10.1186/s13054-015-0795-y · Cited by 203
“This scale assesses study reporting, external validity, internal validity, and power of non-randomized studies and has been ranked in the top six quality assessment scales suitable for use in systematic reviews [25,26].”
BackgroundFilippini, G., Falcone, C., Boiardi, A., Broggi, G., Bruzzone, M. G., Caldiroli, D., Farina, R., Farinotti, M., Fariselli, L., Finocchiaro, G., et al. (2008) · Neuro-Oncology · doi:10.1215/15228517-2007-038 · Cited by 207
Vella-Brodrick, D. A., & Gilowska, K. (2022) · Educational Psychology Review · doi:10.1007/s10648-022-09658-5 · Cited by 175
“The EPHPP has been shown to have robust psychometric properties and is suitable for systematic reviews of effectiveness (Deeks et al., 2003; Jackson & Waters, 2005; Thomas et al., 2004).”
MethodInterventions for prevention of bullying in the workplace
Gillen, P. A., Sinclair, M., Kernohan, W. G., Begley, C. M., & Luyben, A. G. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009778.pub2 · Cited by 195
Early orthodontic treatment for Class III malocclusion: A systematic review and meta-analysis
Woon, S. C., & Thiruvenkatachari, B. (2017) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2016.07.017 · Cited by 191
“We pilot tested a subset of our studies with the Downs and Black and the Newcastle-Ottawa scales.(10) Although both have been widely used for quality assessment, we found the former to be a more comprehensive asessment with a 27-point scale.”
MethodPeel, N. M., Alapatt, L. J., Jones, L. V., & Hubbard, R. E. (2019) · The Journals of Gerontology: Series A · doi:10.1093/gerona/gly140 · Cited by 201
“) using the Newcastle Ottawa scale [8] appropriate to the study design.”
MethodMindfulness and self-esteem: A systematic review
Randal, C., Pratt, D., & Bucci, S. (2015) · Mindfulness · doi:10.1007/s12671-015-0407-6 · Cited by 202
“Deeks et al. (2003) critically appraised a number of quality assessment tools and identified six which they deemed suitable for use in systematic reviews of both randomised and non-randomised studies.”
Lim, E., Harris, G., Patel, A., Adachi, I., Edmonds, L., & Song, F. (2009) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e3181b9ecca · Cited by 200
Self-citationBrinjikji, W., Starke, R. M., Murad, M. H., Fiorella, D., Pereira, V. M., Goyal, M., & Kallmes, D. F. (2018) · Journal of NeuroInterventional Surgery · doi:10.1136/neurintsurg-2017-013179 · Cited by 200
Shah, A., Talley, N. J., Jones, M., Kendall, B. J., Koloski, N., Walker, M. M., Morrison, M., & Holtmann, G. J. (2020) · American Journal of Gastroenterology · doi:10.14309/ajg.0000000000000504 · Cited by 200
Berg-Beckhoff, G., Nielsen, G., & Ladekjær Larsen, E. (2017) · International Journal of Occupational and Environmental Health · doi:10.1080/10773525.2018.1436015 · Cited by 197
“The risk of bias in the studies included was assessed by the Newcastle-Ottawa scale [27,28].”
Health-Related Quality of Life After Anterior Cruciate Ligament Reconstruction: A Systematic Review
Filbay, S. R., Ackerman, I. N., Russell, T. G., Macri, E. M., & Crossley, K. M. (2014) · The American Journal of Sports Medicine · doi:10.1177/0363546513512774 · Cited by 193
“7 No Pro KOOS Sajovic et al(49) 16 64 64 11 50 50 36 HT, 38 PT 42 25 HT, 23 PT (1-84) Yes Pro SF-36 version 2 Swirtun and Renström(51) 13 46 22 5.”
Endoscopic submucosal dissection vs endoscopic mucosal resection for superficial esophageal cancer
Guo, H.-M. (2014) · World Journal of Gastroenterology · doi:10.3748/wjg.v20.i18.5540 · Cited by 198
Chou, R., Aronson, N., Atkins, D., Ismaila, A. S., Santaguida, P., Smith, D. H., Whitlock, E., Wilt, T. J., & Moher, D. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.06.007 · Cited by 184
Screening for breast cancer with mammography
Gøtzsche, P. C., & Nielsen, M. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001877.pub3 · Cited by 194
van Dijk, S. T., van Dijk, A. H., Dijkgraaf, M. G., & Boermeester, M. A. (2018) · British Journal of Surgery · doi:10.1002/bjs.10873 · Cited by 186
Association Between BRCA Status and Triple-Negative Breast Cancer: A Meta-Analysis
Chen, H., Wu, J., Zhang, Z., Tang, Y., Li, X., Liu, S., Cao, S., & Li, X. (2018) · Frontiers in Pharmacology · doi:10.3389/fphar.2018.00909 · Cited by 188
“Quality Assessment The quality of each study was independently evaluated by at least two examiners who read each study and scored it according to the NOS criteria (Deeks et al., 2003).”
MethodVester-Andersen, M., Lundstrøm, L. H., Møller, M. H., Waldau, T., Rosenberg, J., & Møller, A. M. (2014) · British Journal of Anaesthesia · doi:10.1093/bja/aet487 · Cited by 189
Hempel, S., Shekelle, P. G., Liu, J. L., Sherwood Danz, M., Foy, R., Lim, Y.-W., Motala, A., & Rubenstein, L. V. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2014-003151 · Cited by 187
Wang, L., Wang, R., Liu, L., Qiao, D., Baldwin, D. S., & Hou, R. (2019) · Brain, Behavior, and Immunity · doi:10.1016/j.bbi.2019.02.021 · Cited by 185
Mazza, M. G., Lucchi, S., Rossetti, A., & Clerici, M. (2020) · The World Journal of Biological Psychiatry · doi:10.1080/15622975.2019.1583371 · Cited by 191
“A higher score indicates higher methodological quality (Deeks et al. 2003).”
BackgroundCrowe, M., Sheppard, L., & Campbell, A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.08.006 · Cited by 192
“Furthermore, evidence must be stated in the paper and cannot be assumed, which is in keeping with other CATs, reporting guidelines, and procedures for conducting systematic reviews [3,11,12].”
BackgroundMethodPrevalence of CGG expansions of the FMR1 gene in a US population‐based sample
Seltzer, M. M., Baker, M. W., Hong, J., Maenner, M., Greenberg, J., & Mandel, D. (2012) · American Journal of Medical Genetics Part B: Neuropsychiatric Genetics · doi:10.1002/ajmg.b.32065 · Cited by 189
The influence of grandmothers on breastfeeding rates: a systematic review
Negin, J., Coffman, J., Vizintin, P., & Raynes-Greenow, C. (2016) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-016-0880-5 · Cited by 188
The impact of gender dyads on doctor–patient communication: A systematic review
Sandhu, H., Adams, A., Singleton, L., Clark-Carter, D., & Kidd, J. (2009) · Patient Education and Counseling · doi:10.1016/j.pec.2009.07.010 · Cited by 191
Kesselheim, A. S., Stedman, M. R., Bubrick, E. J., Gagne, J. J., Misono, A. S., Lee, J. L., Brookhart, M. A., Avorn, J., & Shrank, W. H. (2010) · Drugs · doi:10.2165/10898530-000000000-00000 · Cited by 190
“The methodological quality of nonrandomized trials was assessed using the 9-star Newcastle-Ottawa scale.([29]) This was done independently by two authors (ASK and WHS), with differences resolved by consensus.”
MethodHepatitis B immunisation for newborn infants of hepatitis B surface antigen-positive mothers
Lee, C., Gong, Y., Brok, J., Boxall, E. H., & Gluud, C. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004790.pub2 · Cited by 184
O’Driscoll, R., Turicchi, J., Beaulieu, K., Scott, S., Matu, J., Deighton, K., Finlayson, G., & Stubbs, J. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2018-099643 · Cited by 188
Discordance between reported intention-to-treat and per protocol analyses
Porta, N., Bonet, C., & Cobo, E. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.09.013 · Cited by 189
Systematic review of the psychological consequences of false-positive screening mammograms
Bond, M., Pavey, T., Welch, K., Cooper, C., Garside, R., Dean, S., & Hyde, C. (2013) · Health Technol Assess · doi:10.3310/hta17130 · Cited by 184
Empirical evidence of the importance of comparative studies of diagnostic test accuracy
Takwoingi, Y., Leeflang, M. M. G., & Deeks, J. J. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-7-201304020-00006 · Cited by 187
“4) Monitoring 1 (4) 0 2 (1) 3 (1) Prognostic/prediction 1 (4) 1 (2) 4 (2) 6 (2) Screening 3 (11) 2 (5) 6 (3) 11 (4) Staging 0 2 (5) 9 (5) 11 (4)”
InfluentialSelf-citationMowatt, G., Zhu, S., Kilonzo, M., Boachie, C., Fraser, C., Griffiths, T., N'Dow, J., Nabi, G., Cook, J., & Vale, L. (2010) · Health Technology Assessment · doi:10.3310/hta14040 · Cited by 313
Prevalence and Measurement of Anxiety in Samples of Patients With Heart Failure: Meta-analysis
Easton, K., Coventry, P., Lovell, K., Carter, L.-A., & Deaton, C. (2016) · Journal of Cardiovascular Nursing · doi:10.1097/jcn.0000000000000265 · Cited by 186
“There is no standard quality assessment instrument for use with observational studies or for use in reviews that seek to document the presence or prevalence of a condition as opposed to measure the effects of an intervention.(29) It is therefore advised that key components be chosen relating to an individual review’s…”
BackgroundBryant, R. J., Sjoberg, D. D., Vickers, A. J., Robinson, M. C., Kumar, R., Marsden, L., Davis, M., Scardino, P. T., Donovan, J., Neal, D. E., et al. (2015) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djv095 · Cited by 184
Shaw, L., Rodgers, Price, C., van Wijck, F., Shackley, P., Steen, N., Barnes, M., Ford, G., & Graham, L. (2010) · Health Technology Assessment · doi:10.3310/hta14260 · Cited by 187
Do Couple-Based Interventions Make a Difference for Couples Affected by Cancer?: A Systematic Review
Regan, T. W., Lambert, S. D., Girgis, A., Kelly, B., Kayser, K., & Turner, J. (2012) · BMC Cancer · doi:10.1186/1471-2407-12-279 · Cited by 185
“[44,45].”
Guest, E., Costa, B., Williamson, H., Meyrick, J., Halliwell, E., & Harcourt, D. (2019) · Body Image · doi:10.1016/j.bodyim.2019.04.002 · Cited by 184
“Additionally, it has established construct and content validity (Jackson & Waters, 2005), and is reported to be suitable for reviews of effectiveness (Deeks et al., 2003).”
BackgroundThe oral health impact of electronic cigarette use: a systematic review
Yang, I., Sandeep, S., & Rodriguez, J. (2020) · Critical Reviews in Toxicology · doi:10.1080/10408444.2020.1713726 · Cited by 176
“It is deemed appropriate for systematic reviews (Deeks et al. 2003) with proven content and construct validity (Thomas et al.”
BackgroundHiggins, J. P., Ramsay, C., Reeves, B. C., Deeks, J. J., Shea, B., Valentine, J. C., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1056 · Cited by 183
“One comprehensive review examined eight studies that had made such comparisons across multiple interventions (Deeks et al., 2003).”
BackgroundMethodInfluentialSelf-citationA general critical appraisal tool: An evaluation of construct validity
Crowe, M., & Sheppard, L. (2011) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.06.004 · Cited by 182
“Tools that use inappropriate scoring systems, such as simplistic summary scores, in such a way that defects in studies may be hidden (Armijo Olivo et al., 2008; Deeks et al., 2003; Jüni et al., 1999); and, 4.”
BackgroundInfluentialRuhayel, Y., Tepeler, A., Dabestani, S., MacLennan, S., Petřík, A., Sarica, K., Seitz, C., Skolarikos, A., Straub, M., Türk, C., et al. (2017) · European Urology · doi:10.1016/j.eururo.2017.01.046 · Cited by 184
A Systematic Review of Pharmacologic Treatments of Pain After Spinal Cord Injury
Teasell, R. W., Mehta, S., Aubut, J.-A. L., Foulon, B., Wolfe, D. L., Hsieh, J. T. C., Townson, A. F., & Short, C. (2010) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2010.01.022 · Cited by 184
A systematic review of the rehabilitation of moderate to severe acquired brain injuries
Teasell, R., Bayona, N., Marshall, S., Cullen, N., Bayley, M., Chundamala, J., Villamere, J., Mackie, D., Rees, L., Hartridge, C., et al. (2007) · Brain Injury · doi:10.1080/02699050701201524 · Cited by 183
Vissers, M. M., Bussmann, J. B., Verhaar, J. A. N., Arends, L. R., Furlan, A. D., & Reijman, M. (2011) · Physical Therapy · doi:10.2522/ptj.20100201 · Cited by 179
Intrathecal Therapy for Cancer and Non-Cancer Pain
Hayek, S. M. (2011) · Pain Physician · doi:10.36076/ppj.2011/14/219 · Cited by 182
Economic Analysis of Physical Activity Interventions
Wu, S., Cohen, D., Shi, Y., Pearson, M., & Sturm, R. (2011) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2010.10.029 · Cited by 178
Paré, G., Tate, M., Johnstone, D., & Kitsiou, S. (2016) · European Journal of Information Systems · doi:10.1057/s41303-016-0020-3 · Cited by 179
“There exist a plethora of instruments within the social and health care sciences which authors can review and choose from (e.g. Higgins & Altman, 2008; Deeks et al, 2003; Long & Godfrey, 2004).”
BackgroundMethodMoltrecht, B., Deighton, J., Patalay, P., & Edbrooke-Childs, J. (2021) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-020-01498-4 · Cited by 179
“The EPHPP has been reported suitable for systematic reviews and evidence has shown good content and construct validity [23, 24].”
Surgery for rotator cuff disease
Coghlan, J. A., Buchbinder, R., Green, S., Johnston, R. V., & Bell, S. N. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005619.pub2 · Cited by 176
Mohammed, M. A., Deeks, J. J., Girling, A., Rudge, G., Carmalt, M., Stevens, A. J., & Lilford, R. J. (2009) · BMJ · doi:10.1136/bmj.b780 · Cited by 178
Self-citationBlood glucose self-monitoring in type 2 diabetes: a randomised controlled trial
Farmer, A., Wade, A., French, D., Simon, J., Yudkin, P., Gray, A., Craven, A., Goyder, L., Holman, R., Mant, D., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13150 · Cited by 180
Ewen, A. M., Stewart, S., St Clair Gibson, A., Kashyap, S. N., & Caplan, N. (2012) · Gait & Posture · doi:10.1016/j.gaitpost.2011.12.024 · Cited by 177
“[15] examined 194 previously published instruments which had been used to assess the quality of non-randomised studies in terms of selection bias and control of confounding factors.”
Method
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.