Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
(1963) · Medical Entomology and Zoology · Cited by 15,085
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
Items 301–400 of 1,024
Rotator cuff disorders: Recognition and management among patients with shoulder pain
Gomoll, A. H., Katz, J. N., Warner, J. J. P., & Millett, P. J. (2004) · Arthritis & Rheumatism · doi:10.1002/art.20668 · Cited by 179
Cuello-Garcia, C. A., Santesso, N., Morgan, R. L., Verbeek, J., Thayer, K., Ansari, M. T., Meerpohl, J., Schwingshackl, L., Katikireddi, S. V., Brozek, J. L., et al. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.11.026 · Cited by 172
Laparoscopic Versus Open Live Donor Nephrectomy in Renal Transplantation: A Meta-Analysis
Kok, N. F., IJzermans, J. N., & Alwayn, I. P. (2008) · Annals of Surgery · doi:10.1097/sla.0b013e3181884330 · Cited by 179
End-of-life care pathways for improving outcomes in caring for the dying
Chan, R. J., Webster, J., & Bowers, A. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008006.pub4 · Cited by 174
Amor, D. J., Xu, J. X., Halliday, J. L., Francis, I., Healy, D. L., Breheny, S., Baker, H. W. G., & Jaques, A. M. (2009) · Human Reproduction · doi:10.1093/humrep/dep046 · Cited by 178
Li, J., Woodward, A., Hou, X.-Y., Zhu, T., Zhang, J., Brown, H., Yang, J., Qin, R., Gao, J., Gu, S., et al. (2017) · Science of The Total Environment · doi:10.1016/j.scitotenv.2016.10.070 · Cited by 178
Wang, D., Cummins, C., Bayliss, S., Sandercock, J., & Burls, A. (2008) · Health Technology Assessment · doi:10.3310/hta12360 · Cited by 175
Hootman, J. M., Driban, J. B., Sitler, M. R., Harris, K. P., & Cattano, N. M. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.41 · Cited by 172
“…case-control, cohort) are more diverse, there are fewer quality rating instruments available, authors often create their own study-specific quality rating tool, and data on validity and reliability are sparse (Deeks et al., 2003; Sanderson et al., 2007; Mallen et al., 2006; Simunovic et al., 2009).”
BackgroundPositive affect and sleep: A systematic review
Ong, A. D., Kim, S., Young, S., & Steptoe, A. (2017) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2016.07.006 · Cited by 174
Educational Interventions for Cervical Cancer Screening Behavior of Women: A Systematic Review
Naz, M. S. G., Kariman, N., Ebadi, A., Ozgoli, G., Ghasemi, V., & Fakari, F. R. (2018) · PubMed · doi:10.22034/apjcp.2018.19.4.875 · Cited by 167
“Based on their total score, quality of studies is rated as weak (1.00–1.50), moderate (1.51–2.50) or strong (2.51–3.00) (Armijo-Olivo et al., 2012; Deeks et al., 2003; Thomas et al., 2004).”
BackgroundInfluentialClinical Practice Guidelines for Gastric Cancer in Korea: An Evidence-Based Approach
Lee, J. H., Kim, J. G., Jung, H.-K., Kim, J. H., Jeong, W. K., Jeon, T. J., Kim, J. M., Kim, Y. I., Ryu, K. W., Kong, S.-H., et al. (2014) · Journal of Gastric Cancer · doi:10.5230/jgc.2014.14.2.87 · Cited by 176
Torchalla, I., Nosen, L., Rostam, H., & Allen, P. (2012) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2011.09.001 · Cited by 174
“The QAT has been recommended for use in systematic reviews that include nonrandomized intervention studies (Deeks et al., 2003).”
MethodWilson, J., Yao, G., Raftery, J., Bohlius, J., Brunskill, S., Sandercock, J., Bayliss, S., Moss, P., Stanworth, S., & Hyde, C. (2007) · Health Technology Assessment · doi:10.3310/hta11130 · Cited by 173
Smith, T. O., Back, T., Toms, A. P., & Hing, C. B. (2011) · Clinical Radiology · doi:10.1016/j.crad.2011.05.007 · Cited by 174
Ainsworth, R., & Lewis, J. S. (2007) · British Journal of Sports Medicine · doi:10.1136/bjsm.2006.032524 · Cited by 174
Morrell, C., Warner, R., Slade, P., Dixon, S., Walters, S., Paley, G., & Brugha, T. (2009) · Health Technology Assessment · doi:10.3310/hta13300 · Cited by 176
Helicopter emergency medical services for adults with major trauma
Galvagno Jr, S. M., Sikorski, R., Hirshon, J. M., Floccare, D., Stephens, C., Beecher, D., & Thomas, S. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009228.pub3 · Cited by 165
A Systematic Review of Adherence With Medications for Diabetes
Wens, J., Vermeire, E., Van Royen, P., & Hearnshaw, H. (2004) · Diabetes Care · doi:10.2337/diacare.27.9.2284 · Cited by 171
Nurse‐led vs. conventional physician‐led follow‐up for patients with cancer: systematic review
Lewis, R., Neal, R. D., Williams, N. H., France, B., Wilkinson, C., Hendry, M., Russell, D., Russell, I., Hughes, D. A., Stuart, N. S. A., et al. (2009) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2008.04927.x · Cited by 171
“The quality of effectiveness studies was assessed using the checklist developed by Downs and Black (1998), modified according to the suggestions by Deeks et al. (2003) and adapted for use with cancer care follow-up studies.”
Systematic Review of Psychological Interventions for Pediatric Feeding Problems
Lukens, C. T., & Silverman, A. H. (2014) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsu040 · Cited by 169
“, clinician preferences, patient characteristics, clinical history) that are related to treatment outcome, introducing a source of systematic bias that ultimately leads to an overestimation of treatment efficacy (Deeks et al., 2003).”
Meta-analysis of laparoscopic vs open liver resection for hepatocellular carcinoma
Xiong, J.-J. (2012) · World Journal of Gastroenterology · doi:10.3748/wjg.v18.i45.6657 · Cited by 161
Litton, E., Carnegie, V., Elliott, R., & Webb, S. A. R. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000001557 · Cited by 164
Barriers to the conduct of randomised clinical trials within all disease areas
Djurisic, S., Rath, A., Gaber, S., Garattini, S., Bertele, V., Ngwabyt, S.-N., Hivert, V., Neugebauer, E. A. M., Laville, M., Hiesmayr, M., et al. (2017) · Trials · doi:10.1186/s13063-017-2099-9 · Cited by 169
Evidence-based guidelines for use of probiotics in preterm neonates
Deshpande, G. C., Rao, S. C., Keil, A. D., & Patole, S. K. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-92 · Cited by 166
Contamination in trials of educational interventions
Keogh-Brown, M., Bachmann, M., Shepstone, L., Hewitt, C., Howe, A., Ramsay, C., Song, F., Miles, J., Torgerson, D., Miles, S., et al. (2007) · Health Technology Assessment · doi:10.3310/hta11430 · Cited by 168
Self-citationBarker, A., Maratos, E. C., Edmonds, L., & Lim, E. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61163-5 · Cited by 170
Including non‐randomized studies on intervention effects
Reeves, B. C., Deeks, J. J., Higgins, J. P., Shea, B., Tugwell, P., Wells, G. A., & on behalf of the Cochrane Non‐Randomized Studies of Interventions Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch24 · Cited by 153
Self-citationFollow-up of cancer in primary care versus secondary care: systematic review
Lewis, R. A., Neal, R. D., Williams, N. H., France, B., Hendry, M., Russell, D., Hughes, D. A., Russell, I., Stuart, N. S., Weller, D., et al. (2009) · British Journal of General Practice · doi:10.3399/bjgp09x453567 · Cited by 166
Clinical risk factors for Achilles tendinopathy: a systematic review
van der Vlist, A. C., Breda, S. J., Oei, E. H. G., Verhaar, J. A. N., & de Vos, R. (2019) · British Journal of Sports Medicine · doi:10.1136/bjsports-2018-099991 · Cited by 206
Risk for newly diagnosed diabetes after COVID-19: a systematic review and meta-analysis
Zhang, T., Mei, Q., Zhang, Z., Walline, J. H., Liu, Y., Zhu, H., & Zhang, S. (2022) · BMC Medicine · doi:10.1186/s12916-022-02656-y · Cited by 158
Benjamin, D. R., Frawley, H. C., Shields, N., van de Water, A. T. M., & Taylor, N. F. (2019) · Physiotherapy · doi:10.1016/j.physio.2018.07.002 · Cited by 165
A Systematic Review of Subclinical Keratoconus and Forme Fruste Keratoconus
Henriquez, M. A., Hadid, M., & Izquierdo, L. (2020) · Journal of Refractive Surgery · doi:10.3928/1081597x-20200212-03 · Cited by 161
Prospective first-trimester screening for trisomy 21 in 30,564 pregnancies
Avgidou, K., Papageorghiou, A., Bindra, R., Spencer, K., & Nicolaides, K. H. (2005) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2005.03.021 · Cited by 162
Yamada, J., Shorkey, A., Barwick, M., Widger, K., & Stevens, B. J. (2015) · BMJ Open · doi:10.1136/bmjopen-2014-006808 · Cited by 161
“The EPHPP assesses methodological quality in systematic reviews of effectiveness.(17) Reliability and content and construct validity of the tool have been established.”
BackgroundBanzi, R., Moja, L., Pistotti, V., Facchini, A., & Liberati, A. (2011) · Health Research Policy and Systems · doi:10.1186/1478-4505-9-26 · Cited by 160
Arezzo, A., Passera, R., Marchese, N., Galloro, G., Manta, R., & Cirocchi, R. (2016) · United European Gastroenterology Journal · doi:10.1177/2050640615585470 · Cited by 161
Reichenpfader, U., Gartlehner, G., Morgan, L. C., Greenblatt, A., Nussbaumer, B., Hansen, R. A., Van Noord, M., Lux, L., & Gaynes, B. N. (2014) · Drug Safety · doi:10.1007/s40264-013-0129-4 · Cited by 160
“studies, we used criteria outlined by Deeks and colleagues [22].”
The relationship between sleep and postpartum mental disorders: A systematic review
Lawson, A., Murphy, K. E., Sloan, E. P., Uleryk, E. M., & Dalfen, A. (2015) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.01.017 · Cited by 194
Induction of lucid dreams: A systematic review of evidence
Stumbrys, T., Erlacher, D., Schädlich, M., & Schredl, M. (2012) · Consciousness and Cognition · doi:10.1016/j.concog.2012.07.003 · Cited by 202
Obesity in Pediatric ALL Survivors: A Meta-Analysis
Zhang, F. F., Kelly, M. J., Saltzman, E., Must, A., Roberts, S. B., & Parsons, S. K. (2014) · Pediatrics · doi:10.1542/peds.2013-3332 · Cited by 159
Aneuploidy screening in the first trimester
Spencer, K. (2007) · American Journal of Medical Genetics Part C Seminars in Medical Genetics · doi:10.1002/ajmg.c.30119 · Cited by 160
Lewis, R. A., Williams, N. H., Sutton, A. J., Burton, K. A., Din, N., Matar, H. E., Hendry, M., Phillips, C. J., Nafees, S. B., Fitzsimmons, D. F., et al. (2013) · The Spine Journal · doi:10.1016/j.spinee.2013.08.049 · Cited by 173
“The RCT is widely regarded as the design of choice when assessing the effectiveness of health care interventions [167] and we acknowledge the controversy over the inclusion of nonrandomized evidence.”
BackgroundBanks, K., Newman, E., & Saleem, J. (2015) · Journal of Clinical Psychology · doi:10.1002/jclp.22200 · Cited by 156
GRACE principles: recognizing high-quality observational studies of comparative effectiveness.
Dreyer, N. A., Schneeweiß, S., McNeil, B. J., Berger, M. L., Walker, A. M., Ollendorf, D. A., & Gliklich, R. E. (2010) · PubMed · Cited by 177
Manchikanti, L., Hirsch, J. A., & Smith, H. S. (2008) · Pain Physician · doi:10.36076/ppj.2008/11/717 · Cited by 161
Sheikh, A., Mendelson, Z., & Liu, J. K. (2014) · World Neurosurgery · doi:10.1016/j.wneu.2014.06.024 · Cited by 155
Devoto, L., Celentano, V., Cohen, R., Khan, J. S., & Chand, M. (2017) · International Journal of Colorectal Disease · doi:10.1007/s00384-017-2848-y · Cited by 158
Anesthesia-Related Outcomes for Endovascular Stroke Revascularization
Brinjikji, W., Pasternak, J. J., Murad, M. H., Cloft, H. J., Welch, T. L., Kallmes, D. F., & Rabinstein, A. A. (2017) · Stroke · doi:10.1161/strokeaha.117.017786 · Cited by 154
Impact of dental implant length on early failure rates: a meta-analysis of observational studies
Pommer, B., Frantal, S., Willer, J., Posch, M., Watzek, G., & Tepper, G. (2011) · Journal Of Clinical Periodontology · doi:10.1111/j.1600-051x.2011.01750.x · Cited by 153
Fragile X targeted pharmacotherapy: lessons learned and future directions
Erickson, C. A., Davenport, M. H., Schaefer, T. L., Wink, L. K., Pedapati, E. V., Sweeney, J. A., Fitzpatrick, S. E., Brown, W. T., Budimirovic, D., Hagerman, R. J., et al. (2017) · Journal of Neurodevelopmental Disorders · doi:10.1186/s11689-017-9186-9 · Cited by 154
Bambra, C. L., Whitehead, M. M., Sowden, A. J., Akers, J., & Petticrew, M. P. (2008) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.12.023 · Cited by 193
Self-citationHIV and male circumcision—a systematic review with assessment of the quality of studies
Siegfried, N., Muller, M., Deeks, J. J., Volmink, J. A., Egger, M., Low, N. N., Walker, A. S., & Williamson, P. R. (2005) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(05)01309-5 · Cited by 152
Self-citationA checklist to assess the quality of survey studies in psychology
Protogerou, C., & Hagger, M. S. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100031 · Cited by 146
Gartlehner, G., Thieda, P., Hansen, R. A., Gaynes, B. N., DeVeaugh-Geiss, A., Krebs, E. E., & Lohr, K. N. (2008) · Drug Safety · doi:10.2165/00002018-200831100-00004 · Cited by 150
Changes in attachment representations during psychological therapy
Taylor, P., Rietzschel, J., Danquah, A., & Berry, K. (2015) · Psychotherapy Research · doi:10.1080/10503307.2014.886791 · Cited by 150
Updated Method Guidelines for Cochrane Musculoskeletal Group Systematic Reviews and Metaanalyses
Ghogomu, E. A. T., Maxwell, L. J., Buchbinder, R., Rader, T., Pardo Pardo, J., Johnston, R. V., Christensen, R. D. K., Rutjes, A. W. S., Winzenberg, T. M., Singh, J. A., et al. (2014) · The Journal of Rheumatology · doi:10.3899/jrheum.121306 · Cited by 151
Squires, J. E., Hoben, M., Linklater, S., Carleton, H. L., Graham, N., & Estabrooks, C. A. (2015) · Nursing Research and Practice · doi:10.1155/2015/157924 · Cited by 143
“This tool has been judged suitable to be used in systematic reviews of effectiveness (measuring interventions) [89] and been shown to have content and construct validity [90].”
Initiatives to shorten duration of untreated psychosis: systematic review
Lloyd-Evans, B., Crosby, M., Stockton, S., Pilling, S., Hobbs, L., Hinton, M., & Johnson, S. (2011) · British Journal of Psychiatry · doi:10.1192/bjp.bp.109.075622 · Cited by 149
Endovascular repair of ruptured abdominal aortic aneurysms: A systematic review and meta-analysis
Mastracci, T. M., Garrido‐Olivares, L., Cinà, C. S., & Clase, C. M. (2008) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2007.07.052 · Cited by 149
β blockers for elective surgery in elderly patients: population based, retrospective cohort study
Redelmeier, D. A., Scales, D. C., & Kopp, A. (2005) · BMJ · doi:10.1136/bmj.38603.746944.3a · Cited by 160
Models for Potentially Biased Evidence in Meta-Analysis using Empirically Based Priors
Welton, N. J., Ades, A. E., Carlin, J. B., Altman, D. G., & Sterne, J. A. C. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00548.x · Cited by 150
“Although there have been several evaluations of differences between intervention effects in randomized controlled trials and observational studies (Deeks et al., 2003), we are not aware of published estimates of the variance of these differences.”
BackgroundSelf-citationChen, Y., Jowett, S., Barton, P., Malottki, K., Hyde, C., Gibbs, J., Pepke-Zaba, J., Fry-Smith, A., Roberts, J., & Moore, D. (2009) · Health Technology Assessment · doi:10.3310/hta13490 · Cited by 148
Bryant, J., Bonevski, B., Paul, C., McElduff, P., & Attia, J. (2011) · Addiction · doi:10.1111/j.1360-0443.2011.03467.x · Cited by 145
“Assessment of methodological quality Studies included in the review were assessed for methodological quality using the Effective Public Health Practice Project Quality Assessment Tool for quantitative studies [31–33].”
Tannen, R. L., Weiner, M. G., & Xie, D. (2009) · BMJ · doi:10.1136/bmj.b81 · Cited by 145
Marshall, A., Altman, D. G., & Holder, R. L. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-112 · Cited by 146
Self-citationSignaling defects in iPSC-derived fragile X premutation neurons
Liu, J., Kościelska, K. A., Cao, Z., Hulsizer, S., Grace, N., Mitchell, G., Nacey, C., Githinji, J., McGee, J., Garcia-Arocena, D., et al. (2012) · Human Molecular Genetics · doi:10.1093/hmg/dds207 · Cited by 146
Randhawa, M. S., Vishwanath, R., Rai, M. P., Wang, L., Randhawa, A. K., Abela, G. S., & Dhar, G. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.2175 · Cited by 160
In-hospital care pathways for stroke
Kwan, J. S. K., & Sandercock, P. A. (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002924.pub2 · Cited by 171
Cameron, C., Fireman, B., Hutton, B., Clifford, T., Coyle, D., Wells, G., Dormuth, C. R., Platt, R., & Toh, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0133-0 · Cited by 140
Critical appraisal of nonrandomized studies—A review of recommended and commonly used tools
Quigley, J. M., Thompson, J. C., Halfpenny, N. J., & Scott, D. A. (2019) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12889 · Cited by 144
The rise of big clinical databases
Cook, J. A., & Collins, G. S. (2015) · British journal of surgery · doi:10.1002/bjs.9723 · Cited by 153
“More caution is needed when a causal interpretation is made (as is true for any observational study 26 , 58 ).”
BackgroundUttley, L., Scope, A., Stevenson, M., Rawdin, A., Taylor Buck, E., Sutton, A., Stevens, J., Kaltenthaler, E., Dent-Brown, K., & Wood, C. (2015) · Health Technology Assessment · doi:10.3310/hta19180 · Cited by 134
“The consequence of including data from non-randomised studies into the review is that the resulting data are biased and therefore not robust or sufficient to inform and contribute to the evidence base.(41,42) The inclusion and exclusion criteria for the quantitative review are shown in Figure 2.”
BackgroundNoorani, H. Z., Husereau, D., Boudreau, R., & Skidmore, B. (2007) · International Journal of Technology Assessment in Health Care · doi:10.1017/s026646230707050x · Cited by 142
The effect of neutering on the risk of mammary tumours in dogs – a systematic review
Beauvais, W. A., Cardwell, J. M., & Brodbelt, D. C. (2012) · Journal of Small Animal Practice · doi:10.1111/j.1748-5827.2011.01220.x · Cited by 142
Cambridge, W. A., Fairfield, C., Powell, J. J., Harrison, E. M., Søreide, K., Wigmore, S. J., & Guest, R. V. (2021) · Annals of Surgery · doi:10.1097/sla.0000000000003801 · Cited by 141
“) critically appraised the quality of each study independently using the Newcastle-Ottawa scale for nonrandomized studies.(16) Funnel plots were constructed to assess publication bias for outcomes reported by a minimum of 8 studies.”
MethodZaid, H. B., Parker, W. P., Safdar, N. S., Gershman, B., Erwin, P. J., Murad, M. H., Boorjian, S. A., Costello, B. A., Thompson, R. H., & Leibovich, B. C. (2016) · The Journal of Urology · doi:10.1016/j.juro.2016.07.079 · Cited by 140
“Newcastle-Ottawa Scale for cohort studies to assess the risk of bias.(12) This tool was applied uniformly to each study (supplementary Appendix 1, http://jurology.”
MethodA Conceptual Framework for Critical Appraisal in Systematic Mixed Studies Reviews
Hong, Q. N., & Pluye, P. (2019) · Journal of Mixed Methods Research · doi:10.1177/1558689818770058 · Cited by 139
“Currently, there are more than 500 critical appraisal tools for various study designs (Bai, Shukla, Bak, & Wells, 2012; Deeks et al., 2003; Katrak, Bialocerkowski, Massy-Westropp, Kumar, & Grimmer, 2004; West et al., 2002), but there is no clear guidance regarding which tool and approach to use, nor how or why to use…”
BackgroundAndronis, L., Barton, P. M., & Bryan, S. (2009) · Health Technology Assessment · doi:10.3310/hta13290 · Cited by 174
HIV-Associated Cognitive Impairment in Perinatally Infected Children: A Meta-analysis
Phillips, N. J., Amos, T., Kuo, C., Hoare, J., Ipser, J., Thomas, K. G. F., & Stein, D. J. (2016) · PEDIATRICS · doi:10.1542/peds.2016-0893 · Cited by 139
On the use of systematic reviews to inform environmental policies
Bilotta, G. S., Milner, A. M., & Boyd, I. (2014) · Environmental Science & Policy · doi:10.1016/j.envsci.2014.05.010 · Cited by 137
Bell, C. M., Hatch, W. V., Fischer, H. D., Cernat, G., Paterson, J. M., Gruneir, A., Gill, S. S., Bronskill, S. E., Anderson, G., & Rochon, P. A. (2009) · JAMA · doi:10.1001/jama.2009.683 · Cited by 146
Little, P., Turner, S., Rumsby, K., Warner, G., Moore, M., Lowes, J., Smith, H., Hawke, C., Turner, D., Leydon, G., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13190 · Cited by 137
Berlim, M. T., McGirr, A., den Eynde, F. V., Fleck, M. P., & Giacobbe, P. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.02.016 · Cited by 148
Srinivasan, M., Vazquez, L., Rieder, P., Moráguez, O. D., Bernard, J., & Belser, U. C. (2013) · Clinical Oral Implants Research · doi:10.1111/clr.12125 · Cited by 136
Schumacher, S., Niemeyer, H., Engel, S., Cwik, J. C., Laufer, S., Klusmann, H., & Knaevelsrud, C. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.02.005 · Cited by 128
Anaesthesia Management for Awake Craniotomy: Systematic Review and Meta-Analysis
Stevanovic, A., Rossaint, R., Veldeman, M., Bilotta, F., & Coburn, M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0156448 · Cited by 153
“Furthermore, the estimated treatment-effect tends to be larger in non-randomised studies [80].”
BackgroundStenneberg, M. S., Rood, M., de Bie, R. A., Schmitt, M. A., Cattrysse, E., & Scholten-Peeters, G. G. M. (2016) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2016.10.003 · Cited by 149
Prospective validation of first-trimester combined screening for trisomy 21
Kagan, K. O., Etchegaray, A., Zhou, Y., Wright, D., & Nicolaides, K. H. (2009) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.6412 · Cited by 132
Most clinical tests cannot accurately diagnose rotator cuff pathology: a systematic review
Hughes, P. C., Taylor, N. F., & Green, R. A. (2008) · Australian Journal of Physiotherapy · doi:10.1016/s0004-9514(08)70022-9 · Cited by 131
Valentine, J. C., & Thompson, S. G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1064 · Cited by 130
Male circumcision for prevention of heterosexual acquisition of HIV in men
Siegfried, N., Muller, M., Volmink, J., Deeks, J. J., Egger, M., Low, N. N., Weiss, H. H., Walker, S. A., & Williamson, P. R. (2003) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003362 · Cited by 128
Self-citationAtkins, D., Fink, K., & Slutsky, J. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-142-12_part_2-200506211-00002 · Cited by 129
“As reviewed in another paper in this supplement (15), the frequent inclusion of nonrandomized studies has stimulated a variety of approaches to evaluating quality of observational studies and other study designs (15, 16).”
BackgroundSurgery for rotator cuff tears
Karjalainen, T., Jain, N. B., Heikkinen, J., Johnston, R. V., Page, C. M., & Buchbinder, R. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013502 · Cited by 165
Warner, L., Klausner, J. D., Rietmeijer, C. A., Malotte, C. K., O'Donnell, L., Margolis, A. D., Greenwood, G. L., Richardson, D., Vrungos, S., O'Donnell, C. R., et al. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050135 · Cited by 128
“primary benefits obtained from randomization [24,25].”
BackgroundRoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions
Seo, H.-J., Kim, S. Y., Lee, Y. J., & Park, J.-E. (2023) · Korean Journal of Family Medicine · doi:10.4082/kjfm.23.0034 · Cited by 128
“13) This might result in incomparable comparison groups.”
BackgroundRichardson, A., McNoe, B., Derrett, S., & Harcombe, H. (2018) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2018.03.018 · Cited by 127
Communication about environmental health risks: A systematic review
Fitzpatrick‐Lewis, D., Yost, J., Ciliska, D. K., & Krishnaratne, S. (2010) · Environmental Health · doi:10.1186/1476-069x-9-67 · Cited by 127
Mari, A., Campi, R., Tellini, R., Gandaglia, G., Albisinni, S., Abufaraj, M. M., Hatzichristodoulou, G., Montorsi, F., van Velthoven, R., Carini, M., et al. (2017) · World Journal of Urology · doi:10.1007/s00345-017-2115-4 · Cited by 126
Fernandéz‐Pello, S., Hora, M., Kuusk, T., Tahbaz, R., Dabestani, S., Abu‐Ghanem, Y., Albiges, L. K., Giles, R. H., Hofmann, F., Kuczyk, M. A., et al. (2019) · European Urology Oncology · doi:10.1016/j.euo.2019.04.005 · Cited by 126
Weight change with atypical antipsychotics in the treatment of schizophrenia
Haddad, P. M. (2005) · Journal of Psychopharmacology · doi:10.1177/0269881105058378 · Cited by 148
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.