Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
(1963) · Medical Entomology and Zoology · Cited by 15,085
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationStang, A. (2010) · European Journal of Epidemiology · doi:10.1007/s10654-010-9491-z · Cited by 17,396
“concluded that with a few caveats (missing item for the appropriateness of the analysis, lack of information related to the reliability and validity), the NOS is “suitable for use in a systematic review” and is “easy to use” [3].”
BackgroundMethodInfluentialMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
“[305] [306] Bias jeopardises even RCTs, however, if investigators carry out such trials improperly.”
BackgroundSelf-citationGRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Systematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,089
Weight Gain During Pregnancy: Reexamining the Guidelines
Rasmussen, K. M., & Yaktine, A. L. (2009) · Cited by 4,473
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Prince, S. A., Adamo, K. B., Hamel, M., Hardt, J., Connor Gorber, S., & Tremblay, M. (2008) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/1479-5868-5-56 · Cited by 3,162
“The Downs and Black instrument was recommended for assessing risk of bias in observational studies in a recent systematic review [8] and other assessments [9] and was employed in this review to assess study quality including reporting, external validity, and internal validity (bias).”
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Hoy, D., Brooks, P., Woolf, A., Blyth, F., March, L., Bain, C., Baker, P., Smith, E., & Buchbinder, R. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.11.014 · Cited by 2,555
“Reviews have found that tools used for assessing quality of nonrandomized studies are poorly developed or have been specifically designed to assess randomized controlled trials and thus fail to include key criteria important for nonrandomized studies [12].”
BackgroundMethodMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2012) · International Journal of Surgery · doi:10.1016/j.ijsu.2011.10.001 · Cited by 2,345
“Non-randomised trials tend to result in larger estimated treat-ment effects than randomised trials [305,306].”
BackgroundSelf-citationThomas, B. H., Ciliska, D., Dobbins, M., & Micucci, S. (2004) · Worldviews on Evidence-Based Nursing · doi:10.1111/j.1524-475x.2004.04006.x · Cited by 2,118
“Both of these instruments were recommended for future use (Deeks et al. 2003).”
BackgroundMethodInfluentialZeng, X., Zhang, Y., Kwong, J. S. W., Zhang, C., Li, S., Sun, F., Niu, Y., & Du, L. (2015) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12141 · Cited by 2,082
“The latter tool that was developed by Reisch et al, and was major for drug therapeutic studies, and is now used by the Cochrane Inflammatory Bowel Disease Group (Inflammatory Bowel Disease and Functional Bowel Disorders Review Group) (26).”
BackgroundMethodInfluentialGorber, S. C., Tremblay, M., Moher, D., & Gorber, B. (2007) · Obesity Reviews · doi:10.1111/j.1467-789x.2007.00347.x · Cited by 1,945
Screening for breast cancer with mammography
Gøtzsche, P. C., & Jørgensen, K. J. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001877.pub5 · Cited by 1,869
Developing and Evaluating Complex Interventions: New Guidance
Craig, P., Dieppe, P. A., MacIntyre, S., Michie, S., Nazareth, I., & Petticrew, M. P. (2006) · UCL Discovery (University College London) · Cited by 1,974
Self-citationMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.004 · Cited by 1,858
Self-citationInterpretation of the evidence for the efficacy and safety of statin therapy
Collins, R., Reith, C., Emberson, J., Armitage, J., Baigent, C., Blackwell, L., Blumenthal, R., Danesh, J., Smith, G. D., DeMets, D., et al. (2016) · The Lancet · doi:10.1016/s0140-6736(16)31357-5 · Cited by 1,857
“been criticized (chiefly because of concerns about the methods used to select the studies and compare the results) and their findings have been inconsistent.(106) It has”
ResultA Systematic Review of Risk Factors for Intimate Partner Violence
Capaldi, D. M., Knoble, N. B., Shortt, J. W., & Kim, H. K. (2012) · Partner Abuse · doi:10.1891/1946-6560.3.2.231 · Cited by 1,844
“…were removed, study quality was assessed using a modification of the Downs and Black (1998) checklist—a well-validated appraisal tool with high inter-rater reliability approved for use in systematic reviews (Centre for Reviews and Dissemination, 2009; Deeks et al., 2003; Higgins & Green, 2011).”
Ma, L.-L., Wang, Y.-Y., Yang, Z.-H., Huang, D., Weng, H., & Zeng, X.-T. (2020) · Military Medical Research · doi:10.1186/s40779-020-00238-8 · Cited by 1,648
Armijo‐Olivo, S., Stiles, C. R., Hagen, N. A., Biondo, P. D., & Cummings, G. G. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2010.01516.x · Cited by 1,539
Sanderson, S., Tatt, I. D., & Higgins, J. P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym018 · Cited by 1,531
Evaluation of the Quality of Prognosis Studies in Systematic Reviews
Hayden, J. A., Côté, P., & Bombardier, C. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-6-200603210-00010 · Cited by 1,452
“Our results are similar to the findings of Deeks and colleagues (4), who looked at systematic reviews of intervention effectiveness, including nonrandomized studies.”
BackgroundResultGRADE guidelines: 9. Rating up the quality of evidence
Guyatt, G. H., Oxman, A. D., Sultan, S., Glasziou, P., Akl, E. A., Alonso-Coello, P., Atkins, D., Kunz, R., Brozek, J., Montori, V., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.06.004 · Cited by 1,336
“Furthermore, residual confounding may be substantial even when good prognostic data are available, and adjusting for case mix is not always successful [15].”
BackgroundEffect of lower sodium intake on health: systematic review and meta-analyses
Aburto, N. J., Ziolkovska, A., Hooper, L., Elliott, P., Cappuccio, F. P., & Meerpohl, J. J. (2013) · BMJ · doi:10.1136/bmj.f1326 · Cited by 1,242
Kim, S. Y., Park, J. E., Lee, Y. J., Seo, H.-J., Sheen, S.-S., Hahn, S., Jang, B.-H., & Son, H.-J. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.016 · Cited by 1,165
“[4].”
Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107
Gorber, S. C., Schofield-Hurwitz, S., Hardt, J., Levasseur, G., & Tremblay, M. (2009) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntn010 · Cited by 1,079
Adverse childhood experiences and associated health outcomes: A systematic review and meta-analysis
Petruccelli, K., Davis, J., & Berman, T. (2019) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2019.104127 · Cited by 1,026
Aburto, N. J., Hanson, S., Gutierrez, H., Hooper, L., Elliott, P., & Cappuccio, F. P. (2013) · BMJ · doi:10.1136/bmj.f1378 · Cited by 938
Craig, P., Cooper, C., Gunnell, D., Haw, S., Lawson, K., Macintyre, S., Ogilvie, D., Petticrew, M., Reeves, B., Sutton, M., et al. (2012) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2011-200375 · Cited by 912
“In the case of non-randomised studies, there is no such general solution to the pervasive problem of confounding.(20) Instead there is a range of partial solutions which can be used in some, often very restricted, circumstances but not others.”
BackgroundSelf-citationGlobal reach of ageism on older persons' health: a systematic review
Chang, E.-S., Kannoth, S., Levy, S., Wang, S.-Y., Lee, J. E., & Levy, B. R. (2020) · PLOS ONE · doi:10.1371/journal.pone.0220857 · Cited by 912
Hagerman, R. J., Berry-Kravis, E., Hazlett, H. C., Bailey, D. B., Moine, H., Kooy, R. F., Tassone, F., Gantois, I., Sonenberg, N., Mandel, J. L., et al. (2017) · Nature Reviews Disease Primers · doi:10.1038/nrdp.2017.65 · Cited by 838
Bolen, S., Feldman, L., Vassy, J., Wilson, L., Yeh, H.-C., Marinopoulos, S., Wiley, C., Selvin, E., Wilson, R., Bass, E. B., et al. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-6-200709180-00178 · Cited by 823
Individual determinants of research utilization by nurses: a systematic review update
Squires, J. E., Estabrooks, C. A., Gustavsson, P., & Wallin, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-1 · Cited by 798
“The Quality Assessment Tool for Quantitative Studies Tool, developed for the Canadian Effective Public Health Practice Project, has been judged suitable to be used in systematic reviews of interventions [24,25].”
MethodSocioeconomic Inequality and Caries: A Systematic Review and Meta-Analysis
Schwendicke, F., Dörfer, C. E., Schlattmann, P., Page, L. F., Thomson, W. M., & Paris, S. (2015) · Journal of Dental Research · doi:10.1177/0022034514557546 · Cited by 786
“The included studies were nevertheless regarded as “fit for the purpose” of analyzing social inequality (Petticrew and Roberts 2003), since such studies have been found to increase general uncertainty rather than introduce systematic bias (Deeks et al. 2003).”
BackgroundCook, D. A., & Reed, D. A. (2015) · Academic Medicine · doi:10.1097/acm.0000000000000786 · Cited by 781
Bachelot, T., Romieu, G., Campone, M., Diéras, V., Cropet, C., Dalenc, F., Jimenez, M., Le Rhun, E., Pierga, J.-Y., Gonçalves, A., et al. (2013) · The Lancet Oncology · doi:10.1016/s1470-2045(12)70432-1 · Cited by 737
Petrelli, F., Tomasello, G., Borgonovo, K., Ghidini, M., Turati, L., Dallera, P., Passalacqua, R., Sgroi, G., & Barni, S. (2017) · JAMA Oncology · doi:10.1001/jamaoncol.2016.4227 · Cited by 729
Yan, T. D., Black, D., Bannon, P. G., & McCaughan, B. C. (2009) · Journal of Clinical Oncology · doi:10.1200/jco.2008.18.2733 · Cited by 725
Long-term opioid management for chronic noncancer pain
Noble, M., Treadwell, J. R., Tregear, S. J., Coates, V. H., Wiffen, P. J., Akafomo, C., Schoelles, K. M., & Chou, R. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006605.pub2 · Cited by 735
Teixeira, P. J., Carraça, E. V., Marques, M. M., Rutter, H., Oppert, J.-M., De Bourdeaudhuij, I., Lakerveld, J., & Brug, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0323-6 · Cited by 711
“The current adaptation was based on recommendations from several authors [31,32], and has been used in a previous systematic review conducted as part of the SPOTLIGHT project [33].”
MethodCottrell, M. A., Galea, O. A., O’Leary, S. P., Hill, A. J., & Russell, T. G. (2017) · Clinical Rehabilitation · doi:10.1177/0269215516645148 · Cited by 708
Siega-Riz, A. M., Viswanathan, M., Moos, M.-K., Deierlein, A., Mumford, S., Knaack, J., Thieda, P., Lux, L. J., & Lohr, K. N. (2009) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2009.07.002 · Cited by 708
Jakobsen, J. C., Wetterslev, J., Winkel, P., Lange, T., & Gluud, C. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-120 · Cited by 704
“analyses of trials with low risk of bias are more reliable than results of single large trials [6-11].”
ResultInterventions to promote walking: systematic review
Ogilvie, D., Foster, C. E., Rothnie, H., Cavill, N., Hamilton, V., Fitzsimons, C. F., & Mutrie, N. (2007) · BMJ · doi:10.1136/bmj.39198.722720.be · Cited by 695
Philips, Z., Ginnelly, L., Sculpher, M., Claxton, K., Golder, S., Riemsma, R., Woolacoot, N., & Glanville, J. (2004) · Health Technology Assessment · doi:10.3310/hta8360 · Cited by 697
Indirect comparisons of competing interventions
Glenny, A., Altman, D., Song, F., Sakarovitch, C., Deeks, J., D'Amico, R., Bradburn, M., & Eastwood, A. (2005) · Health Technology Assessment · doi:10.3310/hta9260 · Cited by 677
Self-citationSavović, J., Jones, H., Altman, D., Harris, R., Jűni, P., Pildal, J., Als-Nielsen, B., Balk, E., Gluud, C., Gluud, L., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16350 · Cited by 668
Self-citationScreening for fetal aneuploidies at 11 to 13 weeks
Nicolaides, K. H. (2011) · Prenatal Diagnosis · doi:10.1002/pd.2637 · Cited by 642
Mazzaglia, G., Ambrosioni, E., Alacqua, M., Filippi, A., Sessa, E., Immordino, V., Borghi, C., Brignoli, O., Caputi, A. P., Cricelli, C., et al. (2009) · Circulation · doi:10.1161/circulationaha.108.830299 · Cited by 637
Bias in clinical intervention research.
Gluud, L. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj069 · Cited by 636
“Biasassociated with nonrandom allocation was also analyzed in a review summarizing the results of 69 studies that compared randomized trials with observational studies (21).”
BackgroundMethodPriorities Among Effective Clinical Preventive Services
Maciosek, M. V., Coffield, A. B., Edwards, N. M., Flottemesch, T. J., Goodman, M. J., & Solberg, L. I. (2006) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2006.03.012 · Cited by 625
Nuchal translucency and other first-trimester sonographic markers of chromosomal abnormalities
Nicolaides, K. H. (2004) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2004.03.090 · Cited by 626
Hobart, J., & Cano, S. (2009) · Health Technology Assessment · doi:10.3310/hta13120 · Cited by 621
Nowak, C., & Heinrichs, N. (2008) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-008-0033-0 · Cited by 621
“…reliability and inter-rater agreement and is easy to use; (iii) it is rather comprehensive pertaining to potential biases addressed (although Deeks et al. 2003, still recommended inclusion of additional items like one addressing baseline comparability of groups, which we appreciated by way…”
MethodTrastuzumab containing regimens for early breast cancer
Moja, L., Tagliabue, L., Balduzzi, S., Parmelli, E., Pistotti, V., Guarneri, V., & D'Amico, R. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006243.pub2 · Cited by 604
Self-citationBrinjikji, W., Diehn, F. E., Jarvik, J. G., Carr, C. M., Kallmes, D. F., Murad, M. H., & Luetmer, P. H. (2015) · American Journal of Neuroradiology · doi:10.3174/ajnr.a4498 · Cited by 590
Pollock, A., & Berge, E. (2018) · International Journal of Stroke · doi:10.1177/1747493017743796 · Cited by 530
“the existence of a wide number of scales to assess quality (71), the use of scales is explicitly discouraged by Cochrane (68) as the validity and transparency of such summative scales can be questioned (72).”
BackgroundMazzaferro, V., Bhoori, S., Sposito, C., Bongini, M., Langer, M., Miceli, R., & Mariani, L. (2011) · Liver Transplantation · doi:10.1002/lt.22365 · Cited by 593
When are randomised trials unnecessary? Picking signal from noise
Glasziou, P., Chalmers, I., Rawlins, M., & McCulloch, P. (2007) · BMJ · doi:10.1136/bmj.39070.527986.68 · Cited by 589
Birthweight and mortality in adulthood: a systematic review and meta-analysis
Risnes, K. R., Vatten, L. J., Baker, J. L., Jameson, K., Sovio, U., Kajantie, E., Osler, M., Morley, R., Jokela, M., Painter, R. C., et al. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyq267 · Cited by 576
Gameplay Engagement and Learning in Game-Based Learning: A Systematic Review
Abdul Jabbar, A. I., & Felicia, P. (2015) · Review of Educational Research · doi:10.3102/0034654315577210 · Cited by 576
“…from 1 (i.e., the weakest form of evidence) to 5 (i.e., the strongest form of evidence) for the study design, based on a classification and design hierarchy adapted from the Centre for Reviews and Dissemination (CRD) Report 4 (cited in Deeks et al., 2003), as illustrated in Table S2 (online only).”
MethodThe frequency of influenza and bacterial coinfection: a systematic review and meta-analysis.
Klein, E. Y., Monteforte, B., Gupta, A., Jiang, W., May, L., Hsieh, Y., & Dugas, A. (2016) · Influenza and Other Respiratory Viruses · doi:10.1111/irv.12398 · Cited by 574
Criteria for the systematic review of health promotion and public health interventions
Jackson, N., & Waters, E. (2005) · Health Promotion International · doi:10.1093/heapro/dai022 · Cited by 572
“(Deeks et al., 2003), and can be used for RCTs, quasi-experimental studies and uncontrolled studies.”
BackgroundMethodImpact of multiple cesarean deliveries on maternal morbidity: a systematic review
Marshall, N. E., Fu, R., & Guise, J.-M. (2011) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2011.06.035 · Cited by 565
Peer-Delivered Recovery Support Services for Addictions in the United States: A Systematic Review
Bassuk, E. L., Hanson, J., Greene, R. N., Richard, M., & Laudet, A. (2016) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2016.01.003 · Cited by 551
“…has beenwidely used for assessing quality in public health studies and has been shown to have adequate reliability, content, and construct validity with observational studies (Armijo-Olivo, Stiles, Hagen, Biondo & Cummings, 2012; Deeks et al., 2003; Thomas, Ciliska, Dobbins, & Micucci, 2004).”
BackgroundLloyd-Evans, B., Mayo-Wilson, E., Harrison, B., Istead, H., Brown, E., Pilling, S., Johnson, S., & Kendall, T. (2014) · BMC Psychiatry · doi:10.1186/1471-244x-14-39 · Cited by 558
“In line with best practice guidelines for systematic reviews [54], we have not sought to supplement existing randomised trial evidence by including non-randomised trials in our review [55].”
Burr, J., Mowatt, G., Hernández, R., Siddiqui, M., Cook, J., Lourenco, T., Ramsay, C., Vale, L., Fraser, C., Azuara-Blanco, A., et al. (2007) · Health Technology Assessment · doi:10.3310/hta11410 · Cited by 549
Self-citationFukuchi, C. A., Fukuchi, R. K., & Duarte, M. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1063-z · Cited by 526
“From the original checklist, only item 27 was removed due to its ambiguity [43].”
BackgroundAnglemyer, A., Horvath, H. T., & Bero, L. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub2 · Cited by 534
Alldred, S. K., Takwoingi, Y., Guo, B., Pennant, M., Deeks, J. J., Neilson, J. P., & Alfirevic, Z. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012599 · Cited by 529
Self-citationSpurling, G. K., Mansfield, P. R., Montgomery, B. D., Lexchin, J., Doust, J., Othman, N., & Vitry, A. I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000352 · Cited by 526
“80%), and use of appropriate statistical tests [23].”
BackgroundThompson, A. K., Kelley, B. F., Prokop, L. J., Murad, M. H., & Baum, C. L. (2016) · JAMA Dermatology · doi:10.1001/jamadermatol.2015.4994 · Cited by 521
Umer, A., Kelley, G. A., Cottrell, L. E., Giacobbi, P., Innes, K. E., & Lilly, C. L. (2017) · BMC Public Health · doi:10.1186/s12889-017-4691-z · Cited by 515
Systematic Review and Meta-Analysis of Preterm Birth and Later Systolic Blood Pressure
de Jong, F., Monuteaux, M. C., van Elburg, R. M., Gillman, M. W., & Belfort, M. B. (2012) · Hypertension · doi:10.1161/hypertensionaha.111.181784 · Cited by 516
Preventing occupational stress in healthcare workers
Marine, A., Ruotsalainen, J. H., Serra, C., & Verbeek, J. H. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002892.pub2 · Cited by 499
Male circumcision for prevention of heterosexual acquisition of HIV in men
Siegfried, N., Muller, M., Deeks, J. J., & Volmink, J. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003362.pub2 · Cited by 494
Self-citationAdams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486
Yu, J.-T., Xu, W., Tan, C.-C., Andrieu, S., Suckling, J., Evangelou, E., Pan, A., Zhang, C., Jia, J., Feng, L., et al. (2020) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp-2019-321913 · Cited by 482
The Magic of Randomization versus the Myth of Real-World Evidence
Collins, R., Bowman, L., Landray, M., & Peto, R. (2020) · New England Journal of Medicine · doi:10.1056/nejmsb1901642 · Cited by 478
Cancelliere, C., Cassidy, J. D., Ammendolia, C., & Côté, P. (2011) · BMC Public Health · doi:10.1186/1471-2458-11-395 · Cited by 460
Psychological factors affecting the outcome of total hip and knee arthroplasty: a systematic review
Vissers, M. M., Bussmann, J. B., Verhaar, J. A. N., Busschbach, J. J. V., Bierma-Zeinstra, S. M. A., & Reijman, M. (2012) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2011.07.003 · Cited by 473
Motivational interviewing for substance abuse
Smedslund, G., Berg, R. C., Hammerstrøm, K. T., Steiro, A., Leiknes, K. A., Dahl, H. M., & Karlsen, K. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008063.pub2 · Cited by 408
A systematic review of factors affecting vaccine uptake in young children
Smith, L. E., Amlôt, R., Weinman, J., Yiend, J., & Rubin, G. J. (2017) · Vaccine · doi:10.1016/j.vaccine.2017.09.046 · Cited by 467
“Risk of bias was assessed using an adaptation of the Downs & Black [17] checklist, which is suitable for use in systematic reviews [18] and has been validated [19].”
MethodGilmore, B., & McAuliffe, E. (2013) · BMC Public Health · doi:10.1186/1471-2458-13-847 · Cited by 460
“they work, should be selected by the communities, should be should be supported by the health system but not necessarily a part of fessional workers” [19]”
BackgroundMethodAdamo, K. B., Prince, S. A., Tricco, A. C., Connor-Gorber, S., & Tremblay, M. (2009) · International Journal of Pediatric Obesity · doi:10.1080/17477160802315010 · Cited by 463
Is there evidence that walking groups have health benefits? A systematic review and meta-analysis
Hanson, S., & Jones, A. (2015) · British Journal of Sports Medicine · doi:10.1136/bjsports-2014-094157 · Cited by 452
“As not all studies were randomised controlled trials, a tool used by Ogilvie et al 20 was adapted to assess risk of bias and internal validity 26 with nine items on a binary scale.”
MethodAssessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017) · doi:10.23970/ahrqepcmethguide2 · Cited by 448
Epidemiology of fragile X syndrome: A systematic review and meta‐analysis
Hunter, J., Rivero‐Arias, O., Angelov, A., Kim, E., Fotheringham, I., & Leal, J. (2014) · American Journal of Medical Genetics Part A · doi:10.1002/ajmg.a.36511 · Cited by 442
Margulis, A. V., Pladevall, M., Riera-Guardia, N., Varas-Lorenzo, C., Hazell, L., Berkman, N., Viswanathan, M., & Perez-Gutthann, S. (2014) · Clinical Epidemiology · doi:10.2147/clep.s66677 · Cited by 441
Kesselheim, A. S. (2008) · JAMA · doi:10.1001/jama.2008.758 · Cited by 431
Tokhi, M., Comrie-Thomson, L., Davis, J., Portela, A., Chersich, M., & Luchters, S. (2018) · PLOS ONE · doi:10.1371/journal.pone.0191620 · Cited by 411
“Study Study design [39] Intervention Setting Duration”
Drosos, G. C., Vedder, D., Houben, E., Boekel, L., Atzeni, F., Badreh, S., Boumpas, D. T., Brodin, N., Bruce, I. N., González-Gay, M. Á., et al. (2022) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2021-221733 · Cited by 406
“Quality assessment was performed using the Cochrane risk- of-bias tool 33 for randomised clinical trials and the Newcastle-Ottawa Scale 34 for observational studies.”
MethodInterventions to prevent global childhood overweight and obesity: a systematic review
Bleich, S. N., Vercammen, K. A., Zatz, L. Y., Frelier, J. M., Ebbeling, C. B., & Peeters, A. (2018) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(17)30358-3 · Cited by 420
Preterm Birth and the Metabolic Syndrome in Adult Life: A Systematic Review and Meta-analysis
Parkinson, J. R. C., Hyde, M. J., Gale, C., Santhakumaran, S., & Modi, N. (2013) · Pediatrics · doi:10.1542/peds.2012-2177 · Cited by 412
Brown, K. F., Kroll, J. S., Hudson, M. J., Ramsay, M., Green, J., Long, S. J., Vincent, C. A., Fraser, G., & Sevdalis, N. (2010) · Vaccine · doi:10.1016/j.vaccine.2010.04.052 · Cited by 412
The FMR1 premutation and reproduction
Wittenberger, M. D., Hagerman, R. J., Sherman, S. L., McConkie-Rosell, A., Welt, C. K., Rebar, R. W., Corrigan, E. C., Simpson, J. L., & Nelson, L. M. (2007) · Fertility and Sterility · doi:10.1016/j.fertnstert.2006.09.004 · Cited by 412
Diercks, R., Bron, C., Dorrestijn, O., Meskers, C., Naber, R., de Ruiter, T., Willems, J., Winters, J., & van der Woude, H. J. (2014) · Acta Orthopaedica · doi:10.3109/17453674.2014.920991 · Cited by 397
Vaginal Birth After Cesarean: New Insights on Maternal and Neonatal Outcomes
Guise, J.-M., Denman, M. A., Emeis, C., Marshall, N., Walker, M., Fu, R., Janik, R., Nygren, P., Eden, K. B., & McDonagh, M. (2010) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e3181df925f · Cited by 402
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.