Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
(1963) · Medical Entomology and Zoology · Cited by 15,085
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
Items 401–500 of 1,024
Kollias, A., Dafni, M., Poulidakis, E., Ntineri, A., & Stergiou, G. S. (2014) · Journal of Hypertension · doi:10.1097/hjh.0000000000000384 · Cited by 125
Schwendicke, F., & Stolpe, M. (2014) · Journal of Endodontics · doi:10.1016/j.joen.2014.07.028 · Cited by 140
“However, such bias may not systematically affect the results but could merely increase the level of uncertainty, which we accounted for by means of sensitivity analyses (25).”
A Research Methodology to Conduct Effective Research Syntheses: Meta-Analysis
Üstün, U., & Eryılmaz, A. (2014) · Education and Science · doi:10.15390/eb.2014.3379 · Cited by 123
“There are also a number of studies to review these assessment tools for study quality in the literature (Deeks et al., 2003; Herbison et al., 2006; Jüni et al., 2001; Jüni, Witschi, Bloch, & Egger, 1999; Wells & Littell, 2009).”
BackgroundMethodThe role of tumor-infiltrating lymphocytes in cholangiocarcinoma
Liu, D., Heij, L. R., Czigány, Z., Dahl, E., Lang, S. A., Ulmer, T. F., Luedde, T., Neumann, U. P., & Bednarsch, J. (2022) · Journal of Experimental & Clinical Cancer Research · doi:10.1186/s13046-022-02340-2 · Cited by 133
“All included translational studies reporting oncological outcome were evaluated in accordance with the Newcastle-Ottawa scale [16].”
MethodBray, I., Reece, R., Sinnett, D., Martin, F., & Hayward, R. (2022) · Environmental Research · doi:10.1016/j.envres.2022.114081 · Cited by 124
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005; Deeks et al., 2003).”
Problem–Based Learning in Dental Education: A Systematic Review of the Literature
Bassir, S. H., Sadr‐Eshkevari, P., Amirikhorheh, S., & Karimbux, N. Y. (2014) · Journal of Dental Education · doi:10.1002/j.0022-0337.2014.78.1.tb05661.x · Cited by 144
Immersive Interactive Technologies for Positive Change: A Scoping Review and Design Considerations
Kitson, A., Prpa, M., & Riecke, B. E. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01354 · Cited by 163
“A review article looked at 60 research evaluation systems and identified the Downs and Black checklist as one of the best evaluation systems available (Deeks et al., 2003).”
BackgroundCase study in major quotation errors: a critical commentary on the Newcastle–Ottawa scale
Stang, A., Jonas, S., & Poole, C. (2018) · European Journal of Epidemiology · doi:10.1007/s10654-018-0443-3 · Cited by 134
Verde, P. E., & Ohmann, C. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1122 · Cited by 123
“The idea behind this type of adjustment comes from the empirical finding of Deeks et al. (2003), which describe that non-randomized trials tend to present unbalance in patients’ characteristics between studies arms.”
BackgroundSilverman, S. R., Schertz, L., Yuen, H. K., Lowman, J. D., & Bickel, C. S. (2012) · Spinal Cord · doi:10.1038/sc.2012.78 · Cited by 132
Preconceptional and prenatal screening for fragile X syndrome: Experience with 40 000 tests
Berkenstadt, M., Ries‐Levavi, L., Cuckle, H., Peleg, L., & Barkai, G. (2007) · Prenatal Diagnosis · doi:10.1002/pd.1815 · Cited by 124
Ottenheijm, R. P. G., Jansen, M. J., Staal, J. B., den Bruel, A. V., Weijers, R. E., de Bie, R. A., & Dinant, G. (2010) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2010.07.017 · Cited by 136
Ye, S., Yang, J., Cao, D., Lang, J., & Shen, K. (2014) · International Journal of Gynecological Cancer · doi:10.1097/igc.0000000000000207 · Cited by 122
“Methodological Assessment of Studies Methodological qualities of the studies were assessed as recommended by the National Health Service Center for Reviews and Dissemination.(11) Given that there was no reference standard on the topic, we rated the studies on their internal validity, such as that of Vistad et al,(8) b…”
MethodBrennan, A., Bansback, N. J., Nixon, R. M., Madan, J. J., Harrison, M. J., Watson, K. D., & Symmons, D. P. M. (2007) · Rheumatology · doi:10.1093/rheumatology/kem115 · Cited by 125
To what extent do nurses use research in clinical practice? A systematic review
Squires, J. E., Hutchinson, A. M., Boström, A., O’Rourke, H. M., Cobban, S. J., & Estabrooks, C. A. (2011) · Implementation Science · doi:10.1186/1748-5908-6-21 · Cited by 159
“This tool has been judged suitable to be used in systematic reviews of effectiveness (measuring interventions) [47], and been shown to have content and construct validity [48].”
MethodBae, J. (2016) · Epidemiology and Health · doi:10.4178/epih.e2016014 · Cited by 136
“The NOS is suitable for SR due to its easy application [11], and has been widely used due to recommendations from the Cochrane Collaboration [12].”
MethodThe Diagnostic Accuracy of Special Tests for Rotator Cuff Tear: The ROW Cohort Study
Jain, N. B., Luz, J., Higgins, L. D., Dong, Y., Warner, J. J. P., Matzkin, E. G., & Katz, J. N. (2016) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000000566 · Cited by 143
Langan, E., Blake, C., & Lonsdale, C. (2013) · Psychology of Sport and Exercise · doi:10.1016/j.psychsport.2012.06.007 · Cited by 122
Benton, J. S., Anderson, J., Hunter, R. F., & French, D. P. (2016) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-016-0433-3 · Cited by 120
“This tool was chosen because it is specifically designed for non-randomised studies [32].”
MethodSleep Problems and Workplace Violence: A Systematic Review and Meta-Analysis
Magnavita, N., Stasio, E. D., Capitanelli, I., Lops, E. A., Chirico, F., & Garbarino, S. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00997 · Cited by 163
“The quality of cohort and case-control studies was assessed by the Newcastle-Ottawa Scale (NOS) that evaluates selection, comparability and exposure criteria, attributing a maximum score of 9 points (Deeks et al., 2003; Wells et al., 2012).”
MethodSystematic review of surgical management of synchronous colorectal liver metastases
Lykoudis, P. M., O’Reilly, D. J., Nastos, C., & Fusai, G. (2014) · British journal of surgery · doi:10.1002/bjs.9449 · Cited by 134
Mindfulness-Based Program for Infertility: Efficacy study
Galhardo, A., Cunha, M., & Pinto-Gouveia, J. (2013) · Fertility and Sterility · doi:10.1016/j.fertnstert.2013.05.036 · Cited by 120
Gait modifications to change lower extremity gait biomechanics in runners: a systematic review
Napier, C., Cochrane, C. K., Taunton, J. E., & Hunt, M. A. (2015) · British Journal of Sports Medicine · doi:10.1136/bjsports-2014-094393 · Cited by 120
McPherson, P., Krotofil, J., & Killaspy, H. (2018) · BMC Psychiatry · doi:10.1186/s12888-018-1725-8 · Cited by 112
“The Quality Assessment Tool for Quantitative Studies (QATQS) [14] was used to assess article quality; the tool is recommended for use in systematic reviews by Deeks and colleagues [15] and displays acceptable psychometric properties [14].”
MethodModifiable risk factors for patellar tendinopathy in athletes: a systematic review and meta-analysis
Sprague, A. L., Smith, A. H., Knox, P. J., Pohlig, R. T., & Silbernagel, K. G. (2018) · British Journal of Sports Medicine · doi:10.1136/bjsports-2017-099000 · Cited by 134
Uhlig, J., Strauß, A., Rücker, G., Hosseini, A. S. A., Lotz, J., Trojan, L., Kim, H. S., & Uhlig, A. (2018) · European Radiology · doi:10.1007/s00330-018-5660-3 · Cited by 121
Patel, J. N., Klein, D. S., Sreekumar, S., Liporace, F. A., & Yoon, R. S. (2019) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-18-00425 · Cited by 132
Rocha, R. B., Dondossola, E. R., Grande, A. J., Colonetti, T., Ceretta, L. B., Passos, I. C., Quevedo, J., & da Rosa, M. I. (2016) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2016.08.004 · Cited by 119
Ribavirin for patients with Crimean–Congo haemorrhagic fever: a systematic review and meta-analysis
Aşçıoğlu, S., Leblebicioğlu, H., Vahaboğlu, H., & Chan, K. A. (2011) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkr136 · Cited by 132
Lundstrøm, L. H., Møller, A. M., Rosenstock, C. V., Astrup, G., Gätke, M. R., & Wetterslev, J. (2009) · British Journal of Anaesthesia · doi:10.1093/bja/aep124 · Cited by 121
Medical Respite Programs for Homeless Patients: A Systematic Review
Doran, K. M., Ragins, K. T., Gross, C. P., & Zerger, S. (2013) · Journal of Health Care for the Poor and Underserved · doi:10.1353/hpu.2013.0053 · Cited by 129
Systematic review of public-targeted communication interventions to improve antibiotic use
Cross, E. L. A., Tolfree, R., & Kipping, R. (2017) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkw520 · Cited by 118
Lam, S. K. H., & Owen, A. (2007) · BMJ · doi:10.1136/bmj.39343.511389.be · Cited by 116
What are effective strategies for the implementation of care bundles on ICUs: a systematic review
Borgert, M. J., Goossens, A., & Dongelmans, D. A. (2015) · Implementation Science · doi:10.1186/s13012-015-0306-1 · Cited by 137
“A valid checklist to assess the quality is currently lacking [17].”
Backgroundvan Meer, B. L., Meuffels, D. E., van Eijsden, W. A., Verhaar, J. A. N., Bierma‐Zeinstra, S., & Reijman, M. (2015) · British Journal of Sports Medicine · doi:10.1136/bjsports-2013-093258 · Cited by 131
“16 17”
Manchikanti, L., Singh, V., Smith, H. S., & Hirsch, J. A. (2009) · Pain Physician · doi:10.36076/ppj.2009/12/73 · Cited by 120
Primary Substance Use Prevention Programs for Children and Youth: A Systematic Review
Tremblay, M., Baydala, L., Khan, M., Currie, C., Morley, K., Burkholder, C., Davidson, R., & Stillar, A. (2020) · Pediatrics · doi:10.1542/peds.2019-2747 · Cited by 116
Treatment of Medial Tibial Stress Syndrome: A Systematic Review
Winters, M., Eskes, M., Weir, A., Moen, M. H., BACKX, F. J. G., & Bakker, E. W. P. (2013) · Sports Medicine · doi:10.1007/s40279-013-0087-0 · Cited by 116
Contribution of Fetal Tricuspid Regurgitation in First-Trimester Screening for Major Cardiac Defects
Pereira, S., Ganapathy, R., Syngelaki, A., Maiz, N., & Nicolaides, K. H. (2011) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e31821aa720 · Cited by 116
Research methods to change clinical practice for patients with rare cancers
Billingham, L. J., Malottki, K., & Steven, N. M. (2016) · The Lancet Oncology · doi:10.1016/s1470-2045(15)00396-4 · Cited by 118
Rowe, B., Guo, X., Villa‐Roel, C., Schull, M., Holroyd, B., Bullard, M., Vandermeer, B., Ospina, M., & Innes, G. (2011) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2010.00984.x · Cited by 125
Richards, R. E., van den Noort, J. C., Dekker, J., & Harlaar, J. (2016) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2016.07.006 · Cited by 116
Human recombinant protein C for severe sepsis and septic shock in adult and paediatric patients
Martí-Carvajal, A. J., Solà, I., Gluud, C. N., Lathyris, D., & Anand, V. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004388.pub6 · Cited by 140
Meyer, K., & Shea, J. (2017) · Nutrients · doi:10.3390/nu9070711 · Cited by 115
Kiyofuji, S., Graffeo, C. S., Perry, A., Murad, M. H., Flemming, K. D., Lanzino, G. S., Rangel-Castilla, L., & Brinjikji, W. (2017) · Journal of NeuroInterventional Surgery · doi:10.1136/neurintsurg-2017-013312 · Cited by 115
Effect of SGLT2-Inhibitors on Epicardial Adipose Tissue: A Meta-Analysis
Masson, W. M., Cobo, A. L., & Nogueira, J. P. (2021) · Cells · doi:10.3390/cells10082150 · Cited by 130
Spencer, K., Cowans, N. J., Avgidou, K., & Nicolaides, K. H. (2006) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.3809 · Cited by 117
Ultrasound Characteristics of Symptomatic Carotid Plaques: A Systematic Review and Meta-Analysis
Brinjikji, W., Rabinstein, A. A., Lanzino, G. S., Murad, M. H., Williamson, E. E., DeMarco, J. K., & Huston, J. (2015) · Cerebrovascular Diseases · doi:10.1159/000437339 · Cited by 114
Carr, S., Lhussier, M., Forster, N., Geddes, L., Deane, Pennington, M., Visram, S., White, M., Michie, S., Donaldson, C., et al. (2011) · Health Technology Assessment · doi:10.3310/hta15090 · Cited by 115
Efficacy of psychedelic treatments on depressive symptoms: A meta-analysis
Romeo, B., Karila, L., Martelli, C., & Benyamina, A. (2020) · Journal of Psychopharmacology · doi:10.1177/0269881120919957 · Cited by 113
“Quality assessment of the included studies was performed with the modified Newcastle-Ottawa Scale (Deeks et al., 2003) for open-label uncontrolled trials and with the Cochrane Collaboration’s Tool to assess the risk of bias (Higgins et al., 2003) for randomized doubled blind clinical trials.”
MethodMeta-analysis of anatomic resection versus nonanatomic resection for hepatocellular carcinoma
Zhou, Y., Xu, D., Wu, L., & Li, B. (2011) · Langenbeck s Archives of Surgery · doi:10.1007/s00423-011-0784-9 · Cited by 114
Sandercock, P. A., Berge, E., Dennis, M. S., Forbes, J., Hand, P. J., Kwan, J. S. K., Lewis, S. C., Lindley, R. I., Neilson, A. R., & Wardlaw, J. M. (2004) · Stroke · doi:10.1161/01.str.0000126871.98801.6e · Cited by 126
McFarlane, C. A., & Kaplan, I. (2012) · Transcultural Psychiatry · doi:10.1177/1363461512447608 · Cited by 113
“We rated the methodological quality of the studies using the Quality Assessment Tool (Thomas, Ciliska, Dobbins, & Micucci, 2004), which has been ranked as one of the best available for systematic reviews with multiple study types (Deeks et al., 2003).”
MethodHensen, B., Baggaley, R., Wong, V. J., Grabbe, K. L., Shaffer, N., Lo, Y. J., & Hargreaves, J. (2012) · Tropical Medicine & International Health · doi:10.1111/j.1365-3156.2011.02893.x · Cited by 115
“…yet only one of the NRS controlled for these in analyses and only the study from Botswana compared groups to determine whether they had any significant differences in characteristics, including social and demographic characteristics (Deeks et al. 2003; Creek et al. 2007; Stringer et al. 2001).”
BackgroundResultInfluentialAnderson, J., Ford, T. J., Soneson, E., Coon, J. T., Humphrey, A., Rogers, M., Moore, D. A., Jones, P. B., Clarke, E., & Howarth, E. (2018) · Psychological Medicine · doi:10.1017/s0033291718002490 · Cited by 135
“, 2012), which has been deemed suitable to use systematic reviews of effectiveness (Deeks et al., 2003).”
Digital transformation of organizations: what do we know and where to go next?
Jedynak, M., Czakon, W., Kuźniarska, A., & Mania, K. (2021) · Journal of Organizational Change Management · doi:10.1108/jocm-10-2020-0336 · Cited by 147
Napierala, H., Krüger, K., Kuschick, D., Heintze, C., Herrmann, W., & Holzinger, F. (2022) · International Journal of Integrated Care · doi:10.5334/ijic.6472 · Cited by 138
Iversen, L. H., Harling, H., Laurberg, S., Wille‐Jørgensen, P., & the Danish Colorectal Cancer Group (2006) · Colorectal Disease · doi:10.1111/j.1463-1318.2006.01095.x · Cited by 113
Chong, E. W.-T., Kreis, A. J., Wong, T. Y., Simpson, J. A., & Guymer, R. H. (2008) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2007.12.005 · Cited by 112
Kemp, J. L., MacDonald, D. A., Collins, N. J., Hatton, A. L., & Crossley, K. M. (2014) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-014-3943-9 · Cited by 112
“This has adequate reliability and validity for assessing nonrandomized studies [12, 13, 49].”
BackgroundEfficacy of tocilizumab in COVID‐19: A systematic review and meta‐analysis
Aziz, M., Haghbin, H., Sitta, E. A., Nawras, Y., Fatima, R., Sharma, S., Lee‐Smith, W., Duggan, J. M., Kammeyer, J. A., Hanrahan, J., et al. (2020) · Journal of Medical Virology · doi:10.1002/jmv.26509 · Cited by 118
“The risk of bias for each individual study was assessed using the NCOS and Risk Of Bias in nonrandomized studies—of Interventions (ROBINS‐I) tool for nonrandomized studies.(14,15) The publication bias was assessed qualitatively by visually assessing the funnel plot.”
MethodResultRasmussen, L. F., Grode, L. B., Lange, J., Barat, I., & Gregersen, M. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-040057 · Cited by 133
Selenium for preventing Kashin-Beck osteoarthropathy in children: a meta-analysis
Zou, K., Liu, G., Wu, T., & Du, L. (2008) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2008.06.011 · Cited by 113
“The quality of non-RCTs was assessed using a checklist by Deeks et al (2003).”
MethodA matching method for improving covariate balance in cost‐effectiveness analyses
Sekhon, J. S., & Grieve, R. (2011) · Health Economics · doi:10.1002/hec.1748 · Cited by 130
Reeves, B. C., Higgins, J. P. T., Ramsay, C., Shea, B., Tugwell, P., & Wells, G. A. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1068 · Cited by 112
“Moreover, regression adjustment using measured risk factors does not usually fully correct for the biases (Deeks et al., 2003).”
BackgroundThe Impact of Pain Assessment on Critically Ill Patients’ Outcomes: A Systematic Review
Georgiou, Ε., Hadjibalassi, M., Lambrinou, E., Andreou, P., & Papathanassoglou, E. D. E. (2015) · BioMed Research International · doi:10.1155/2015/503830 · Cited by 126
Grigoriu, A., Dinomais, M., Rémy‐Néris, O., & Brochard, S. (2015) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2015.05.002 · Cited by 111
Gibbons, C., Bruce, J., Carpenter, J. R., Wilson, A. P. R., Wilson, J. A., Pearson, A., Lamping, D. L., Krukowski, Z. H., & Reeves, B. (2011) · Health Technology Assessment · doi:10.3310/hta15300 · Cited by 116
Smith, K. E., Bambra, C., Joyce, K. E., Perkins, N., Hunter, D. J., & Blenkinsopp, E. A. (2009) · Journal of Public Health · doi:10.1093/pubmed/fdp002 · Cited by 109
“18,19,20,21 The critical appraisal criteria were applied with respect to the general design of studies, once it had been decided to include them (as opposed to being applied to the ability of studies to address the systematic review question), and the results were used for descriptive purposes…”
MethodThree risk of bias tools lead to opposite conclusions in observational research synthesis
Losilla, J.-M., Oliveras, I., Marin-Garcia, J. A., & Vives, J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.05.021 · Cited by 111
Shen, F., Cai, W.-S., Li, J., Feng, Z., Cao, J., & Xu, B. (2015) · Biological Trace Element Research · doi:10.1007/s12011-015-0304-9 · Cited by 113
“The qualities of all included studies were assessed using the Newcastle–Ottawa Scale (NOS) (Supplement Material Figure S1) [28].”
MethodThe safety of inactivated influenza vaccines in pregnancy for birth outcomes: a systematic review
Giles, M., Krishnaswamy, S., Macartney, K. K., & Cheng, A. (2018) · Human Vaccines & Immunotherapeutics · doi:10.1080/21645515.2018.1540807 · Cited by 115
Carollo, M., Crisafulli, S., Vitturi, G., Besco, M., Hinek, D., Sartorio, A., Tanara, V., Spadacini, G., Selleri, M., Zanconato, V., et al. (2024) · Journal of the American Geriatrics Society · doi:10.1111/jgs.19035 · Cited by 110
“21 The RoB 2 is structured into five bias domains, including bias due to the randomization process (D1), deviations from intended interventions (D2), missing outcome data (D3), outcome measurement (D4), and the selection of the reported result (D5).”
Wierstra, B. T., Kadri, S. S., Alomar, S. A., Burbano, X., Barrisford, G. W., & Kao, R. L. C. (2016) · Critical Care · doi:10.1186/s13054-016-1291-8 · Cited by 109
“” Studies were assigned a “ low-quality ” rating if there was no illness severity assessment between cohorts or at the time of randomization ( n = 8), significant differences ( p 0.05) between cohort groups ( n = 7) at baseline, incomplete basic demographic data at baseline ( n = 6) to exclude baseline differences, or…”
MethodInfluentialLundstrøm, L. H., Møller, A. M., Rosenstock, C., Astrup, G., Gätke, M. R., Wetterslev, J., & Database, T. D. A. (2009) · Anaesthesia · doi:10.1111/j.1365-2044.2009.06057.x · Cited by 107
Reeves, B. C., & Murphy, G. J. (2008) · Current Opinion in Cardiology · doi:10.1097/hco.0b013e328310fc95 · Cited by 109
Synthesis of observational studies should consider credibility ceilings
Salanti, G., & Ioannidis, J. P. A. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.05.014 · Cited by 108
“Despite the fact that findings from observational studies are often subject to bias [1], they are still the best available evidence in many research fields.”
BackgroundRowe, B. H., Villa‐Roel, C., Guo, X., Bullard, M. J., Ospina, M. B., Vandermeer, B. W., Innes, G. D., Schull, M. J., & Holroyd, B. R. (2011) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2011.01081.x · Cited by 132
Basic study design influences the results of orthodontic clinical investigations
Papageorgiou, S. N., Xavier, G. M., & Cobourne, M. T. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.008 · Cited by 108
“This difference is usually interpreted as an overestimation in non-RCTs and not an underestimation in RCTs [37].”
BackgroundHow Can We Improve Retention of the First-Time Donor? A Systematic Review of the Current Evidence
Bagot, K. L., Murray, A. L., & Masser, B. (2016) · Transfusion Medicine Reviews · doi:10.1016/j.tmrv.2016.02.002 · Cited by 127
“The Effective Public Health Practice Project [40–42] “Quality assessment tool for quantitative studies” was used to evaluate all studies.”
MethodGarattini, S., Jakobsen, J. C., Wetterslev, J., Bertelé, V., Banzi, R., Rath, A., Neugebauer, E. A. M., Laville, M., Masson, Y., Hivert, V., et al. (2016) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2016.03.020 · Cited by 108
Sørensen, A. K. B., Bak, K., Krarup, A. L., Thune, C. H., Nygaard, M., Jørgensen, U., Sloth, C., & Torp‐Pedersen, S. (2006) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2006.06.010 · Cited by 120
Lavender and sleep: A systematic review of the evidence
Fismer, K., & Pilkington, K. (2012) · European Journal of Integrative Medicine · doi:10.1016/j.eujim.2012.08.001 · Cited by 125
“[69] Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al.”
BackgroundMethodDavies, L. M., Lewis, S., Jones, P. B., Barnes, T. R. E., Gaughran, F., Hayhurst, K., Markwick, A., & Lloyd, H. (2007) · British Journal of Psychiatry · doi:10.1192/bjp.bp.106.028654 · Cited by 107
LAHNER, E., ANNIBALE, B., & DELLE FAVE, G. (2009) · Alimentary Pharmacology & Therapeutics · doi:10.1111/j.1365-2036.2009.03993.x · Cited by 107
Is Concurrent Training Efficacious Antihypertensive Therapy? A Meta-Analysis
CORSO, L. M. L., MACDONALD, H. V., JOHNSON, B. T., FARINATTI, P., LIVINGSTON, J., ZALESKI, A. L., BLANCHARD, A., & PESCATELLO, L. S. (2016) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000001056 · Cited by 107
Interventions for people bereaved through suicide: systematic review
McDaid, C., Trowman, R., Golder, S., Hawton, K., & Sowden, A. (2008) · British Journal of Psychiatry · doi:10.1192/bjp.bp.107.040824 · Cited by 107
Self-citationSun, W. H., Miu, H. Y. H., Wong, C. K. H., Tucker, J. D., & Wong, W. C. W. (2016) · The Journal of Sex Research · doi:10.1080/00224499.2016.1247779 · Cited by 123
“This tool was judged suitable for use in systematic reviews of effectiveness (Deeks et al., 2003), and recommended by Armijo-Olivo, Stiles, Hagen, Biondo, and Cummings (2012) when compared with the Cochrane Collaboration Risk of Bias Tool, with its content and construct validity well established (Thomas, Ciliska, Dobb…”
MethodSandeep, B., Huang, X., Duan, Y., & Xiao, Z. (2022) · International Journal of Surgery · doi:10.1016/j.ijsu.2022.106854 · Cited by 124
Detection of rotator cuff tears: the value of MRI following ultrasound
Rutten, M. J. C. M., Spaargaren, G.-J., van Loon, T., Malefijt, M. C. D. W., Kiemeney, L. A. L. M., & Jager, G. J. (2009) · European Radiology · doi:10.1007/s00330-009-1561-9 · Cited by 121
Bambra, C., Whitehead, M., Sowden, A., Akers, J., & Petticrew, M. (2008) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2007.067249 · Cited by 97
“3).[30-32] Two reviewers (CB and MP) independently appraised the included studies according to these criteria.”
BackgroundMethodSelf-citationIs there really a link between benzodiazepine use and the risk of dementia?
Billioti de Gage, S., Pariente, A., & Bégaud, B. (2015) · Expert Opinion on Drug Safety · doi:10.1517/14740338.2015.1014796 · Cited by 108
“2 Evaluation of the quality of the study We used the Newcastle-Ottawa scale, specifically developed for non-randomized studies [17] and considered as one of the two most useful tools for systematic reviews and meta-analysis [18].”
MethodSchneider, M. P., Gross, T., Bachmann, L. M., Blok, B. F. M., Castro-Diaz, D. M., POPOLO, G. D., Groen, J., Hamid, R., Karsenty, G., Pannek, J., et al. (2015) · European Urology · doi:10.1016/j.eururo.2015.07.001 · Cited by 106
“This is a pragmatic approach informed by methodological literature pertaining to assessing risks of bias in nonrandomised studies [8,13].”
MethodBrinjikji, W., Demchuk, A. M., Murad, M. H., Rabinstein, A. A., McDonald, R. J., McDonald, J. S., & Kallmes, D. F. (2017) · Stroke · doi:10.1161/strokeaha.117.016771 · Cited by 119
A systematic review of interventions to foster physician resilience
Fox, S., Lydon, S., Byrne, D., Madden, C., Connolly, F., & O’Connor, P. (2018) · Postgraduate Medical Journal · doi:10.1136/postgradmedj-2017-135212 · Cited by 105
A systematic review of presumed consent systems for deceased organ donation
Rithalia, A. D., McDaid, C., Suekarran, S., Norman, G., Myers, L., & Sowden, A. J. (2009) · Health Technology Assessment · doi:10.3310/hta13260 · Cited by 128
Self-citationQuality assessment tools for evidence from environmental science
Bilotta, G. S., Milner, A. M., & Boyd, I. L. (2014) · Environmental Evidence · doi:10.1186/2047-2382-3-14 · Cited by 107
“It is therefore very important to consider carefully, the choice of quality assessment tool to be used in SRs [19].”
BackgroundMethodInfluentialBrown, M. J., Sinclair, M., Liddle, D., Hill, A. J., Madden, E., & Stockdale, J. (2012) · PLoS ONE · doi:10.1371/journal.pone.0039503 · Cited by 104
“This tool includes criteria for assessing selection bias, allocation bias, confounders, blinding, data collection methods and withdrawals/dropouts, and has been used in public health interventions and systematic reviews of effectiveness [34], [40].”
MethodKemp, J. L., Collins, N. J., Makdissi, M., Schache, A. G., Machotka, Z., & Crossley, K. M. (2011) · British Journal of Sports Medicine · doi:10.1136/bjsports-2011-090428 · Cited by 113
Professionals underestimate patients' pain: a comprehensive review
Seers, T., Derry, S., Seers, K., & Moore, A. (2018) · Pain · doi:10.1097/j.pain.0000000000001165 · Cited by 127
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.