Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
(1963) · Medical Entomology and Zoology · Cited by 15,085
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
Items 501–600 of 1,024
Lien-Iversen, T., Morgan, D. B., Jensen, C., Risberg, M. A., Engebretsen, L., & Viberg, B. L. (2019) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-100765 · Cited by 111
Case-mix adjustment in non-randomised observational evaluations: the constant risk fallacy
Nicholl, J. P. (2007) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2007.061747 · Cited by 122
Gowers, S., Clark, A., Roberts, C., Byford, S., Barrett, B., Griffiths, A., Edwards, V, Bryan, C., Smethurst, N., Rowlands, L., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14150 · Cited by 107
Priorities Among Effective Clinical Preventive Services
Maciosek, M. V., Edwards, N. M., Coffield, A. B., Flottemesch, T. J., Nelson, W. W., Goodman, M. J., & Solberg, L. I. (2006) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2006.03.011 · Cited by 111
Ruby, A., Knight, A., Perel, P., Blanchet, K., & Roberts, B. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138303 · Cited by 102
Neurofeedback as a form of cognitive rehabilitation therapy following stroke: A systematic review
Renton, T., Tibbles, A., & Topolovec‐Vranic, J. (2017) · PLoS ONE · doi:10.1371/journal.pone.0177290 · Cited by 117
Performance of screening tests for child physical abuse in accident and emergency departments
Woodman, J., Pitt, M., Wentz, R., Taylor, B., Hodes, D., & Gilbert, R. E. (2008) · Health Technology Assessment · doi:10.3310/hta12330 · Cited by 114
Ibitoye, M. O., Hamzaid, N. A., Hasnan, N., Wahab, A. K. A., & Davis, G. M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0149024 · Cited by 106
“The quality of the studies was measured using the Downs and Black (D&B) Scale [28] relevant in assessing both randomised and non-randomised studies and adjudged a valid scale for methodological quality assessments [29].”
MethodLi, S., Wang, Z., Liu, Y., Cao, J., Zheng, H., Jing, Y., Han, L., Ma, X., Xia, R., & Yu, L. (2020) · Ear Nose & Throat Journal · doi:10.1177/0145561320943362 · Cited by 103
Frampton, G. K., Whaley, P., Bennett, M. G., Bilotta, G. S., Dorne, J. L. C. M., Eales, J. F., James, K. L., Kohl, C., Land, M., Livoreil, B., et al. (2022) · Environmental Evidence · doi:10.1186/s13750-022-00264-0 · Cited by 122
“Risk of bias tools Many tools exist for critically appraising studies, covering a range of experimental and observational study designs [32, 34, 35, 56, 57].”
Nonfluoride caries-preventive agents
Rethman, M. P., Beltrán‐Aguilar, E. D., Billings, R. J., Burne, R. A., Clark, M. B., Donly, K. J., Hujoel, P. P., Katz, B. P., Milgrom, P., Sohn, W., et al. (2011) · The Journal of the American Dental Association · doi:10.14219/jada.archive.2011.0329 · Cited by 128
Huque, H., Eramudugolla, R., Chidiac, B., Ee, N., Ehrenfeld, L., Matthews, F. E., Peters, R., & Anstey, K. J. (2023) · Journal of Alzheimer's Disease · doi:10.3233/jad-220724 · Cited by 102
Bailey, D. B., Berry-Kravis, E., Wheeler, A., Raspa, M., Merrien, F., Ricart, J., Koumaras, B., Rosenkranz, G., Tomlinson, M., von Raison, F., et al. (2016) · Journal of Neurodevelopmental Disorders · doi:10.1186/s11689-015-9134-5 · Cited by 101
Quasi-experimental study designs series—paper 6: risk of bias assessment
Waddington, H., Aloe, A. M., Becker, B. J., Djimeu, E. W., Hombrados, J. G., Tugwell, P., Wells, G., & Reeves, B. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.02.015 · Cited by 97
“[48] Deeks J, Dinnes R, D’Amico R, Sowden AJ, Sakarovitch C, Song F,”
MethodO'Donnell, C. M., McLoughlin, L., Patterson, C. C., Clarke, M. J., McCourt, K. C., McBrien, M. E., McAuley, D. F., & Shields, M. O. (2017) · British Journal of Anaesthesia · doi:10.1016/j.bja.2017.09.002 · Cited by 113
Singh, S., Zaki, R. A., & Farid, N. D. N. (2019) · Journal of Adolescence · doi:10.1016/j.adolescence.2019.06.004 · Cited by 103
Li, W., Zhou, F., Zhang, L., Ng, C. H., Ungvari, G. S., Li, J., & Xiang, Y. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.04.051 · Cited by 118
Effectiveness of Fertility Awareness–Based Methods for Pregnancy Prevention: A Systematic Review
Peragallo Urrutia, R., Polis, C. B., Jensen, E. T., Greene, M. E., Kennedy, E., & Stanford, J. B. (2018) · Obstetrics & Gynecology · doi:10.1097/aog.0000000000002784 · Cited by 101
Bear, H. A., Edbrooke-Childs, J., Norton, S., Krause, K. R., & Wolpert, M. (2020) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2019.12.002 · Cited by 101
Li, R., Li, B., Kreher, D. A., Benjamin, A. R., Gubbels, A. L., & Smith, S. M. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2020.03.002 · Cited by 107
Rowell, N. P., & Williams, C. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002935.pub2 · Cited by 101
Challenges in Using Nonrandomized Studies in Systematic Reviews of Treatment Interventions
Norris, S. L., & Atkins, D. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-142-12_part_2-200506211-00011 · Cited by 102
“However, the debate continues on the role of nonrandomized studies in formulating recommendations about treatments (3, 6).”
BackgroundInfluentialMcAlindon, T. E., Kissin, E. Y., Nazarian, L. N., Ranganath, V. K., Prakash, S., Taylor, M. B., Bannuru, R. R., Srinivasan, S., Gogia, M., McMahon, M. A., et al. (2012) · Arthritis Care & Research · doi:10.1002/acr.21836 · Cited by 101
Yang, Z., Liu, H., Xie, X., Tan, Z., Qin, T., & Kang, P. (2014) · The Bone & Joint Journal · doi:10.1302/0301-620x.96b12.34378 · Cited by 101
Union rate after operative treatment of humeral shaft nonunion – A systematic review
Peters, R. M., Claessen, F. M. A. P., Doornberg, J. N., Kolovich, G. P., Diercks, R. L., & van den Bekerom, M. P. J. (2015) · Injury · doi:10.1016/j.injury.2015.09.041 · Cited by 110
“To assess the methodological quality we used modified questions from existing quality assessment tools [8,9,42].”
MethodVenUS II: a randomised controlled trial of larval therapy in the management of leg ulcers
Dumville, J. C., Worthy, G., Soares, M. O., Bland, J., Cullum, N. A., Dowson, C. G., Iglesias, C., McCaughan, D., Mitchell, J., Nelson, E. A., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13550 · Cited by 116
Smith, T. O., Daniell, H., Geere, J., Toms, A. P., & Hing, C. B. (2012) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2011.12.008 · Cited by 100
Nj, A., Abudou, M., Candeias, & Wu, T. (2014) · Cited by 100
Foerster, B., Pozo, C. D., Abufaraj, M. M., Mari, A., Kimura, S., D’Andrea, D., John, H. A., & Shariat, S. F. (2018) · JAMA Oncology · doi:10.1001/jamaoncol.2018.1071 · Cited by 100
Small CGG repeat expansion alleles of FMR1 gene are associated with parkinsonism
Loesch, D. Z., Khaniani, M. S., Slater, H., Rubio, J. P., Bui, M., Kotschet, K. E., D'Souza, W. J., Venn, A., Kalitsis, P., Choo, A., et al. (2009) · Clinical Genetics · doi:10.1111/j.1399-0004.2009.01275.x · Cited by 100
The introduction of QF-PCR in prenatal diagnosis of fetal aneuploidies: time for reconsideration
Nicolini, U., Lalatta, F., Natacci, F., Curcio, C., & Bui, T. (2004) · Human Reproduction Update · doi:10.1093/humupd/dmh046 · Cited by 100
Diabetes and Cancer—An AACE/ACE Consensus Statement
Handelsman, Y., LeRoith, D., Bloomgarden, Z. T., Dagogo-Jack, S. E., EINHORN, D. F., Garber, A. J., Grunberger, G., Harrell, R. M., Gagel, R. F., Lebovitz, H. E., et al. (2013) · Endocrine Practice · doi:10.4158/ep13248.cs · Cited by 105
Maurer, L. F., Espie, C. A., & Kyle, S. D. (2018) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2018.07.005 · Cited by 97
“It is considered one of the ‘best’ tools for evaluating randomised and non-randomised intervention studies [22].”
Ryan, É. J., O'Sullivan, D. P., Kelly, M. E., Syed, A. Z., Neary, P. C., O’Connell, P. R., Kavanagh, D. O., Winter, D. C., & O’Riordan, J. M. (2019) · British journal of surgery · doi:10.1002/bjs.11220 · Cited by 102
Breslin, G., Shannon, S., Cummings, M., & Leavey, G. (2022) · Systematic Reviews · doi:10.1186/s13643-022-01932-5 · Cited by 117
“For non-randomised studies, we used the ‘Quality Assessment Tool for Quantitative Studies’ (QATSQ) [37] that is recommended for use in systematic reviews [38].”
MethodKuß, O., Legler, T., & Börgermann, J. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.005 · Cited by 107
Lucid dreaming incidence: A quality effects meta-analysis of 50 years of research
Saunders, D. T., Roe, C. A., Smith, G., & Clegg, H. (2016) · Consciousness and Cognition · doi:10.1016/j.concog.2016.06.002 · Cited by 124
Brea-Gómez, B., Torres-Sánchez, I., Ortiz-Rubio, A., Calvache-Mateo, A., Cabrera-Martos, I., López-López, L., & Valenza, M. C. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182211806 · Cited by 98
Vogt, K. S., & Norman, P. (2019) · Psychology and Psychotherapy: Theory, Research and Practice · doi:10.1111/papt.12194 · Cited by 98
“The measure has been rated as one of the top six quality assessment tools suitable for systematic reviews (Deeks et al., 2003).”
A Systematic Review of Transcendent States Across Meditation and Contemplative Traditions
Wahbeh, H., Sagher, A., Back, W., Pundhir, P., & Travis, F. T. (2017) · EXPLORE · doi:10.1016/j.explore.2017.07.007 · Cited by 120
“18,48 It grades study quality and risk of bias on important constructs, including: blinding, randomization, adequate reporting, attrition, sample size, determination, and control group usage.”
Interventions in the alcohol server setting for preventing injuries
Ker, K., & Chinnock, P. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005244.pub3 · Cited by 100
New Insights on Vaginal Birth After Cesarean: Can It Be Predicted?
Eden, K., McDonagh, M., Denman, M., Marshall, N., Emeis, C. L., Fu, R., Janik, R., Walker, M., & Guise, J. (2010) · Obstetrics and Gynecology · doi:10.1097/aog.0b013e3181f2de49 · Cited by 112
Jackson, N., Denny, S., & Ameratunga, S. N. (2014) · Social Science & Medicine · doi:10.1016/j.socscimed.2014.06.004 · Cited by 118
Wang, H., Fox, C. S., Troy, L. M., Mckeown, N. M., & Jacques, P. F. (2015) · British Journal of Nutrition · doi:10.1017/s0007114515003578 · Cited by 93
Hague, B., Hall, J., & Kellett, S. (2016) · Journal of Behavioral Addictions · doi:10.1556/2006.5.2016.064 · Cited by 103
“First, the Downs and Black (1998) checklist provides a standardized score (0–32) from a list of 27 criteria and is a valid and reliable tool to assess randomized and non-randomized studies (Brouwers et al., 2005; Deeks et al., 2003).”
MethodChronic tobacco smoking and neuropsychological impairments: A systematic review and meta-analysis
Conti, A. A., McLean, L., Tolomeo, S., Steele, J. D., & Baldacchino, A. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.11.017 · Cited by 117
“…inclusion of Non-Randomized Studies (NRS) such as case-control and cross-sectional studies in the review, as the non-random allocation of participants to groups could imply a large and unpredictable bias leading to over-estimations or under-estimations of treatment effects (Deeks et al., 2003).”
Safety and Outcomes of Intravenous Thrombolysis in Posterior Versus Anterior Circulation Stroke
Keselman, B., Gdovinová, Z., Jatužis, D., Melo, T. P. E., Vilionskis, A., Cavallo, R., Frol, S., Jurák, L., Koyuncu, B. A., Nunes, A. P., et al. (2020) · Stroke · doi:10.1161/strokeaha.119.027071 · Cited by 106
EFFICACY OF PROPHYLACTIC ANTIBIOTICS AGAINST MENINGITIS AFTER CRANIOTOMY: A META-ANALYSIS
Barker, F. G. (2007) · Neurosurgery · doi:10.1227/01.neu.0000255425.31797.23 · Cited by 112
Sexual functioning of cervical cancer survivors: A review with a female perspective
Lammerink, E. A. G., de Bock, G. H., Pras, E., Reyners, A. K. L., & Mourits, M. J. E. (2012) · Maturitas · doi:10.1016/j.maturitas.2012.05.006 · Cited by 96
Cheang, R., Gillions, A., & Sparkes, E. (2019) · Journal of Child and Family Studies · doi:10.1007/s10826-019-01413-9 · Cited by 95
“This tool has been judged suitable to use in SRs, providing an overall quality rating for six components: selection bias, study design, confounders, blinding, withdrawals and dropouts, data collection methods and blinding (Deeks et al. 2003).”
MethodLyon, E., Laver, T. A., Yu, P., Jama, M., Young, K., Zoccoli, M. A., & Marlowe, N. (2010) · Journal of Molecular Diagnostics · doi:10.2353/jmoldx.2010.090229 · Cited by 102
Veskimäe, E., Neuzillet, Y., Rouanne, M., MacLennan, S., Lam, T. B. L., Yuan, Y., Compérat, É., Cowan, N. C., Gakis, G., van der Heijden, A. G., et al. (2017) · British Journal of Urology · doi:10.1111/bju.13819 · Cited by 95
Exploring the use of content analysis methodology in consumer research
Vespestad, M. K., & Clancy, A. (2020) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2020.102427 · Cited by 123
Robertson, C., Close, A. J., Fraser, C. M., Gurung, T., Jia, X., Sharma, P., Vale, L. D., Ramsay, C. R., & Pickard, R. S. (2013) · British Journal of Urology · doi:10.1111/bju.12247 · Cited by 108
“84) N = 2322 Ho et al. (2011) [31] Random-effects RR 0.”
Study Quality Assessment in Systematic Reviews of Research on Intervention Effects
Wells, K., & Littell, J. H. (2009) · Research on Social Work Practice · doi:10.1177/1049731508317278 · Cited by 95
“The third review was completed by Deeks et al. (2003).”
InfluentialSchwendicke, F., Graetz, C., Stolpe, M., & Dörfer, C. E. (2014) · Journal of Clinical Periodontology · doi:10.1111/jcpe.12315 · Cited by 94
McGinty, J., Haque, S. M., & Upthegrove, R. (2017) · Schizophrenia Research · doi:10.1016/j.schres.2017.09.040 · Cited by 118
“The Downs and Black instrument (Downs and Black, 1998) was used to assess standardised risk of bias (Deeks et al., 2003).”
MethodResidential alternatives to acute psychiatric hospital admission: systematic review
Lloyd-Evans, B., Slade, M., Jagielska, D., & Johnson, S. (2009) · British Journal of Psychiatry · doi:10.1192/bjp.bp.108.058347 · Cited by 94
Mowatt, G., Houston, J. G., Hernández, R. A., de Verteuil, R., Fraser, C. M., Cuthbertson, B. H., & Vale, L. D. (2009) · Health Technology Assessment · doi:10.3310/hta13070 · Cited by 100
Systematic review: early infant feeding and the prevention of coeliac disease
Szajewska, H., Chmielewska, A., Pieścik‐Lech, M., Ivarsson, A., Kolaček, S., Koletzko, S., Mearin, M. L., Shamir, R., Auricchio, R., Troncone, R., et al. (2012) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.12023 · Cited by 97
“There is no tool for assessing the quality of nonrandomised trials that would be widely recognised as most effective.(13) Again, we chose to use”
Preterm birth and subsequent insulin sensitivity: a systematic review
Tinnion, R., Gillone, J., Cheetham, T., & Embleton, N. (2014) · Archives of Disease in Childhood · doi:10.1136/archdischild-2013-304615 · Cited by 96
Koning, G. G., Wetterslev, J., van Laarhoven, C. J. H. M., & Keus, F. (2013) · PLoS ONE · doi:10.1371/journal.pone.0052599 · Cited by 112
Urinary Incontinence as a Predictor of Death: A Systematic Review and Meta-Analysis
John, G., Bardini, C., Combescure, C., & Dällenbach, P. (2016) · PLoS ONE · doi:10.1371/journal.pone.0158992 · Cited by 104
Boatin, A., Cullinane, F., Torloni, M., & Betrán, A. (2018) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.14774 · Cited by 90
Bakkalci, D., Jia, Y., Winter, J. R., Lewis, J. E. A., Taylor, G. S., & Stagg, H. R. (2020) · Journal of Global Health · doi:10.7189/jogh.10.010405 · Cited by 94
Hurley, E. T., Toale, J. P., Davey, M. S., Colasanti, C. A., Pauzenberger, L., Strauss, E. J., & Mullett, H. (2020) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2020.06.021 · Cited by 94
“MQOE was assessed using the Newcastle-Ottawa scale, which grade studies as unsatisfactory (0-3), satisfactory (4), good (5-6), or very good (7-9).(11) The outcome measures focused on were modeled on a previous model for a meta-analysis evaluating the treatment of anterior shoulder instability(23): (1) Rowe score; (2)…”
MethodKyzas, P. A., Denaxa-Kyza, D., & Ioannidis, J. P. A. (2007) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djk032 · Cited by 94
Overgaard, C., Møller, A. M., Fenger‐Grøn, M., Knudsen, L. B., & Sandall, J. (2011) · BMJ Open · doi:10.1136/bmjopen-2011-000262 · Cited by 119
“As we were concerned that residual confounding might remain, a supplementary regression analysis adjusting for the matching characteristics was performed using both continuous and grouped variables.(47) 48 For ease of interpretation (eg, calculation of confidence bands), ordinal outcomes were dichotomised, but we cont…”
MethodTan, J., Liu, S., Segal, J. B., Alexander, G. C., & McAdams‐DeMarco, M. (2016) · BMC Nephrology · doi:10.1186/s12882-016-0368-6 · Cited by 101
“We performed our qualitative assessment using the recently developed Cochrane Risk of Bias Assessment Tool for Non-Randomized Studies of Interventions, which was designed specifically for non-randomized studies that compare the health effects of two or more intervention [11], and unlike the Newcastle-Ottawa scale, doe…”
MethodThe Bidirectional Relationship between Vision and Cognition
Vu, T. A., Fenwick, E., Gan, A. T. L., Man, R. E. K., Tan, B. K. J., Gupta, P., Ho, K. C., Ortiz, C. A. R., Trompet, S., Gussekloo, J., et al. (2020) · Ophthalmology · doi:10.1016/j.ophtha.2020.12.010 · Cited by 100
Prediction and perinatal outcomes of fetal growth restriction
Breeze, A., & Lees, C. C. (2007) · Seminars in Fetal and Neonatal Medicine · doi:10.1016/j.siny.2007.07.002 · Cited by 96
A systematic review of population screening for fragile X syndrome
Hill, M. K., Archibald, A. D., Cohen, J., & Metcalfe, S. A. (2010) · Genetics in Medicine · doi:10.1097/gim.0b013e3181e38fb6 · Cited by 108
van den Bergh, R. C. N., van Casteren, N. J., den Broeck, T. V., Fordyce, E. R., Gietzmann, W., Stewart, F. J., MacLennan, S., Dabestani, S., Joaquim Bellmunt, B., Bolla, M., et al. (2015) · European Urology · doi:10.1016/j.eururo.2015.11.023 · Cited by 103
“o rc h ie ct o m y 1 9 8 6 6 [4 9 – 8 0 ], 6 5 .”
Porter, N. S., Jason, L. A., Boulton, A., Bothne, N., & Coleman, B. (2010) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2008.0376 · Cited by 91
Injection treatments for patellar tendinopathy
van Ark, M., Zwerver, J., & van den Akker‐Scheek, I. (2011) · British Journal of Sports Medicine · doi:10.1136/bjsm.2010.078824 · Cited by 95
“3 (4-36) mo Recreational athletes 4-week ‘wash-out’ period (no treatment).”
BackgroundInfluentialEndovascular Treatment of Vein of Galen Malformations: A Systematic Review and Meta-Analysis
Brinjikji, W., Krings, T., Murad, M. H., Rouchaud, A., & Meila, D. (2017) · American Journal of Neuroradiology · doi:10.3174/ajnr.a5403 · Cited by 92
Rathke’s cleft cyst recurrence after transsphenoidal surgery: A meta-analysis of 1151 cases
Mendelson, Z., Husain, Q. M., Elmoursi, S., Svider, P. F., Eloy, J. A., & Liu, J. K. (2013) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2013.07.008 · Cited by 92
“Using the Quality Assessment Tool for Quantitative Studies (Effective Public Health Project 2007), it was determined that all of the studies included in this review were given the global rating of ‘‘weak’’ [13–15].”
Clustered burst firing in FMR1 premutation hippocampal neurons: amelioration with allopregnanolone
Cao, Z., Hulsizer, S., Tassone, F., Tang, H.-T., Hagerman, R. J., Rogawski, M. A., Hagerman, P. J., & Pessah, I. N. (2012) · Human Molecular Genetics · doi:10.1093/hmg/dds118 · Cited by 93
Sexual dysfunction following rectal cancer surgery
Celentano, V., Cohen, R., Warusavitarne, J. H., Faiz, O. D., & Chand, M. (2017) · International Journal of Colorectal Disease · doi:10.1007/s00384-017-2826-4 · Cited by 94
“The quality of the included studies was evaluated by the Newcastle-Ottawa Scale (NOS) [11]: on a scale of 9, a greater score was considered to be an indicator of better quality.”
MethodInfluentialGlod, M., Riby, D. M., Honey, E., & Rodgers, J. (2015) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-015-0047-8 · Cited by 91
“Subjective judgement is a part of the evaluation process (Deeks et al. 2003; SIGN 2008); to minimise the reviewer’s subjectivity, the following steps were undertaken.”
MethodPercutaneous Patent Ductus Arteriosus (PDA) Closure During Infancy: A Meta-analysis
Backes, C. H., Rivera, B. K., Bridge, J. A., Armstrong, A. K., Boe, B. A., Berman, D. P., Fick, T. A., Holzer, R. J., Hijazi, Z. M., Abadir, S., et al. (2017) · PEDIATRICS · doi:10.1542/peds.2016-2927 · Cited by 91
Carruthers, H., Astin, F., & Munro, W. (2017) · Intensive and Critical Care Nursing · doi:10.1016/j.iccn.2017.03.003 · Cited by 91
“The latter tool uses a ‘mixed criteria’ approach with specific factual questions about the study design and general judgements on the degree of bias and was identified by Deeks et al (2003) as one of the ‘best’ tools for the quality appraisal of nonrandomised studies.”
MethodEffectiveness of Mental Health Courts in Reducing Recidivism: A Meta-Analysis
Lowder, E. M., Rade, C. B., & Desmarais, S. L. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201700107 · Cited by 91
“Because of the high risk of bias and a shortage of instruments of suitable quality for use in nonrandomized and retrospective investigations (31), we assessed study quality by using two measures: the SIGN Methodology Checklist 3 for Cohort Studies (32) and the Quality Assessment Tool (QAT) for Quantitative Studies (33…”
MethodThe Rate of Radial Head Prosthesis Removal or Revision: A Systematic Review and Meta-Analysis
Kachooei, A. R., Baradaran, A., Ebrahimzadeh, M. H., van Dijk, C. N., & Chen, N. C. (2017) · The Journal Of Hand Surgery · doi:10.1016/j.jhsa.2017.08.031 · Cited by 91
Lu, V. M., Alvi, M. A., McDonald, K. L., & Daniels, D. J. (2019) · Journal of Neurosurgery Pediatrics · doi:10.3171/2018.9.peds18419 · Cited by 100
Cooke, M. R., Marsh, J. L., Clark, M., Nakash, R., Jarvis, R., Hutton, J. L., Szczepura, A. K., Wilson, S., & Lamb, S. E. (2009) · Health Technology Assessment · doi:10.3310/hta13130 · Cited by 113
Berg, R. C., Odgaard‐Jensen, J., Fretheim, A., Underland, V., & Vist, G. E. (2014) · Obstetrics and Gynecology International · doi:10.1155/2014/542859 · Cited by 91
Bond, M., Pitt, M., Akoh, J. A., Moxham, T., Hoyle, M., & Anderson, R. (2009) · Health Technology Assessment · doi:10.3310/hta13380 · Cited by 94
Bellicha, A., Kieusseian, A., Fontvieille, A.-M., Tataranni, A., Charreire, H., & Oppert, J. (2014) · Preventive Medicine · doi:10.1016/j.ypmed.2014.11.001 · Cited by 116
“…with a standardized quality assessment tool, the Effective Public Health Practice Project Quality Assessment Tool for Quantitative Studies 2003 (Thomas et al., 2004, www.ephpp.ca/tools.htlm), which has been developed to evaluate both randomized and non-randomized studies (Deeks et al., 2003).”
Sharpe, R. E., Nazarian, L. N., Parker, L., Rao, V. M., & Levin, D. C. (2012) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2011.09.008 · Cited by 91
Shin, J. Y., Roh, S., Lee, N., & Yang, K. (2016) · Head & Neck · doi:10.1002/hed.24554 · Cited by 94
“Each parameter consists of subcategorized questions based on selection, comparability, and exposure or outcomes.(14,15) If the study methodology was considered to be of the highest quality, then the study was awarded a maximum of 9 stars.”
BackgroundAlldred, S. K., Takwoingi, Y., Guo, B., Pennant, M., Deeks, J. J., Neilson, J. P., & Alfirevic, Z. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012600 · Cited by 90
Self-citationVitamin K to prevent fractures in older women: systematic review and economic evaluation
Stevenson, M., Lloyd-Jones, M., & Papaioannou, D. (2009) · Health Technology Assessment · doi:10.3310/hta13450 · Cited by 90
Stewart, R. J., van Rooyen, C., Korth, M., Chereni, A., da Silva, N. R., & de Wet, T. (2012) · Cited by 90
Schenone, A. L., Cohen, A., Patarroyo, G. D., Harper, L. J., Wang, X., Shishehbor, M. H., Menon, V., & Duggal, A. (2016) · Resuscitation · doi:10.1016/j.resuscitation.2016.07.238 · Cited by 90
Non-specific Low Back Pain and Postural Control During Quiet Standing—A Systematic Review
Koch, C., & Hänsel, F. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00586 · Cited by 109
Gandhi, A., Shah, A., Jones, M. P., Koloski, N. A., Talley, N. J., Morrison, M., & Holtmann, G. (2021) · Gut Microbes · doi:10.1080/19490976.2021.1933313 · Cited by 95
“In addition, the quality of the case-control included studies were assessed using the Newcastle-Ottawa scale (NOS) which judges the selection of the study groups, the comparability of the groups and the ascertainment of the exposure of interest, to assign a maximum score of 9 stars.(24)”
MethodNa, B.-R., Kwak, W.-K., Seo, H.-Y., & Seon, J. K. (2021) · Orthopaedic Journal of Sports Medicine · doi:10.1177/23259671211023099 · Cited by 104
Meta-analysis of heterogeneous clinical trials: An empirical example
Doi, S. A. R., Barendregt, J. J., & Mozurkewich, E. (2010) · Contemporary Clinical Trials · doi:10.1016/j.cct.2010.12.006 · Cited by 97
“Although the importance of such quality assessment of experimental studies is well established, quality assessment of other study designs in systematic reviews is far less well developed [33].”
BackgroundHopewell, S., Omar, O., Hyde, C., Yu, L., Doree, C. J., & Murphy, M. F. (2013) · BMJ Open · doi:10.1136/bmjopen-2012-002154 · Cited by 96
“The impact of such ‘confounding’ can be reduced by adjustment in the statistical analysis, but the success of this is dependent on the technique used, complete identification of the factors which might influence outcome and their accurate measurement in the patients in the study.(2) As all the factors influencing outc…”
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.