Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
(1963) · Medical Entomology and Zoology · Cited by 15,085
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Self-citation(1979) · Medical Entomology and Zoology · Cited by 11,252
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
Items 901–1,000 of 1,024
Glechner, A., Wöckel, A., Gartlehner, G., Thaler, K. J., Strobelberger, M., Griebler, U., & Kreienberg, R. (2012) · European Journal of Cancer · doi:10.1016/j.ejca.2012.09.010 · Cited by 70
Johnson, B. T., Low, R. E., & MacDonald, H. V. (2015) · Psychology & Health · doi:10.1080/08870446.2014.953533 · Cited by 58
McKenna, C., McDaid, C., Suekarran, S., Hawkins, N., Claxton, K., Light, K., Chester, M., Cleland, J. G. F., Woolacott, N. F., & Sculpher, M. (2009) · Health Technology Assessment · doi:10.3310/hta13240 · Cited by 66
Meta-analysis for orthodontists: Part II – Is all that glitters gold?
Papageorgiou, S. N. (2014) · Journal of Orthodontics · doi:10.1179/1465313314y.0000000110 · Cited by 57
“For nonrandomized study designs on the other hand, there are plenty of tools to assess the studies’ internal validity (Deeks et al., 2003).”
BackgroundBang, C. S., Baik, G. H., Shin, I., Kim, J. B., Suk, K. T., Yoon, J. H., Kim, Y. S., & Kim, D. J. (2015) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2014.12.025 · Cited by 64
Influence of different drugs on HIV risk in people who inject: systematic review and meta‐analysis
Tavitian‐Exley, I., Vickerman, P. T., Bastos, F. I., & Boily, M. (2015) · Addiction · doi:10.1111/add.12846 · Cited by 58
Lee, N.-W., Kim, G.-H., Heo, I., Kim, K., Ha, I., Lee, J., Hwang, E.-H., & Shin, B. (2017) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2017/8218139 · Cited by 65
Teleaudiology Services for Rehabilitation With Hearing Aids in Adults: A Systematic Review
Tao, K. F. M., Brennan‐Jones, C. G., Capobianco, D. M., Jayakody, D. M. P., Friedland, P. L., Swanepoel, D. W., & Eikelboom, R. H. (2018) · Journal of Speech Language and Hearing Research · doi:10.1044/2018_jslhr-h-16-0397 · Cited by 65
Naz, M. S. G., Simbar, M., Fakari, F. R., & Ghasemi, V. (2018) · PubMed Central · doi:10.22034/apjcp.2018.19.8.2031 · Cited by 87
“00) and the maximum total score is three (Thomas et al., 2004; Deeks et al., 2003; Armijo-Olivo et al., 2012).”
BackgroundInfluentialMeta-analysis of the role of colonoscopy after an episode of left-sided acute diverticulitis
Rottier, S. J., van Dijk, S. T., van Geloven, A. A. W., Schreurs, W. H., Draaisma, W. A., van Enst, W. A., Puylaert, J. B. C. M., de Boer, M. G. J., Klarenbeek, B. R., Otte, J. A., et al. (2019) · British journal of surgery · doi:10.1002/bjs.11191 · Cited by 58
Limits of Evidence‐based Surgery
Slim, K. (2005) · World Journal of Surgery · doi:10.1007/s00268-005-7922-x · Cited by 64
“Few attempts [23–25] have been made to develop instruments allowing surgeons to evaluate published nonrandomized studies and thus quantify the strength of their results before applying their conclusions to ‘‘individual patients.”
n-3 Fatty acids, Mediterranean diet and cognitive function in normal aging: A systematic review
Masana, M. F., Koyanagi, A., Haro, J. M., & Tyrovolas, S. (2017) · Experimental Gerontology · doi:10.1016/j.exger.2017.02.008 · Cited by 65
“The EPHPP, which was identified as one of the most useful tools for assessing methodological quality of non-randomized studies (Deeks et al., 2003), consists of six components: (a) selection bias, (b)”
MethodRichardson, E., Lewis, J. S., Gibson, J., Morgan, C., Halaki, M., Ginn, K. A., & Yeowell, G. (2020) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2019-000683 · Cited by 68
The Role of Observational Investigations in Comparative Effectiveness Research
Marko, N. F., & Weil, R. J. (2010) · Value in Health · doi:10.1111/j.1524-4733.2010.00786.x · Cited by 61
van Raaij, T. M., Bakker, W. J., Reijman, M., & Verhaar, J. A. N. (2007) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-8-74 · Cited by 57
Assessing the Quality of Simulation-Based Research Articles: A Rating Rubric
Fey, M. K., Gloe, D. S., & Mariani, B. A. (2015) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2015.10.005 · Cited by 61
Pathways through opiate use and offending: A systematic review
Hayhurst, K. P., Pierce, M., Hickman, M., Seddon, T., Dunn, G., Keane, J. A., & Millar, T. (2016) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2016.08.015 · Cited by 71
“This scale is recommended for quality assessment in epidemiological systematic reviews (Deeks et al., 2003) and is regularly used in”
MethodRisk of major limb amputation in diabetic foot ulcer and accompanying disease: A meta-analysis
Shin, J. Y., Roh, S., Sharaf, B. A., & Lee, N. (2017) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2017.07.015 · Cited by 57
Bates, A. S., Ayers, J., Κωστακόπουλος, Ν., Lumsden, T., Schoots, I. G., Willemse, P.-P. M., Yuan, Y., van den Bergh, R. C. N., Grummet, J., van der Poel, H. G., et al. (2021) · European Urology Oncology · doi:10.1016/j.euo.2020.12.008 · Cited by 57
Clomiphene citrate for men with hypogonadism: a systematic review and meta‐analysis
Huijben, M., Lock, M. T. W. T., de Kemp, V. F., de Kort, L. M. O., & van Breda, J. (2022) · Andrology · doi:10.1111/andr.13146 · Cited by 65
Native American Youth and Culturally Sensitive Interventions: A Systematic Review
Jackson, K. F., & Hodge, D. R. (2010) · Research on Social Work Practice · doi:10.1177/1049731509347862 · Cited by 56
“A substantial number of scales have been used to assess the methodological rigor of studies (Deeks et al., 2003; Olivo et al., 2008).”
MethodShock wave therapy for rotator cuff disease with or without calcification
Surace, S. J., Deitch, J. M., Johnston, R. V., & Buchbinder, R. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008962.pub2 · Cited by 71
Roux, É. L., Mellerio, H., Guilmin‐Crépon, S., Gottot, S., Jacquin, P., Boulkedid, R., & Alberti, C. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012338 · Cited by 62
“Medical treatment adherence 6 (21) 2 (8) 4 (50)”
Interventions to Improve Hand Hygiene Compliance in the ICU: A Systematic Review
Lydon, S., Power, M., McSharry, J., Byrne, M., Madden, C., Squires, J. E., & O'Connor, P. E. (2017) · Critical Care Medicine · doi:10.1097/ccm.0000000000002691 · Cited by 58
Cooper, R. E., Laxhman, N., Crellin, N. E., Moncrieff, J., & Priebe, S. (2019) · Schizophrenia Research · doi:10.1016/j.schres.2019.05.020 · Cited by 66
“The Cochrane tool is widely used and the EPHPP has been judged to be suitable for use in systematic reviews of effectiveness (Deeks et al., 2003).”
MethodMeybodi, M. A., Cao, W., Luznik, L., Bashey, A., Zhang, X., Romee, R., Saber, W., Hamadani, M., Weisdorf, D. J., Chu, H., et al. (2019) · Blood Advances · doi:10.1182/bloodadvances.2019000614 · Cited by 59
Ejiri, M., Kawai, H., Ishii, K., Oka, K., & Obuchi, S. P. (2021) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2021.104357 · Cited by 64
Witberg, G., Lador, A., Yahav, D., & Kornowski, R. (2018) · Catheterization and Cardiovascular Interventions · doi:10.1002/ccd.27518 · Cited by 55
“The Newcastle-Ottawa tool was used to assess the quality of the PSM studies [16].”
MethodTomasik, B., Bieńkowski, M., Braun, M., Popat, S., & Dziadziuszko, R. (2021) · Lung Cancer · doi:10.1016/j.lungcan.2021.06.004 · Cited by 63
Warner, N., & Murphy, M. F. (2021) · Drug and Alcohol Review · doi:10.1111/dar.13362 · Cited by 71
Anis, F. S., Adiamah, A., Lobo, D. N., & Sanyal, S. (2021) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.15711 · Cited by 55
Bang, C. S., Baik, G. H., Shin, I., Kim, J. B., Suk, K. T., Yoon, J. H., Kim, Y. S., & Kim, D. J. (2015) · Journal of Korean Medical Science · doi:10.3346/jkms.2015.30.6.749 · Cited by 55
Griebler, U., Bruckmüller, M. U., Kien, C., Dieminger, B., Meidlinger, B., Seper, K., Hitthaller, A., Emprechtinger, R., Wolf, A., & Gartlehner, G. (2015) · Public Health Nutrition · doi:10.1017/s1368980015001354 · Cited by 63
“(18) .”
Ghaffari, S. R., Tahmasebpour, A., Jamal, A., Hantoushzadeh, S., Eslamian, L., Marsoosi, V., Fattahi, F., Rajaei, M., Niroomanesh, S., Borna, S., et al. (2011) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.10051 · Cited by 57
A Systematic Review of Crisis Interventions Used in the Emergency Department
Hamm, M. P., Osmond, M. H., Curran, J., Scott, S. D., Ali, S., Hartling, L., Gokiert, R., Cappelli, M., Hnatko, G., & Newton, A. S. (2010) · Pediatric Emergency Care · doi:10.1097/pec.0b013e3181fe9211 · Cited by 55
“Experimental and observational studies were eligible for review inclusion.(15) Studies that included children and youth (e18 years) or adults (918 years) presenting to the ED with a primary complaint related to mental health were included.”
BackgroundCigarette Smoking and Primary Open Angle Glaucoma: A Systematic Review
Edwards, R., Thornton, J., Ajit, R. R., Harrison, R. S., & Kelly, S. P. (2008) · Journal of Glaucoma · doi:10.1097/ijg.0b013e31815f530c · Cited by 60
“A previous review of quality assessment tools for nonrandomised studies found that none addressed all the key dimensions of quality.(11) We investigated a range of possible tools for observational epidemiologic studies but found none which addressed all the key areas of the design and conduct of studies,(11) nor inclu…”
BackgroundMunsterman, T., Takken, T., & Wittink, H. M. (2012) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-13-202 · Cited by 68
“Although a commonly accepted valid rating instrument concerning the quality assessment of observational studies does not yet exist, there appears to be consensus about important items [19-21].”
Azab, M. A. E., Bharadwaj, S., Jayaraj, M., Hong, A. S., Solaimani, P., Mubder, M. I., Yeom, H., Yoo, J. W., & Volk, M. L. (2019) · Saudi Journal of Gastroenterology · doi:10.4103/sjg.sjg_92_19 · Cited by 55
“Quality assessment We used the Newcastle–Ottawa Scale (NOS) to assess the risk of bias in the included studies.[40,41] Risk of bias in relation to selection, comparability, and assessment of the exposure/outcome was assessed according to nine items using a star allocation scheme.”
MethodPOTTER, J. F., Mistri, A. K., Brodie, F. G., Tchernova, J., Wilson, E., Jagger, C., James, M. A., Ford, G. A., & Robinson, T. G. (2009) · Health Technology Assessment · doi:10.3310/hta13090 · Cited by 58
Tyrer, P. J., Oliver, P., Romeo, R., Knapp, M. R. J., Huline‐Dickens, S., Bouras, N., Ahmed, M. Z., Cooray, S., Deb, S. S., Murphy, D., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13210 · Cited by 59
ICF: Clinical relevance for physiotherapy? A critical review
Allet, L., Bürge, E., & Monnin, D. (2008) · Advances in Physiotherapy · doi:10.1080/14038190802315941 · Cited by 54
Bunn, P. D. S., Inoue, A., & da Silva, E. B. (2018) · Physical Therapy in Sport · doi:10.1016/j.ptsp.2018.11.011 · Cited by 60
Lu, V. M., Goyal, A., Quiñones‐Hinojosa, A., & Chaichana, K. L. (2018) · Clinical Neurology and Neurosurgery · doi:10.1016/j.clineuro.2018.12.013 · Cited by 54
A systematic review of factors associated with side‐effect expectations from medical interventions
Smith, L., Webster, R., & Rubin, G. J. (2020) · Health Expectations · doi:10.1111/hex.13059 · Cited by 62
Khorsand, N., Kooistra, H., van Hest, R. M., Veeger, N. J. G. M., & Meijer, K. (2014) · Thrombosis Research · doi:10.1016/j.thromres.2014.11.019 · Cited by 54
Adherence of Subjects With Cystic Fibrosis to Their Home Program: A Systematic Review
O'Donohoe, R., & Fullen, B. M. (2014) · Respiratory Care · doi:10.4187/respcare.02990 · Cited by 64
Immunosuppressive T-cell antibody induction for heart transplant recipients
Penninga, L., Møller, C. H., Gustafsson, F., Gluud, C. N., & Steinbrüchel, D. A. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008842.pub2 · Cited by 61
Ortolani, P., Marzocchi, A., Marrozzini, C., Palmerini, T., Saia, F., Taglieri, N., Baldazzi, F., Silenzi, S., Reggiani, M. L. B., Guastaroba, P., et al. (2007) · European Heart Journal · doi:10.1093/eurheartj/ehm338 · Cited by 54
Smoking cessation care among patients with head and neck cancer: a systematic review
McCarter, K. L., Martínez, Ú., Britton, B., Baker, A., Bonevski, B., Carter, G. L., Beck, A. K., WRATTEN, C. R., Guillaumier, A., Halpin, S. A., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012296 · Cited by 57
“24 26 Two review authors (KM and UM) independently assessed study quality and discrepancies were resolved through discussion.”
Outcomes of surgical management of TMJ ankylosis: A systematic review and meta-analysis
Mittal, N., Goyal, M., Sardana, D., & Dua, J. (2019) · Journal of Cranio-Maxillofacial Surgery · doi:10.1016/j.jcms.2019.03.029 · Cited by 54
Han, C., & Sun, Y. (2020) · Diseases of the Esophagus · doi:10.1093/dote/doaa081 · Cited by 54
Samsson, K. S., Grimmer, K. A., Larsson, M. E. H., Morris, J. H., & Bernhardsson, S. (2020) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-020-03673-9 · Cited by 54
Cornelius, V. R., Perrio, M. J., Shakir, S. A. W., & Smith, L. A. (2009) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1844 · Cited by 54
“Quality assessment is important because poor methodological quality can lead to an alteration in the estimates of harm.(24,34) However, there is a lack of standardised guidelines for assessing the quality of studies with respect to adverse effects.”
BackgroundMethodRamos‐Corpas, D., Santiago, J. C. H., & Montoya, F. (2006) · Prenatal Diagnosis · doi:10.1002/pd.1345 · Cited by 54
Linezolid and Vancomycin for Methicillin-Resistant Staphylococcus aureus Nosocomial Pneumonia
Powers, J. H., Ross, D. B., Lin, D., & Soreth, J. M. (2004) · CHEST Journal · doi:10.1378/chest.126.1.314 · Cited by 57
Macdonald, E. M., Perrin, B. M., & Kingsley, M. I. (2018) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x17699990 · Cited by 53
Papageorgiou, S. N., Antonoglou, G. N., Tsiranidou, E., Jepsen, S. V., & Jäger, A. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.04.002 · Cited by 53
“[45] Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al.”
BackgroundMori, K., Kimura, S., Parizi, M. K., Enikeev, D. V., Glybochko, P. V., Seebacher, V., Fajković, H., Mostafaei, H., Lysenko, I., Janisch, F., et al. (2019) · Clinical Genitourinary Cancer · doi:10.1016/j.clgc.2019.07.009 · Cited by 63
van Hoof, S. J. M., Quanjel, T. C. C., Kroese, M., Spreeuwenberg, M. D., & Ruwaard, D. (2019) · PLoS ONE · doi:10.1371/journal.pone.0219957 · Cited by 53
Rassie, K. L., Giri, R., Joham, A. E., Teede, H. J., & Mousa, A. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms232415621 · Cited by 64
“Quality appraisal (risk of bias) was conducted on Covidence software, using the Monash Centre for Health Research and Implementation (MCHRI) Evidence Synthesis Program tool [12], based on the Newcastle-Ottawa Scale for non-randomised studies [13].”
MethodMoher, D. (2010) · Journal of Chinese Integrative Medicine · doi:10.3736/jcim20100801 · Cited by 57
Fragile X syndrome carrier screening in the prenatal genetic counseling setting
Cronister, A., DiMaio, M. S., Mahoney, M. J., Donnenfeld, A. E., & Hallam, S. (2005) · Genetics in Medicine · doi:10.1097/01.gim.0000159898.90221.d3 · Cited by 54
Updated Guidelines 2012 for Gastroesophageal Reflux Disease
Jung, H., Hong, S. J., Jo, Y., Jeon, S. W., Cho, Y. K., Lee, K. J., Lee, J. S., Park, H., Shin, E. S., Lee, S. H., et al. (2012) · Korean Journal of Gastroenterology · doi:10.4166/kjg.2012.60.4.195 · Cited by 55
Uhlig, A., Hosseini, A. S. A., Simon, J., Lotz, J., Trojan, L., Schmid, M., & Uhlig, J. (2018) · The Journal of Urology · doi:10.1016/j.juro.2017.11.150 · Cited by 56
Reducing Burden for Caregivers of Older Adults With Mild Cognitive Impairment: A Systematic Review
Domingues, N. S., Verreault, P., & Hudon, C. (2018) · American Journal of Alzheimer s Disease & Other Dementias® · doi:10.1177/1533317518788151 · Cited by 63
Roselló, A., Albuquerque, R., Roselló-Llabrés, X., Marí-Roig, A., Estrugo‐Devesa, A., & López‐López, J. (2020) · Medicina oral, patología oral y cirugía bucal · doi:10.4317/medoral.23632 · Cited by 53
“Although ROBINS has a has been highlighted, for its good validity (20), when assessing surgical records NOS is easy to apply (21), and has been widely used due to recommendations from the Cochrane Collaboration (22).”
MethodLorenzo‐Pouso, A. I., Castelo‐Baz, P., Zorrilla, S. R., Pérez‐Sayáns, M., & Vega, P. (2020) · Acta Odontologica Scandinavica · doi:10.1080/00016357.2020.1859132 · Cited by 59
“In the analysis, the studies with NOS scores (asterisks) of 1–3, 4–6 and 7–9 were defined as low, medium and high quality, respectively.”
MethodInfluentialIndah Winarni, T., Chonchaiya, W., Adams, E., Au, J., Mu, Y., Rivera, S. M., Nguyen, D. V., & Hagerman, R. J. (2012) · Autism Research and Treatment · doi:10.1155/2012/104317 · Cited by 52
Assessing ‘what works’ in international development: meta-analysis for sophisticated dummies
Duvendack, M., Hombrados, J. G., Palmer-Jones, R., & Waddington, H. (2012) · Journal of Development Effectiveness · doi:10.1080/19439342.2012.710642 · Cited by 52
Transportation and homelessness: a systematic review
Murphy, E. R. (2019) · Journal of Social Distress and the Homeless · doi:10.1080/10530789.2019.1582202 · Cited by 75
“…domains for assessing quantitative studies: (1) selection bias; (2) study design; (3) confounders; (4) blinding; (5) data collection methods, and; (6) withdrawals and dropouts (Armijo-Olivo, Stiles, Hagen, Biondo, & Cummings, 2012; Deeks et al., 2003; Thomas, Ciliska, Dobbins, & Micucci, 2004).”
BackgroundFresán, A., Dionisio-García, D. M., González‐Castro, T. B., Ramos-Méndez, M. Á., Castillo-Ávila, R. G., Tovilla‐Zárate, C. A., Juárez‐Rojop, I. E., López‐Narváez, M. L., Mendoza, A. D. G., & Nicolini, H. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.06.053 · Cited by 65
Frerichs, W., Kaltenbacher, E., van de Leur, J. P., & Dean, E. (2012) · Physiotherapy Theory and Practice · doi:10.3109/09593985.2011.654179 · Cited by 54
“The tool’s validity and reliability are reasonably high, except for the three items on external validity (Deeks et al, 2003).”
Jamal, M. H., Doi, S. A. R., Rousseau, M., Edwards, M., Rao, C., Barendregt, J. J., Snell, L., & Meterissian, S. (2012) · British Journal of Surgery · doi:10.1002/bjs.8657 · Cited by 54
Ultrasound of the rotator cuff with MRI and anatomic correlation
Rutten, M. J. C. M., Maresch, B. J., Jager, G. J., Blickman, J. G., & van Holsbeeck, M. (2006) · European Journal of Radiology · doi:10.1016/j.ejrad.2006.12.003 · Cited by 55
Nochaiwong, S., Ruengorn, C., Awiphan, R., Dandecha, P., Noppakun, K., & Phrommintikul, A. (2016) · Open Heart · doi:10.1136/openhrt-2016-000441 · Cited by 55
“across studies, statistical analyses for adjusted and unadjusted risks of outcomes were performed.(45) With survival data, log HR and its variance were calculated.”
Adamczyk, W. M., Luedtke, K., & Saulicz, E. (2017) · Clinical Journal of Pain · doi:10.1097/ajp.0000000000000499 · Cited by 60
Stevens, A. W. M. M., Goossens, P. J. J., Knoppert-van der Klein, E. A. M., Draisma, S., Honig, A., & Kupka, R. W. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.02.018 · Cited by 52
A systematic review: Tools for assessing methodological quality of human observational studies
Wang, Z., Taylor, K., Allman-Farinelli, M., Armstrong, B., Askie, L., Ghersi, D., McKenzie, J., Norris, S. L., Page, M., Rooney, A. A., et al. (2019) · doi:10.31222/osf.io/pnqmy · Cited by 52
Cuenca‐Martínez, F., Herranz‐Gómez, A., Madroñero‐Miguel, B., Reina‐Varona, Á., Touche, R. L., Angulo, S., Pardo-Montero, J., del Corral, T., & López‐de‐Uralde‐Villanueva, I. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9092806 · Cited by 54
“The NOS scores 3 criteria with a range of 0 to 4 stars: grade selection of participants, assessment of exposures, outcomes, and comparability, and control of confounding variables, based on 9 questions.”
MethodInfluentialWang, B., Mei, Z., Ge, X., Li, Y., Zhou, Q., Meng, X., & An, G. (2023) · BMC Cardiovascular Disorders · doi:10.1186/s12872-023-03397-3 · Cited by 54
“The risk of bias of RCT was assessed using the Cochrane Collaboration’s risk of bias tool [9], and the quality of PSM studies was evaluated using the Newcastle–Ottawa Scale [10].”
MethodInfluentialHemingway, H., Henriksson, M., Chen, R., Damant, J., Fitzpatrick, N. K., Abrams, K. R., Hingorani, A. D., Janzon, M., Shipley, M. J., Feder, G. S., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14090 · Cited by 56
Hysterectomy versus hysterectomy plus oophorectomy for premenopausal women
Orozco, L. J., Tristan, M. G., Vreugdenhil, M. M. T., & Salazar, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005638.pub3 · Cited by 53
Observational evidence and strength of evidence domains: case examples
O’Neil, M., Berkman, N., Hartling, L., Chang, S., Anderson, J., Motu’apuaka, M., Guise, J.-M., & McDonagh, M. S. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-35 · Cited by 48
Rheumatoid Arthritis Disadvantages Younger Patients for Cardiovascular Diseases: A Meta-Analysis
Fransen, J., Kazemi‐Bajestani, S. M. R., Bredie, S. J. H., & Popa, C. D. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157360 · Cited by 53
“) independently using the Newcastle-Ottawa Scale; cohort studies can receive a maximum of 4 points (‘stars’) for Selection, 2 points for Comparability, and 3 points for Outcome [12].”
MethodThompson, J., Yoward, S., & Dawson, P. (2016) · Musculoskeletal Care · doi:10.1002/msc.1152 · Cited by 61
“The tool which originated from the EPHPP at McMaster University in Canada was chosen as it had been recommended for use in systematic reviews in which non-randomized trials are to be assessed (Deeks et al., 2003).”
MethodNHG-Standaard Schouderklachten
Winters, J., van der Windt, D., Spinnewijn, W., de Jongh, A. C., van der Heijden, G. J. M. G., Buis, P., Boeke, A. J. P., Feleus, A., & Geraets, J. J. X. R. (2009) · Bohn Stafleu van Loghum eBooks · doi:10.1007/978-90-313-6614-9_77 · Cited by 51
Kim, H. K., Mirjalili, S. A., & Fernandez, J. (2017) · Human Movement Science · doi:10.1016/j.humov.2017.09.012 · Cited by 51
Quality and impact of nurse-initiated analgesia in the emergency department: A systematic review
Varndell, W., Fry, M., & Elliott, D. (2018) · International Emergency Nursing · doi:10.1016/j.ienj.2018.05.003 · Cited by 56
Correlates of physical activity and sedentary behaviour in the Thai population: a systematic review
Liangruenrom, N., Craike, M. J., Biddle, S., Suttikasem, K., & Pedišić, Ž. (2019) · BMC Public Health · doi:10.1186/s12889-019-6708-2 · Cited by 61
Attention-Deficit/Hyperactivity Disorder and Outcome After Concussion: A Systematic Review
Cook, N. E., Iaccarino, M. A., Karr, J. E., & Iverson, G. L. (2020) · Journal of Developmental & Behavioral Pediatrics · doi:10.1097/dbp.0000000000000808 · Cited by 55
“A comprehensive examination of quality assessment measures identified the scale as suitable for systematic reviews.(32) For the current review, when rating the Newcastle-Ottawa Scale, we considered ADHD as the “exposure” and prolonged concussion recovery as the “outcome.”
MethodEnd-of-life care pathways for improving outcomes in caring for the dying
Chan, R. J., & Webster, J. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008006.pub2 · Cited by 68
Norris, S. L., Moher, D., Reeves, B. C., Shea, B., Loke, Y., Garner, S., Anderson, L., Tugwell, P., & Wells, G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1062 · Cited by 50
Impact of Cerebral Autoregulation Monitoring in Cerebrovascular Disease: A Systematic Review
Al‐Kawaz, M., Cho, S., Gottesman, R. F., Suárez, J. I., & Rivera‐Lara, L. (2022) · Neurocritical Care · doi:10.1007/s12028-022-01484-5 · Cited by 50
“Methods Search Strategy This review was preformed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines [6].”
MethodZhang, D., Ma, Y., Xu, J., & Yi, F. (2022) · Medicine · doi:10.1097/md.0000000000029443 · Cited by 52
“We evaluated the included studies according to the NewcastleOttawa scale (NOS).[11] A quantitative scoring device proposed by the Cochrane Collaboration was adopted to assess the studies’ methodological quality.”
MethodDietary Intake and Diabetic Retinopathy: A Systematic Review of the Literature
Shah, J., Cheong, Z. Y., Tan, B., Wong, D. W. K., Liu, X., & Chua, J. (2022) · Nutrients · doi:10.3390/nu14235021 · Cited by 60
“The modified version of the Newcastle–Ottawa Scale (NOS; Figure 4) was used to evaluate the quality of observational studies [31].”
MethodAesthetic experiences and their transformative power: a systematic review
Pizzolante, M., Pelowski, M. J., Demmer, T. R., Bartolotta, S., Sarcinella, E. D., Gaggioli, A., & Chirico, A. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1328449 · Cited by 67
Factors impacting on doctors’ management of acute low back pain: A systematic review
Fullen, B. M., Baxter, D. G., O'Donovan, B. G. G., Doody, C., Daly, L., & Hurley, D. A. (2008) · European Journal of Pain · doi:10.1016/j.ejpain.2008.11.013 · Cited by 53
“…test (Thomas et al., 2004) was used for quantitative studies, as a review of the literature identified this tool as one of the most appropriate for assessing non-randomised studies as well as RCTs (Deeks et al., 2003), and it has been used in 29 previous systematic reviews (www.phru.nhs.uk, 2006).”
MethodInfluentialNewton, A. S., Zou, B., Hamm, M. P., Curran, J., Gupta, S., Dumonceaux, C., & Lewis, M. (2010) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2009.00640.x · Cited by 53
Bambra, C. (2011) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2009.088740 · Cited by 50
“qualitative evidence.[39, 40] Another approach would be to examine the tools used by existing SRs in a similar subject area and adapt them.”
BackgroundRecent advances in the prenatal interrogation of the human fetal genome
Hui, L., & Bianchi, D. W. (2012) · Trends in Genetics · doi:10.1016/j.tig.2012.10.013 · Cited by 58
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.