Evaluating non-randomised intervention studies
Open accessThe DOI matches its Crossref record; 334 of the 334 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To consider methods and related evidence for evaluating bias in non-randomised intervention studies. DATA SOURCES: Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. METHODS: Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. RESULTS: In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. CONCLUSIONS: Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This tool was selected for its comprehensive ability to assess the methodological quality of non‐randomized studies and has shown good reliability and validity.21, 22”
From The frequency of influenza and bacterial coinfection: a systematic review and meta-analysis. (2016) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“In general, the probability of systematic differences in comparison groups at baseline was low, diminishing the risk of selection bias (58).”
From Mindfulness-Based Program for Infertility: Efficacy study (2013) · cites this paper as [58]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The EPHPP has also been reported to have better interrater reliability than the Cochrane Collaboration Risk of Bias Tool”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Deeks et al., 2003
Experimental — accuracy not yet measuredIn the citing paper
“In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13].”
From RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions (2023) · cites this paper as [13]
In the citing paper
“The EPHPP is considered suitable to assess the quality of a wide range of quantitative study designs, including interventions (Jackson and Waters, 2005;Deeks et al., 2003).”
From Exploring the role of exposure to green and blue spaces in preventing anxiety and depression among young people aged 14–24 years living in urban settings: A systematic review and conceptual framework (2022) · cites this paper as Deeks et al., 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Part of this paper’s reference list could not be split into single entries.
Items 201–334 of 334
Bradley, C. (1993) · Diabetes Care · doi:10.2337/diacare.16.2.509 · Cited by 105
Group Education Interventions for People With Low Back Pain: An Overview of the Literature
Cohen, J. E., Goel, V., Frank, J. W., Bombardier, C., Peloso, P., & Guillemin, F. (1994) · Spine · doi:10.1097/00007632-199405310-00004 · Cited by 104
Groin hernia surgery: a systematic review.
Cheek, C., Black, N., Devlin, H. B., Kingsnorth, A. N., Taylor, R. S., & Watkin, D. F. L. (1998) · PubMed · Cited by 102
Verhagen, A. P., de Vet, H. C. W., de Bie, R. A., Kessels, A. G. H., Boers, M., & Knipschild, P. G. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00297-7 · Cited by 102
Measuring effectiveness. What to expect without a randomized control group.
D’Agostino, R. B., & Russell, H. K. (1995) · PubMed · Cited by 100
Planning of Research on Effectiveness of Psychotherapy
Fiske, D. W. (1970) · Archives of General Psychiatry · doi:10.1001/archpsyc.1970.01740250024004 · Cited by 97
Elective cyclosporine withdrawal after renal transplantation. A meta-analysis
Kasiske, B. L. (1993) · JAMA · doi:10.1001/jama.269.3.395 · Cited by 97
Psychological outreach programmes for the depressed elderly: a meta-analysis of effects and dropout
Cuijpers, P. (1998) · International Journal of Geriatric Psychiatry · doi:10.1002/(sici)1099-1166(199801)13:1<41::aid-gps729>3.0.co;2-b · Cited by 97
Chapter 11: Issues in Meta-Analysis
Hedges, L. V. (1986) · Review of Research in Education · doi:10.2307/1167228 · Cited by 96
Covariance analysis in generalized linear measurement error models
Caroll, R. J. (1989) · Statistics in Medicine · doi:10.1002/sim.4780080907 · Cited by 96
Quality assessment of randomized control trials in dental research I. Methods
Antczak, A. A., Tang, J., & Chalmers, T. C. (1986) · Journal of Periodontal Research · doi:10.1111/j.1600-0765.1986.tb01464.x · Cited by 95
A Review of Psychosocial Interventions for Children With Chronic Health Conditions
Bauman, L. J., Drotar, D., Leventhal, J. M., Perrin, E. C., & Pless, I. B. (1997) · Pediatrics · doi:10.1542/peds.100.2.244 · Cited by 93
Effectiveness of Influenza Vaccine in the Elderly
Nichol, K. L., Margolis, K. L., Wouremna, J., & von Sternberg, T. (1996) · Gerontology · doi:10.1159/000213803 · Cited by 92
Adjustment for Regression Dilution in Epidemiological Regression Analyses
Knuiman, M. W., Divitini, M., Buzas, J. S., & FitzGerald, P. E. B. (1998) · Annals of Epidemiology · doi:10.1016/s1047-2797(97)00107-5 · Cited by 92
The safety of early postpartum discharge: A review and critique
GRULLON, K., & GRIMES, D. (1997) · Obstetrics & Gynecology · doi:10.1016/s0029-7844(97)00405-5 · Cited by 89
Declaring independence: why we should be cautious.
Smith, G. D., & Phillips, A. (1990) · Journal of Epidemiology and Community Health · doi:10.1136/jech.44.4.257 · Cited by 88
Assessing Reports of Therapeutic Trials
Lionel, N. D. W., & Herxheimer, A. (1970) · BMJ · doi:10.1136/bmj.3.5723.637 · Cited by 88
A qualitative review of psychosocial treatments for bulimia.
Gl, C., & Merkel, W. T. (1989) · PubMed · Cited by 86
Evaluating Intensive Family Preservation Programs: A Methodological Review
Heneghan, A. M., Horwitz, S. M., & Leventhal, J. M. (1996) · Pediatrics · doi:10.1542/peds.97.4.535 · Cited by 85
Evans, M., & Pollock, A. V. (1985) · Journal of British Surgery · doi:10.1002/bjs.1800720403 · Cited by 85
Partner notification for sexually transmitted diseases: an overview of the evidence.
Ad, O., Scott, E. A., Sellors, J. W., Clarke, J. H., Millson, M. E., Rasooly, I., Frank, J. W., Naus, M., & Goldblatt, E. (1994) · PubMed · Cited by 84
Preschool Vision Screening: Results of a Systematic Review
Snowden, S., & Stewart-Brown, S. L. (1997) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 84
How to read clinical journals.
Sackett, D. L. (1982) · PubMed Central · Cited by 84
Treatment of depression in Parkinson's disease: a meta-analysis
(1995) · The Journal of Neuropsychiatry and Clinical Neurosciences · doi:10.1176/jnp.7.3.281 · Cited by 82
Oral appliances for the treatment of snoring and obstructive sleep apnea
Schoem, S. R. (2000) · Otolaryngology–Head and Neck Surgery · doi:10.1016/s0194-5998(00)70250-6 · Cited by 82
A Potential Pitfall in Control of Covariates in Epidemiologic Studies
Brenner, H. (1998) · Epidemiology · doi:10.1097/00001648-199801000-00014 · Cited by 82
Effectiveness of interventions to prevent delirium in hospitalized patients: a systematic review.
Cole, M. G., Primeau, F. J., & McCusker, J. (1996) · PubMed Central · Cited by 81
A Meta-Analysis of the Effects Of Psychoeducational Care in Adults with Hypertension
DEVINE, E. C., & REIFSCHNEIDER, E. (1995) · Nursing Research · doi:10.1097/00006199-199507000-00009 · Cited by 80
Interferon as treatment for acute hepatitis C
Cammà, C., Almasio, P. L., & Craxı̀, A. (1996) · Digestive Diseases and Sciences · doi:10.1007/bf02088245 · Cited by 79
Medical abortion in early pregnancy: A review of the evidence
Grimes, D. A. (1997) · Obstetrics and Gynecology · doi:10.1016/s0029-7844(97)81439-1 · Cited by 79
Schoemaker, C. (1997) · International Journal of Eating Disorders · doi:10.1002/(sici)1098-108x(199701)21:1<1::aid-eat1>3.0.co;2-r · Cited by 78
Cao, W., Van der Ploeg, C. P. B., Plaisier, A. P., van der Sluijs, I. J. S., & Habbema, J. D. F. (1997) · Tropical Medicine & International Health · doi:10.1111/j.1365-3156.1997.tb00157.x · Cited by 77
Meta-analysis of anterior veneer restorations in clinical studies
Kreulen, C. M., Creugers, N. H. J., & Meijering, A. C. (1998) · Journal of Dentistry · doi:10.1016/s0300-5712(97)00023-7 · Cited by 74
Wellness programs: a review of the evidence.
Watt, D., Verma, S., & Flynn, L. M. (1998) · Europe PMC (PubMed Central) · Cited by 73
Nicolucci, A. (1989) · JAMA · doi:10.1001/jama.262.15.2101 · Cited by 73
(1994) · PubMed · Cited by 69
Failed or misleading adjustment for confounding
Leon, D. A. (1993) · The Lancet · doi:10.1016/0140-6736(93)91599-h · Cited by 69
Acetazolamide or dexamethasone for prevention of acute mountain sickness: a meta-analysis
Ried, L. D., Carter, K. A., & Ellsworth, A. J. (1994) · Journal of Wilderness Medicine · doi:10.1580/0953-9859-5.1.34 · Cited by 68
An evaluation of the quality of therapeutic studies in perinatal medicine
Tyson, J. E., Furzán, J. A., REISCH, J. S., & Mize, S. G. (1983) · The Journal of Pediatrics · doi:10.1016/s0022-3476(83)80277-7 · Cited by 67
Current Management of the Neonatal Abstinence Syndrome: A Critical Analysis of the Evidence
Theis, J. G. W., Selby, P. L., Ikizler, Y., & Koren, G. (2009) · Biology of the Neonate · doi:10.1159/000244435 · Cited by 67
Measuring effects without randomized trials? Options, problems, challenges.
Moses, L. E. (1995) · PubMed · Cited by 67
Sellers, D. E., Crawford, S. L., Bullock, K., & McKinlay, J. B. (1997) · Social Science & Medicine · doi:10.1016/s0277-9536(96)00263-8 · Cited by 64
Checklist for the evaluation of research articles
DuRant, R. H. (1994) · Journal of Adolescent Health · doi:10.1016/1054-139x(94)90381-6 · Cited by 61
Quality of Life in Cardiac Patient Research: A Meta-Analysis
Kinney, M. R., Burfitt, S. N., Stullenbarger, E., Rees, B., & DeBolt, M. R. (1996) · Nursing Research · doi:10.1097/00006199-199605000-00009 · Cited by 58
Primary Prevention Mental Health Programs: The Future is Exciting
Durlak, J. A., & Wells, A. M. (1997) · American Journal of Community Psychology · doi:10.1023/a:1024674631189 · Cited by 55
Community Action for Health Promotion: A Review of Methods and Outcomes 1990-1995
Hancock, L., Sanson‐Fisher, R., Redman, S., Burton, R. C., Burton, L., Butler, J., Girgis, A., Gibberd, R. W., Hensley, M., McClintock, A., et al. (1997) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(18)30168-5 · Cited by 55
An overview of statistical issues and methods of meta-analysis
Schmid, E. J., Koch, G. G., & LaVange, L. M. (1991) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543409108835008 · Cited by 55
Curatolo, M., Petersen‐Felix, S., Scaramozzino, P., & Zbinden, A. M. (1998) · Acta Anaesthesiologica Scandinavica · doi:10.1111/j.1399-6576.1998.tb05349.x · Cited by 55
The role of covariates in estimating treatment effects and risk in long‐term clinical trials
Ford, I., & Norrie, J. (2002) · Statistics in Medicine · doi:10.1002/sim.1294 · Cited by 55
Clinical evaluation of single versus multiple mammary artery bypass.
Morris, J. J., Smith, L. R., Glower, D. D., Muhlbaier, L. H., Reves, J. G., Wechsler, A. S., & Rankin, J. S. (1990) · PubMed · Cited by 55
Lee, C. M., Picard, M., & Blain, M. D. (1994) · Journal of Family Psychology · doi:10.1037/0893-3200.8.1.3 · Cited by 54
A systematic overview of adolescent suicide prevention programs.
Ploeg, J., Ciliska, D. K., Dobbins, M., Hayward, S., Thomas, H., & Underwood, J. (1996) · PubMed · Cited by 53
Critical Appraisal of the Literature
Miser, W. F. (1999) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.12.4.315 · Cited by 53
A Meta-Analysis of the Effectiveness of Occupational Therapy for Older Persons
Carlson, M. E., Fanchiang, S.-P. C., Zemke, R., & Clark, F. A. (1996) · American Journal of Occupational Therapy · doi:10.5014/ajot.50.2.89 · Cited by 51
Compression Treatment – Still Incompletely Understood
Smith, P. D. C. (1999) · Phlebology The Journal of Venous Disease · doi:10.1177/026835559901400101 · Cited by 50
Ciliska, D. K., Hayward, S., Thomas, H., Mitchell, A., Dobbins, M., Underwood, J. M., Rafael, A., & Martin, E. (1996) · PubMed · Cited by 50
Schulz, K. F., Chalmers, I., Altman, D. G., Grimes, D. A., & Doré, C. J. (1995) · PubMed · Cited by 49
Self-citationA METHOD FOR THE ASSESSMENT OF REPORTS OF DRUG TRIALS.
MAHON, W. A., & Daniel, E. E. (1964) · PubMed Central · Cited by 48
[Evaluation of the methodological quality of randomized therapeutic trials].
Poynard, T. (1988) · PubMed · Cited by 46
Quality assessment and meta‐analysis of systemic tetracycline use in chronic adult periodontitis
Hayes, C., Antczak‐Bouckoms, A., & Burdick, E. (1992) · Journal Of Clinical Periodontology · doi:10.1111/j.1600-051x.1992.tb00633.x · Cited by 44
Reporting observational studies
Bracken, M. B. (1989) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.1989.tb02410.x · Cited by 44
Margetts, B. M., Thompson, R. L., Key, T., Duffy, S., Nelson, M., Bingham, S., & Wiseman, M. (1995) · Nutrition and Cancer · doi:10.1080/01635589509514412 · Cited by 44
Principles and practice of research : strategies for surgical investigators
Troidl, H. (1991) · Springer eBooks · Cited by 43
Lijesen, G., Theeuwen, I., Assendelft, W., & Van Der Wal, G. (1995) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.1995.tb05779.x · Cited by 43
Spectral properties of phototherapy for seasonal affective disorder: a meta‐analvsis
Lee, T. M. C., Chan, C. C. H., Paterson, J. G., Janzen, H. L., & Blashko, C. A. (1997) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1997.tb09915.x · Cited by 43
Effects of Respite Care on Patients With Dementia and Their Caregivers
Flint, A. J. (1995) · International Psychogeriatrics · doi:10.1017/s1041610295002249 · Cited by 43
A prototype for integrative review and meta‐analysis of nursing research
Smith, M. C., & Stullenbarger, E. (1991) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.1991.tb01554.x · Cited by 43
Observational Studies and Randomized Trials
Kunz, R., Khan, K. S., & Neumayer, H. (2000) · New England Journal of Medicine · doi:10.1056/nejm200010193431613 · Cited by 43
Impact of random assignment on study outcome: An empirical examination
Ottenbacher, K. J. (1992) · Controlled Clinical Trials · doi:10.1016/0197-2456(92)90029-y · Cited by 41
Measurement of quality of primary studies for meta-analysis.
Brown, S. A. (1991) · PubMed · Cited by 39
de Kruif, Y. P., & van Wegen, E. E. H. (1996) · Physiotherapy · doi:10.1016/s0031-9406(05)66964-7 · Cited by 39
Long acting drug combinations in rheumatoid arthritis: a formal overview.
Boers, M., & Ramsden, M. (1991) · PubMed · Cited by 39
Review of the evidence for the use of erythromycin in the management of persons exposed to pertussis
Dodhia, H., & Miller, E. (1998) · Epidemiology and Infection · doi:10.1017/s0950268897008571 · Cited by 38
Identifying Confounding by Indication through Blinded Prospective Review
Johnston, S. C. (2001) · American Journal of Epidemiology · doi:10.1093/aje/154.3.276 · Cited by 37
Audet, N., Gagnon, R., Ladouceur, R., & Marcil, M. N. (1993) · Europe PMC (PubMed Central) · Cited by 36
A Systematic Review of Single‐Dose Intramuscular Methotrexate for the Treatment of Ectopic Pregnancy
Parker, J., Bisks, A., & Proietto, A. M. (1998) · Australian and New Zealand Journal of Obstetrics and Gynaecology · doi:10.1111/j.1479-828x.1998.tb02988.x · Cited by 36
The Ineffectiveness of Immunosuppressive Therapy in Lymphocytic Myocarditis: An Overview
Garg, A., Shiau, J., & Guyatt, G. (1998) · Annals of Internal Medicine · doi:10.7326/0003-4819-129-4-199808150-00009 · Cited by 36
Bours, G. J. J. W., Ketelaars, C. A. J., Frederiks, C. M. A., Abu‐Saad, H. H., & Wouters, E. F. M. (1998) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.1998.00647.x · Cited by 36
Ernst, E. K., & Barnes, J. (1998) · Cited by 36
A checklist system for critical review of medical literature
KROGH, C. L. (1985) · Medical Education · doi:10.1111/j.1365-2923.1985.tb01343.x · Cited by 35
A Critical Review of Studies of Newborn Discharge Timing
Margolis, L. H. (1995) · Clinical Pediatrics · doi:10.1177/000992289503401201 · Cited by 35
Colonoscopic Surveillance for Cancer in Ulcerative Colitis: A Critical Review
Griffiths, A. M., & Sherman, P. M. (1997) · Journal of Pediatric Gastroenterology and Nutrition · doi:10.1097/00005176-199702000-00015 · Cited by 35
Smith, T. M. F., & Sugden, R. A. (1988) · International Statistical Review · doi:10.2307/1403640 · Cited by 33
Richard, C. S., & McLeod, R. (1997) · Europe PMC (PubMed Central) · Cited by 32
Patient self-administration of medication: a review of the literature
Collingsworth, S., Gould, D. J., & Wainwright, S. P. (1997) · International Journal of Nursing Studies · doi:10.1016/s0020-7489(97)00013-8 · Cited by 31
Review of Effectiveness of Sexual Health Promotion Interventions for Young People (updated)
Oakley, A., Oliver, S. R., Peersman, G., & Thomas, J. (1995) · Cited by 30
Active Management of Labor: A Meta-Analysis of Cesarean Delivery Rates for Dystocia in Nulliparas
Glantz, J. C., & McNanley, T. J. (1997) · Obstetrical & Gynecological Survey · doi:10.1097/00006254-199708000-00021 · Cited by 30
Hyperbaric oxygen for multiple sclerosis
Kleijnen, J., & Knipschild, P. G. (1995) · Acta Neurologica Scandinavica · doi:10.1111/j.1600-0404.1995.tb07017.x · Cited by 29
Alcoholism treatment outcome studies, 1980–1992: Methodological characteristics and quality
Morley, J. A., Finney, J. W., Monahan, S. C., & Floyd, A. S. (1996) · Addictive Behaviors · doi:10.1016/0306-4603(95)00073-9 · Cited by 28
Design issues in cohort studies
Prentice, R. L. (1995) · Statistical Methods in Medical Research · doi:10.1177/096228029500400402 · Cited by 28
Oliveira, I. R., de Sena, E. P., Pereira, E. L. A., MIRANDA, A. C. D. C., Oliveira, N. F., Ribeiro, M. G., Castro-e-Silva, E., Dardennes, R. M., SAMUEL-LAJEUNESSE, B., & Marcílio, C. (1996) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1111/j.1365-2710.1996.tb01143.x · Cited by 28
Moncrieff, J., & Drummond, C. (1998) · Addiction · doi:10.1046/j.1360-0443.1998.9368113.x · Cited by 27
Riben, P., Mathias, R., Campbell, E., & Wiens, M. (1994) · PubMed · Cited by 25
Naylor, C. D., Detsky, A. S., O’Rourke, K., & Fonberg, E. (1987) · Renal Failure · doi:10.3109/08860228709047649 · Cited by 24
A research appraisal checklist for evaluating nursing research reports.
Duffy, M. E. (1985) · PubMed · Cited by 22
Heacock, H. J., Koehoorn, M. W., & Tan, J. (1997) · Applied Ergonomics · doi:10.1016/s0003-6870(96)00066-x · Cited by 22
An Instrument for Evaluating Experimental Educational Research Report
Suydam, M. N. (1968) · The Journal of Educational Research · doi:10.1080/00220671.1968.10883645 · Cited by 22
Assessing the methodological quality of research in narrative reviews and meta-analyses.
Bangert‐Drowns, R. L., Wells-Parker, E. N., & Chevillard, I. (1997) · American Psychological Association eBooks · doi:10.1037/10222-012 · Cited by 22
Clinical investigation: A guide to its evaluation
Hines, D. C., & Goldzieher, J. W. (1969) · American Journal of Obstetrics and Gynecology · doi:10.1016/0002-9378(69)90280-4 · Cited by 19
The effect of therapy on the mental health of the elderly
Burckhardt, C. S. (1987) · Research in Nursing & Health · doi:10.1002/nur.4770100411 · Cited by 19
Critical appraisal: How to read a clinical research paper
Vickers, A. J. (1995) · Complementary Therapies in Medicine · doi:10.1016/s0965-2299(95)80057-3 · Cited by 18
Chynn, E. W. (1994) · Archives of Ophthalmology · doi:10.1001/archopht.1994.01090190017009 · Cited by 16
Role of topical immunotherapy in the treatment of alopecia areata
Naldi, L., Parazzini, F., CAINELLI, T., & Reading Group (1990) · Journal of the American Academy of Dermatology · doi:10.1016/0190-9622(90)70092-v · Cited by 15
de Oliveira, I., Dardennes, R. M., Amorim, E., Diquet, B., de Sena, E. P., Moreira, E., Castro-e-Silva, E., Payan, C., Fermanian, J., Marcílio, C., et al. (1995) · Fundamental and Clinical Pharmacology · doi:10.1111/j.1472-8206.1995.tb00524.x · Cited by 15
Antioxidant Nutrients in Pregnancy: a Systematic Review of the Literature
Mathews, F. (1996) · Nutrition Research Reviews · doi:10.1079/nrr19960011 · Cited by 15
A Check List for Evaluating Experimental Research in Psychology and Education
Farquhar, W. W., & Krumboltz, J. D. (1959) · The Journal of Educational Research · doi:10.1080/00220671.1959.10882596 · Cited by 14
Beel-Bates, C. (1991) · PubMed · Cited by 14
An algorithm for prospective individual matching in a non-randomized clinical trial
Charpentier, P. A., Bogardus, S. T., & Inouye, S. K. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00399-7 · Cited by 13
META-ANALYSIS OF RESEARCH ON DUI REMEDIAL INTERVENTIONS
Wells-Parker, E. N., & Bangert‐Drowns, R. L. (1990) · Alcohol, drugs and driving · Cited by 12
Psychotherapy outcome research: An application of a new method for evaluating research methodology.
Morris, S., Turner, C. W., & Szykula, S. A. (1988) · Psychotherapy · doi:10.1037/h0085319 · Cited by 12
A Critical Appraisal of Positioning Infants in the Neonatal Intensive Care Unit
Long, T. M., & Soderstrom, E. (1996) · Physical & Occupational Therapy In Pediatrics · doi:10.1080/j006v15n03_02 · Cited by 12
Critical Appraisal of Published Research
Schechter, M. T., LeBlanc, F. E., & Lawrence, V. A. (1991) · Springer eBooks · doi:10.1007/978-1-4684-0371-8_10 · Cited by 10
Meta-Analysis of Physiological Outcomes of Hospital-Based Infant Intervention Programs
Krywanio, M. L. (1994) · Nursing Research · doi:10.1097/00006199-199405000-00002 · Cited by 10
Judging Educational Research Based on Experiments and Surveys
Wolf, R. M. (1993) · Medical Entomology and Zoology · Cited by 9
Efficacy of self-help and alternative treatments of premenstrual syndrome
Carter, J., & Verhoef, M. J. (1994) · Women s Health Issues · doi:10.1016/s1049-3867(05)80052-0 · Cited by 9
A Guide to Understanding and Evaluating Research Articles
Nielsen, M. E., & Reilly, P. L. (1985) · Gifted Child Quarterly · doi:10.1177/001698628502900210 · Cited by 9
Meijman, F. J., & MELKER, R. A. D. (1995) · Family Practice · doi:10.1093/fampra/12.1.93 · Cited by 9
Judging Educational Research Based on Experiments and Surveys
Wolf, R. M. (1993) · Cited by 9
Cooper, B. C., Mullins, P. R., Jones, M. R., & Lang, S. D. R. (1994) · Clinical Drug Investigation · doi:10.1007/bf03258472 · Cited by 8
Blood glucose and diabetic retinopathy: A critical appraisal of new evidence
Mulrow, C. D., & Lichtenstein, M. J. (1986) · Journal of General Internal Medicine · doi:10.1007/bf02599803 · Cited by 8
Ciliska, D. K., Kelly, C. T., Petrov, N., & Chalmers, J. P. (1995) · Cited by 8
What guidelines should be followed in critically evaluating research reports?
Mj, W., & Me, F. (1979) · PubMed · Cited by 7
The Impact of Geriatric Medical Services on Mental State
Cole, M. G. (1993) · International Psychogeriatrics · doi:10.1017/s1041610293001425 · Cited by 6
Rutherford, W. F. (1994) · Air Medical Journal · doi:10.1016/s1067-991x(05)80002-1 · Cited by 5
Methodological Guidelines for Reading Drug-Evaluation Research
King, P. R. (1984) · The Canadian Journal of Psychiatry · doi:10.1177/070674378402900706 · Cited by 5
Essential design features of a controlled clinical trial
Meinert, C. L., & Tonascia, S. (1986) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195035681.003.0008 · Cited by 5
Agoraphobia: a critical review of methodology in behavioural treatment research
Gray, M. A., & McPherson, I. G. (1982) · Current Psychological Reviews · doi:10.1007/bf02684453 · Cited by 4
Letter: Assessment of papers of clinical trials.
Nyberg, G. (1974) · PubMed · Cited by 4
Common Deficiencies of NIDRR Research Applications
Thomas, J. P., & Lawrence, T. S. (1990) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/00002060-199004000-00005 · Cited by 3
The Factorial Validity of a Rating Scale for the Evaluation of Research Articles
Shay, C. B., Zimmerman, W. S., & Michael, W. B. (1972) · Educational and Psychological Measurement · doi:10.1177/001316447203200225 · Cited by 3
Leboeuf, C. (1990) · PubMed · Cited by 3
Jonsson, C., Mårtens, S., & Sjöqvist, F. (1969) · Nordic Journal of Psychiatry · doi:10.3109/08039486909103684 · Cited by 2
Teaching Psychotherapy Outcome Research Methodology Using a Research-Based Checklist
Osberg, T. M. (1997) · Teaching of Psychology · doi:10.1207/s15328023top2404_13 · Cited by 1
D’Agostino (1998) · doi:10.1002/(sici)1097-0258(19981015)17:19<2265::aid-sim918>3.0.co;2-b
Research works printed but not linked: 19
- Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487-97.
- Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidence-based medicine". Am J Med 1997;103:529-35.
- Horwitz RI, Feinstein AR. The application of therapeutic trial principles to improve the design of epidemiologic research. J Chron Dis 1981;34:575-83.
- Byar DP. Problems with using observational data bases to compare treatments. Stat Med 1991;10:663-6.
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.
- Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane Methodology Review). In The Cochrane Library, Issue 4. Oxford: Update Software; 2002.
- Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986;4:1114-20.
- Lipsey MW, Wilson DB. The efficacy of psychological educational, and behavioural treatments: confirmation from meta-analysis. Am Psychol 1993;48:1181-209.
- Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989;8:455-66.
- Pocock SJ, Elbourne DR. Randomized trials or observational tribulations? N Engl J Med 2000;342:1907-9.
- Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996;12:195-208.
- Critical Appraisal Skills Programme. 12 questions to help you make sense of a cohort study. Oxford: Critical Appraisal Skills Programme; 1999.
- Kleijnen J, Knipschild P, Ter Riet G. Clinical trials of homeopathy. BMJ 1991;302:316-32.
- van der Windt DA, van der Heijden GJ, Scholten RJ, Koes BW, Bouter LM. The efficacy of non-steroidal anti-inflammatory drugs for shoulder complaints. J Clin Epidemiol 1995;48:691-704.
- Kleijnen J, Riet ter G, Knipschild P. Vitamine C en verkoudheid. Overzicht van een megadosis literatuur. Ned Tijdschr Geneeskd 1989;133:1535.
- Riet ter G, Kleijnen J, Knipschild P. De meta- analyse als review-methode [De effectiviteit van acupunctuur]. Huisarts en Wetenschap 1989;32: 176-81.
- Nicolucci A, Grilli R, Alexanian A, Apolone G, Torri V, Liberati A. Quality, evolution, and clinical implications of randomized, controlled trials on the treatment of lung cancer. JAMA 1989; 262:2101-7.
- Baker DW, Jones R, Hodges J, Massie BM, Konstam MA, Rose EA. Management of heart failure. III. The role of revascularization in the treatment of patients with moderate or severe left ventricular systolic dysfunction. JAMA 1994;272:1528-34.
- Coleridge Smith P. The management of chronic venous disorders of the leg: an evidence-based report of an International Task Force. Phlebology 1999;14:3-19.
Cited by
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationStang, A. (2010) · European Journal of Epidemiology · doi:10.1007/s10654-010-9491-z · Cited by 17,396
“concluded that with a few caveats (missing item for the appropriateness of the analysis, lack of information related to the reliability and validity), the NOS is “suitable for use in a systematic review” and is “easy to use” [3].”
BackgroundMethodInfluentialMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
“[305] [306] Bias jeopardises even RCTs, however, if investigators carry out such trials improperly.”
BackgroundSelf-citationGRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Systematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,089
Metadata
- Authors
- Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.
- Affiliations
- University of Oxford (GB)
- University of Southampton (GB)
- University of York (GB)
- University of Birmingham (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Health Technology Assessment
- Published
- 2003-09
- Volume
- 7
- Issue
- 27
- Pages
- iii-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1981501248
- Semantic Scholar
- 3b3a2281d097cc0c60e93d42d570bd2dfadb6492
- Cited by
- 2,907
- References
- 332
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 107 · 2023: 115 · 2022: 142 · 2021: 188 · 2020: 176 · 2019: 164 · 2018: 148 · 2017: 191 · 2016: 169 · 2015: 185 · 2014: 177 · 2013: 141 · 2012: 123
- FWCI
- 81.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/26/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- bias in observational studies, internal validity, quality assessment tools, risk adjustment, residual confounding, propensity score methods, meta-epidemiology, treatment effect estimation, prognostic factors, selection bias, confounding, reporting bias, reporting quality, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Diagnosis-Related Groups, Humans, Quality Control, Research Design, Reproducibility of Results, Bias, Selection Bias, Patient Selection, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.