The DOI matches its Crossref record.
How study design affects outcomes in comparisons of therapy. I: Medical
Abstract
We analysed 113 reports published in 1980 in a sample of medical journals to relate features of study design to the magnitude of gains attributed to new therapies over old. Overall we rated 87 per cent of new therapies as improvements over standard therapies. The mean gain (measured by the Mann—Whitney statistic) was relatively constant across study designs, except for non‐randomized controlled trials with sequential assignment to therapy, which showed a significantly higher likelihood that a patient would do better on the innovation than on standard therapy (p = 0.004). Randomized controlled trials that did not use a double‐blind design had a higher likelihood of showing a gain for the innovation than did double‐blind trials (p = 0.02). Any evaluation of an innovation may include both bias and the true efficacy of the new therapy, therefore we may consider making adjustments for the average bias associated with a study design. When interpreting an evaluation of a new therapy, readers should consider the impact of the following average adjustments to the Mann—Whitney statistic: for trials with non‐random sequential assignment a decrease of 0.15, for non‐double‐blind randomized controlled trials a decrease of 0.11.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
A comparison of observational studies and randomized, controlled trials.
Benson, K., & Hartz, A. J. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422506 · Cited by 2,347
Updated Method Guidelines for Systematic Reviews in the Cochrane Collaboration Back Review Group
van Tulder, M., Furlan, A., Bombardier, C., & Bouter, L. (2003) · Spine · doi:10.1097/01.brs.0000065484.95996.af · Cited by 2,180
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Chang, S.-H., Stoll, C. R. T., Song, J., Varela, J. E., Eagon, C. J., & Colditz, G. A. (2014) · JAMA Surgery · doi:10.1001/jamasurg.2013.3654 · Cited by 1,688
Self-citation2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group
Furlan, A. D., Pennick, V., Bombardier, C., & van Tulder, M. (2009) · Spine · doi:10.1097/brs.0b013e3181b1c99f · Cited by 1,663
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308
Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions
Evans, D. (2003) · Journal of Clinical Nursing · doi:10.1046/j.1365-2702.2003.00662.x · Cited by 993
Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies
Ioannidis, J. P. A. (2001) · JAMA · doi:10.1001/jama.286.7.821 · Cited by 919
Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review
Olivo, S. A., Macedo, L. G., Gadotti, I. C., Fuentes, J., Stanton, T., & Magee, D. J. (2008) · Physical Therapy · doi:10.2522/ptj.20070147 · Cited by 829
Kunz, R., & Oxman, A. D. (1998) · BMJ · doi:10.1136/bmj.317.7167.1185 · Cited by 770
Als-Nielsen, B., Chen, W., Gluud, C., & Kjaergard, L. L. (2003) · JAMA · doi:10.1001/jama.290.7.921 · Cited by 746
2015 Updated Method Guideline for Systematic Reviews in the Cochrane Back and Neck Group
Furlan, A. D., Malmivaara, A., Chou, R., Maher, C. G., Deyo, R. A., Schoene, M., Bronfort, G., & van Tulder, M. W. (2015) · Spine · doi:10.1097/brs.0000000000001061 · Cited by 704
The double-blind, randomized, placebo-controlled trial
Kaptchuk, T. J. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00347-4 · Cited by 595
Treatment-Emergent Sexual Dysfunction Related to Antidepressants: A Meta-Analysis
Serretti, A., & Chiesa, A. (2009) · Journal of Clinical Psychopharmacology · doi:10.1097/jcp.0b013e3181a5233f · Cited by 583
PEDro. A database of randomized trials and systematic reviews in physiotherapy
Sherrington, C., Herbert, R. D., Maher, C. G., & Moseley, A. M. (2000) · Manual Therapy · doi:10.1054/math.2000.0372 · Cited by 508
Balk, E. M. (2002) · JAMA · doi:10.1001/jama.287.22.2973 · Cited by 478
Randomisation to protect against selection bias in healthcare trials
Odgaard-Jensen, J., Vist, G. E., Timmer, A., Kunz, R., Akl, E. A., Schünemann, H., Briel, M., Nordmann, A. J., Pregno, S., & Oxman, A. D. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000012.pub3 · Cited by 439
Assessing the Quality of Randomized Controlled Trials:Current Issues and Future Directions
Moher, D., Jadad, A. R., & Tugwell, P. (1996) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300009570 · Cited by 436
Wilson, K., Gibson, N., Willan, A., & Cook, D. (2000) · Archives of Internal Medicine · doi:10.1001/archinte.160.7.939 · Cited by 415
Chiesa, A., & Serretti, A. (2014) · Substance Use & Misuse · doi:10.3109/10826084.2013.770027 · Cited by 348
Rising, K., Bacchetti, P., & Bero, L. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050217 · Cited by 328
MacLehose, Reeves, Harvey, Sheldon, Russell, & Black (2000) · Health Technology Assessment · doi:10.3310/hta4340 · Cited by 311
A Proposal for Structured Reporting of Randomized Controlled Trials
Andrew, E. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520240054041 · Cited by 309
Evidence of experimental bias in the life sciences: why we need blind data recording
Holman, L., Head, M. L., Lanfear, R., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002190 · Cited by 263
Devereaux, P. J. (2001) · JAMA · doi:10.1001/jama.285.15.2000 · Cited by 258
Bero, L., Oostvogel, F., Bacchetti, P., & Lee, K. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040184 · Cited by 249
Heterogeneity in Meta-analysis of Data from Epidemiologic Studies: A Commentary
Colditz, G. A., Burdick, E., & Mosteller, F. (1995) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117644 · Cited by 245
Self-citationEmergency medicine animal research: does use of randomization and blinding affect the results?
Bebarta, V., Luyten, D., & Heard, K. (2003) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2003.tb00056.x · Cited by 244
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207
A systematic review of workplace interventions to prevent low back pain
Maher, C. G. (2000) · Australian Journal of Physiotherapy · doi:10.1016/s0004-9514(14)60287-7 · Cited by 196
Development of The Cochrane Collaboration's Central Register of Controlled Clinical Trials
Dickersin, K., Manheimer, E., Wieland, S., Robinson, K. A., Lefebvre, C., & McDonald, S. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001004 · Cited by 198
Development of a quality assessment instrument for trials of treatments for depression and neurosis
Moncrieff, J., Churchill, R., Drummond, D. C., & McGuire, H. (2001) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.108 · Cited by 185
Androgen Treatment and Muscle Strength in Elderly Men: A Meta‐Analysis
Ottenbacher, K. J., Ottenbacher, M. E., Ottenbacher, A. J., Acha, A. A., & Ostir, G. V. (2006) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2006.00938.x · Cited by 173
Antidepressants plus benzodiazepines for major depression
Furukawa, T. A., Streiner, D., Young, L. T., & Kinoshita, Y. (2001) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001026 · Cited by 161
Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis
Hempel, S., Miles, J. N., Booth, M. J., Wang, Z., Morton, S. C., & Shekelle, P. G. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-107 · Cited by 137
Why use placebos in clinical trials? A narrative review of the methodological literature
Vickers, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00139-0 · Cited by 121
Mhaskar, R., Djulbegovic, B., Magazin, A., Soares, H. P., & Kumar, A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.016 · Cited by 102
Bouça-Machado, R., Rosário, M., Alarcão, J., Correia-Guedes, L., Abreu, D., & Ferreira, J. J. (2017) · BMC Palliative Care · doi:10.1186/s12904-016-0181-9 · Cited by 95
Colle, F., Rannou, F., Revel, M., Fermanian, J., & Poiraudeau, S. (2002) · Archives of Physical Medicine and Rehabilitation · doi:10.1053/apmr.2002.35657 · Cited by 68
Psychotherapy for depression in children and adolescents
Watanabe, N., Churchill, R., Hunot, V., & Furukawa, T. A. (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005334 · Cited by 24
RetractedMindfulness-Based Interventions
Bockstaele, B. V., Salemink, E., Ostafin, B. D., Meijer, & Bögels, S. M. (2015) · Clinical Psychology · doi:10.1002/9781394259229.ch19 · Cited by 1
Metadata
- Authors
- Colditz, G. A., Miller, J. N., & Mosteller, F.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 1989-04
- Volume
- 8
- Issue
- 4
- Pages
- 441-454
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2169997396
- Cited by
- 520
- References
- 0
- FWCI
- 5.97
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.