The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Statistical Problems in the Reporting of Clinical Trials
Abstract
Reports of clinical trials often contain a wealth of data comparing treatments. This can lead to problems in interpretation, particularly when significance testing is used extensively. We examined 45 reports of comparative trials published in the British Medical Journal, the Lancet, or the New England Journal of Medicine to illustrate these statistical problems. The issues we considered included the analysis of multiple end points, the analysis of repeated measurements over time, subgroup analyses, trials of multiple treatments, and the overall number of significance tests in a trial report. Interpretation of large amounts of data is complicated by the common failure to specify in advance the intended size of a trial or statistical stopping rules for interim analyses. In addition, summaries or abstracts of trials tend to emphasize the more statistically significant end points. Overall, the reporting of clinical trials appears to be biased toward an exaggeration of treatment differences. Trials should have a clearer predefined policy for data analysis and reporting. In particular, a limited number of primary treatment comparisons should be specified in advance. The overuse of arbitrary significance levels (for example, P less than 0.05) is detrimental to good scientific reporting, and more emphasis should be given to the magnitude of treatment differences and to estimation methods such as confidence intervals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A method for assessing the quality of a randomized control trial
Chalmers, T. C., Smith, H., Blackburn, B., Silverman, B., Schroeder, B., Reitman, D., & Ambroz, A. (1981) · Controlled Clinical Trials · doi:10.1016/0197-2456(81)90056-8 · Cited by 1,349
Research works printed but not linked: 25
- Antczak, 1986, Periodontal Res, 21, 305, 10.1111/j.1600-0765.1986.tb01464.x · DOI
- DerSimonian, 1982, N Engl J Med, 306, 1332, 10.1056/NEJM198206033062204 · DOI
- Moskowitz, 1983, Alcoholism (NY), 7, 42, 10.1111/j.1530-0277.1983.tb05409.x · DOI
- Liberati, 1986, J Clin Oncol, 4, 942, 10.1200/JCO.1986.4.6.942 · DOI
- Antczak, 1986, Periodontal Res, 21, 315, 10.1111/j.1600-0765.1986.tb01465.x · DOI
- EC/IC Bypass Study Group, 1985, N Engl J Med, 313, 1191, 10.1056/NEJM198511073131904 · DOI · Paper page
- Mak, 1985, Br Med J, 291, 623, 10.1136/bmj.291.6496.623 · DOI
- Miller RG Jr. Simultaneous statistical inference. New York: Springer-Verlag, 1981. · DOI
- O'Brien, 1984, Biometrics, 40, 1079, 10.2307/2531158 · DOI · Paper page
- McCormick, 1986, N Engl J Med, 314, 20, 10.1056/NEJM198601023140104 · DOI · Paper page
- Pocock SJ. Clinical trials: a practical approach. Chichester. England: John Wiley, 1983.
- Medical Research Council Working Party, 1985, Br Med J, 291, 97, 10.1136/bmj.291.6488.97 · DOI · Paper page
- Altman, 1984, Br Med J, 289, 336, 10.1136/bmj.289.6441.336 · DOI
- Lavori, 1983, N Engl J Med, 309, 1291, 10.1056/NEJM198311243092105 · DOI
- Greene, 1985, N Engl J Med, 313, 133, 10.1056/NEJM198507183130301 · DOI
- de Klerk, 1986, Aust NZ J Med, 16, 637, 10.1111/j.1445-5994.1986.tb00001.x · DOI
- Healy MJR. Some problems of repeated measurements. In: Bithell JF, Coppi R. eds. Perspectives in medical statistics: proceedings of the European Symposium on Medical Statistics, Rome, 1980. London: Academic Press. 1981:155–71.
- Armitage P. Statistical methods in medical research. Oxford: Blackwell, 1971:202–7.
- Gardner, 1986, Br Med J, 292, 746, 10.1136/bmj.292.6522.746 · DOI · Paper page
- Simon, 1986, Ann Intern Med, 105, 429, 10.7326/0003-4819-105-3-429 · DOI · Paper page
- Cappuccio, 1985, Br Med J, 291, 235, 10.1136/bmj.291.6490.235 · DOI
- Altman, 1980, Br Med J, 281, 1336, 10.1136/bmj.281.6251.1336 · DOI · Paper page
- Gore, 1981, Br Med J, 282, 1687, 10.1136/bmj.282.6277.1687 · DOI
- Geller, 1987, Biometrics, 43, 213, 10.2307/2531962 · DOI
- Fleming, 1984, Controlled Clin Trials, 5, 348, 10.1016/S0197-2456(84)80014-8 · DOI
Cited by
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000412 · Cited by 7,300
Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72351 · Cited by 6,277
The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200
More powerful procedures for multiple significance testing
Hochberg, Y., & Benjamini, Y. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090710 · Cited by 2,715
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2012) · International Journal of Surgery · doi:10.1016/j.ijsu.2011.10.001 · Cited by 2,345
Husereau, D., Drummond, M., Petrou, S., Carswell, C., Moher, D., Greenberg, D., Augustovski, F., Briggs, A. H., Mauskopf, J., & Loder, E. (2013) · Value in Health · doi:10.1016/j.jval.2013.02.002 · Cited by 1,998
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.004 · Cited by 1,858
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Subgroup analysis and other (mis)uses of baseline data in clinical trials
Assmann, S. F., Pocock, S. J., Enos, L. E., & Kasten, L. E. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02039-0 · Cited by 1,167
Self-citationPocock, S. J., Assmann, S. E., Enos, L. E., & Kasten, L. E. (2002) · Statistics in Medicine · doi:10.1002/sim.1296 · Cited by 1,077
Self-citationDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039
Subgroup analysis in randomised controlled trials: importance, indications, and interpretation
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17709-5 · Cited by 942
A consumer's guide to subgroup analyses
Oxman, A. D., & Guyatt, G. H. (1992) · Annals of Internal Medicine · doi:10.7326/0003-4819-116-1-78 · Cited by 914
The scandal of poor medical research.
Altman, D. G. (1994) · BMJ · doi:10.1136/bmj.308.6924.283 · Cited by 897
Subgroup analyses in randomized trials: risks of subgroup-specific analyses;
Brookes, S. T., Whitely, E., Egger, M., Smith, G. D., Mulheran, P. A., & Peters, T. J. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.08.009 · Cited by 775
Husereau, D., Drummond, M., Augustovski, F., de Bekker-Grob, E., Briggs, A. H., Carswell, C., Caulley, L., Chaiyakunapruk, N., Greenberg, D., Loder, E., et al. (2022) · Value in Health · doi:10.1016/j.jval.2021.10.008 · Cited by 847
The unit of analysis error in studies about physicians' patient care behavior
Divine, G. W., Brown, J. T., & Frazier, L. M. (1992) · Journal of General Internal Medicine · doi:10.1007/bf02599201 · Cited by 749
Als-Nielsen, B., Chen, W., Gluud, C., & Kjaergard, L. L. (2003) · JAMA · doi:10.1001/jama.290.7.921 · Cited by 746
Multiple comparisons: philosophies and illustrations
Curran-Everett, D. (2000) · American Journal of Physiology-Regulatory, Integrative and Comparative Physiology · doi:10.1152/ajpregu.2000.279.1.r1 · Cited by 743
Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2012) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2012.02.010 · Cited by 728
Hopewell, S., Clarke, M., Moher, D., Wager, E., Middleton, P., Altman, D. G., Schulz, K. F., & and the CONSORT Group (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050020 · Cited by 674
Frison, L., & Pocock, S. J. (1992) · Statistics in Medicine · doi:10.1002/sim.4780111304 · Cited by 593
Self-citationBrookes, S. T., Whitley, E., Peters, T. J., Mulheran, P. A., Egger, M., & Davey Smith, G. (2001) · Health Technology Assessment · doi:10.3310/hta5330 · Cited by 580
Statistical Power, Sample Size, and Their Reporting in Randomized Controlled Trials
Moher, D. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020048013 · Cited by 608
More informative abstracts revisited.
Haynes, R. B., Mulrow, C. D., Huth, E. J., Altman, D. G., & Gardner, M. J. (1990) · Annals of Internal Medicine · doi:10.7326/0003-4819-113-1-69 · Cited by 526
Randomisation and baseline comparisons in clinical trials
Altman, D. G., & Doré, C. J. (1990) · The Lancet · doi:10.1016/0140-6736(90)90014-v · Cited by 508
Epidemiology and reporting of randomised trials published in PubMed journals
Chan, A.-W., & Altman, D. G. (2005) · The Lancet · doi:10.1016/s0140-6736(05)71879-1 · Cited by 474
Review of survival analyses published in cancer journals
Altman, D., De Stavola, B., Love, S., & Stepniewska, K. (1995) · British Journal of Cancer · doi:10.1038/bjc.1995.364 · Cited by 438
Assessing the Quality of Randomized Controlled Trials:Current Issues and Future Directions
Moher, D., Jadad, A. R., & Tugwell, P. (1996) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300009570 · Cited by 436
Publication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427
Factors that limit the quality, number and progress of randomised controlled trials
Prescott, Counsell, Gillespie, Grant, Russell, Kiauka, Colthart, Ross, Shepherd, & Russell (1999) · Health Technology Assessment · doi:10.3310/hta3200 · Cited by 390
Gøtzsche, P. C. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90017-2 · Cited by 339
‘Spin’ in published biomedical literature: A methodological systematic review
Chiu, K., Grundy, Q., & Bero, L. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002173 · Cited by 317
Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine
Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-121-1-199407010-00003 · Cited by 324
A Proposal for Structured Reporting of Randomized Controlled Trials
Andrew, E. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520240054041 · Cited by 309
To test or not to test: Preliminary assessment of normality when comparing two independent samples
Rochon, J., Gondan, M., & Kieser, M. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-81 · Cited by 296
Publication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266
CONSORT 2025 explanation and elaboration: updated guideline for reporting randomised trials
Hopewell, S., Chan, A.-W., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081124 · Cited by 205
Pocock, S. J. (1997) · Controlled Clinical Trials · doi:10.1016/s0197-2456(97)00008-1 · Cited by 184
Pragmatic randomised controlled trials in psychiatry
Hotopf, M., Churchill, R., & Lewis, G. (1999) · British Journal of Psychiatry · doi:10.1192/bjp.175.3.217 · Cited by 165
Blinding during data analysis and writing of manuscripts
Gøtzsche, P. C. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00263-4 · Cited by 123
Sargeant, J. M., O’connor, A. M., Gardner, I. A., Dickson, J. S., Torrence, M. E., Dohoo, C. M. P. I. R., Lefebvre, S. L., Morley, P. S., Ramirez, A., & Snedeker, K. (2010) · Journal of Food Protection · doi:10.4315/0362-028x-73.3.579 · Cited by 97
Wallach, J. D., Sullivan, P. G., Trepanowski, J. F., Steyerberg, E. W., & Ioannidis, J. P. A. (2016) · BMJ · doi:10.1136/bmj.i5826 · Cited by 89
CLINICAL RESEARCH AND STATISTICAL METHODS IN THE UROLOGY LITERATURE
SCALES, C. D., NORRIS, R. D., PETERSON, B. L., PREMINGER, G. M., & DAHM, P. (2005) · Journal of Urology · doi:10.1097/01.ju.0000173640.91654.b5 · Cited by 87
Believability of relative risks and odds ratios in abstracts: Cross sectional study.
Gøtzsche, P. C. (2006) · BMJ · doi:10.1136/bmj.38895.410451.79 · Cited by 85
Kraemer, H. C. (1992) · Psychoneuroendocrinology · doi:10.1016/0306-4530(92)90013-w · Cited by 63
Toward prospective registration of clinical trials
Meinert, C. L. (1988) · Controlled Clinical Trials · doi:10.1016/0197-2456(88)90002-5 · Cited by 57
Altwairgi, A. K., Booth, C. M., Hopman, W. M., & Baetz, T. D. (2012) · Journal of Clinical Oncology · doi:10.1200/jco.2012.41.8319 · Cited by 43
Scientific rigor and the art of motorcycle maintenance
Munafò, M., Noble, S., Browne, W. J., Brunner, D., Button, K., Ferreira, J., Holmans, P., Langbehn, D., Lewis, G., Lindquist, M., et al. (2014) · Nature Biotechnology · doi:10.1038/nbt.3004 · Cited by 43
Rushton, L. (2000) · Occupational and Environmental Medicine · doi:10.1136/oem.57.1.1 · Cited by 40
Psychosocial interventions for survivors of rape and sexual assault experienced during adulthood
O'Doherty, L., Whelan, M., Carter, G. J., Brown, K., Tarzia, L., Hegarty, K., Feder, G., & Brown, S. J. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013456.pub2 · Cited by 37
Reis, A., de Geus, J., Wambier, L., Schroeder, M., & Loguercio, A. (2018) · Operative Dentistry · doi:10.2341/17-060-l · Cited by 35
Fan, J., Song, F., & Bachmann, M. O. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.12.009 · Cited by 31
LOGUERCIO, A. D., MARAN, B. M., HANZEN, T. A., PAULA, A. M. D., PERDIGÃO, J., & REIS, A. (2017) · Brazilian Oral Research · doi:10.1590/1807-3107bor-2017.vol31.0060 · Cited by 34
Trends and predictors of biomedical research quality, 1990-2015: a meta-research study
Catillon, M. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-030342 · Cited by 27
Challenges and Solutions to Pre- and Post-Randomization Subgroup Analyses
Desai, M., Pieper, K. S., & Mahaffey, K. (2014) · Current Cardiology Reports · doi:10.1007/s11886-014-0531-2 · Cited by 26
The Adequacy of Reporting Randomized, Controlled Trials in the Evaluation of Antidepressants
Streiner, D. L., & Joffe, R. (1998) · The Canadian Journal of Psychiatry · doi:10.1177/070674379804301008 · Cited by 15
Completeness of Safety Reporting in Randomized Trials: An Evaluation of 7 Medical Areas
Ioannidis, J. P. A., & Lau, J. (2001) · JAMA · doi:10.1001/jama.285.4.437 · Cited by 3
Metadata
- Authors
- Pocock, S. J., Hughes, M. D., & Lee, R. J.
- Venue
- New England Journal of Medicine
- Publisher
- Massachusetts Medical Society
- Published
- 1987-08-13
- Volume
- 317
- Issue
- 7
- Pages
- 426-432
- ISSN
- 0028-4793 (ISSN-L)
- 1533-4406
- OpenAlex
- W1977833499
- Cited by
- 662
- References
- 1
- FWCI
- 33.13
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
