Evaluating the Evidence: Statistical Methods in Randomized Controlled Trials in the Urological Literature
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
PURPOSE: Randomized controlled trials potentially provide the highest level of evidence to inform clinical decision making. Appropriate use of statistical methods is a critical aspect of all clinical research, including randomized controlled trials. We report the first formal evaluation to our knowledge of the statistical methods of randomized controlled trials published in the urological literature in 1996 and 2004. MATERIALS AND METHODS: All human subjects randomized controlled trials published in 4 leading urology journals in 1996 and 2004 were identified for formal review. A standardized evaluation form was developed based on the Consolidated Standards of Reporting Trials statement. Each article was evaluated by 2 independent reviewers with formal training in research design and biostatistics who were blinded to study authors and institution. Discrepancies were settled by consensus. RESULTS: A total of 152 randomized controlled trials were reviewed (65 in 1996, 87 in 2004). The median sample size (IQR) per arm of parallel design randomized controlled trials published in 1996 and 2004 was 36 (11, 96) and 50 (26, 134) study subjects, respectively (p = 0.157). Sample size justifications were provided by 19% of studies in 1996 and 47% of studies in 2004 (p = 0.001). Of randomized controlled trials 16 (25%) vs 32 (37%) identified a single primary outcome variable (p = 0.110). Effect size estimates for primary or secondary outcome variables were provided by 5% vs 13% (p = 0.090) and the precision of the effect was detailed by 5% vs 10% of randomized controlled trials (p = 0.195). CONCLUSIONS: This formal review suggests that statistical analysis in urological randomized controlled trials has improved. However, considerable deficiencies remain. Ongoing education in applied statistics may further improve urological randomized controlled trial reporting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
What is meant by intention to treat analysis? Survey of published randomised controlled trials
Hollis, S., & Campbell, F. (1999) · BMJ · doi:10.1136/bmj.319.7211.670 · Cited by 2,160
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Statistical Considerations in the Intent-to-Treat Principle
Lachin, J. M. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00046-5 · Cited by 652
Statistical Power, Sample Size, and Their Reporting in Randomized Controlled Trials
Moher, D. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020048013 · Cited by 608
Balk, E. M. (2002) · JAMA · doi:10.1001/jama.287.22.2973 · Cited by 478
Gøtzsche, P. C. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90017-2 · Cited by 339
CLINICAL RESEARCH AND STATISTICAL METHODS IN THE UROLOGY LITERATURE
SCALES, C. D., NORRIS, R. D., PETERSON, B. L., PREMINGER, G. M., & DAHM, P. (2005) · Journal of Urology · doi:10.1097/01.ju.0000173640.91654.b5 · Cited by 87
Self-citation
Research works printed but not linked: 13
- Sackett, 1996, Evidence based medicine: what it is and what it isn't, BMJ, 312, 71, 10.1136/bmj.312.7023.71 · DOI · Paper page
- Scales, 2007, Evidence based clinical practice: a primer for urologists, J Urol, 178, 775, 10.1016/j.juro.2007.05.020 · DOI
- Begg, 1996, Improving the quality of reporting of randomized controlled trials, JAMA, 276, 637, 10.1001/jama.276.8.637 · doi:10.1001/jama.276.8.637 · Paper page
- Moher, 2001, The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials, Lancet, 357, 1191, 10.1016/S0140-6736(00)04337-3 · DOI · Paper page
- Schulz, 1995, Empirical evidence of bias, JAMA, 273, 408, 10.1001/jama.273.5.408 · doi:10.1001/jama.273.5.408
- Hutton, 2000, Bias in meta-analysis due to outcome variable selection within studies, Appl Stat, 49, 359
- Ruiz-Canela, 2000, Intention to treat analysis is related to methodological quality, BMJ, 320, 1007, 10.1136/bmj.320.7240.1007 · DOI · Paper page
- Hall, 1996, Methodologic standards in surgical trials, Surgery, 119, 466, 10.1016/S0039-6060(96)80149-8 · DOI
- Assmann, 2000, Subgroup analysis and other (mis)uses of baseline data in clinical trials, Lancet, 355, 1064, 10.1016/S0140-6736(00)02039-0 · DOI · Paper page
- Breau, 2006, Inadequate statistical power of negative clinical trials in urological literature, J Urol, 176, 263, 10.1016/S0022-5347(06)00505-2 · DOI · Paper page
- Welk, 2006, Randomized controlled trials in pediatric urology: room for improvement, J Urol, 176, 306, 10.1016/S0022-5347(06)00560-X · DOI
- Scales, 2007, A critical assessment of the quality of reporting of randomized, controlled trials in the urology literature, J Urol, 177, 1090, 10.1016/j.juro.2006.10.027 · DOI
- Altman, 2001, The revised CONSORT statement for reporting randomized trials: explanation and elaboration, Ann Intern Med, 134, 663, 10.7326/0003-4819-134-8-200104170-00012 · DOI · Paper page
Cited by
Guidance for Developers of Health Research Reporting Guidelines
Moher, D., Schulz, K. F., Simera, I., & Altman, D. G. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000217 · Cited by 1,437
“This is just one of numerous examples of a large and disturbing literature indicating the general failure in the quality of reporting health research [2–6].”
BackgroundHopewell, S., Dutton, S., Yu, L.-M., Chan, A.-W., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c723 · Cited by 504
Statistical Analyses and Methods in the Published Literature: The SAMPL Guidelines*
Lang, T. A., & Altman, D. G. (2014) · Guidelines for Reporting Health Research: A User's Manual · doi:10.1002/9781118715598.ch25 · Cited by 57
Braga, L. H., Pemberton, J., DeMaria, J., & Lorenzo, A. J. (2011) · Journal of Urology · doi:10.1016/j.juro.2011.03.044 · Cited by 50
Liu, X., Zhang, X., Wen, S., Peng, L., Hong, Q., & Kang, D. (2015) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12173 · Cited by 22
Metadata
- Authors
- Scales, C. D., Norris, R. D., Preminger, G. M., Vieweg, J., Peterson, B. L., & Dahm, P.
- Venue
- Journal of Urology
- Published
- 2008-10
- Volume
- 180
- Issue
- 4
- Pages
- 1463-1467
- ISSN
- 0022-5347 (ISSN-L)
- 1527-3792
- OpenAlex
- W1977319231
- Cited by
- 32
- References
- 7
- FWCI
- 4.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.