Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols
Open accessAbstract
Objectives To assess whether the reported methodological quality of randomized controlled trials (RCTs) reflects the actual methodological quality and to evaluate the association of effect size (ES) and sample size with methodological quality.Study design and setting Systematic review. This is a retrospective analysis of all consecutive phase III RCTs published by eight National Cancer Institute Cooperative Groups up to 2006. Data were extracted from protocols (actual quality) and publications (reported quality) for each study.Results Four hundred twenty-nine RCTs met the inclusion criteria. Overall reporting of methodological quality was poor and did not reflect the actual high methodological quality of RCTs. The results showed no association between sample size and actual methodological quality of a trial. Poor reporting of allocation concealment and blinding exaggerated the ES by 6% (ratio of hazard ratio [RHR]: 0.94; 95% confidence interval [CI]: 0.88, 0.99) and 24% (RHR: 1.24; 95% CI: 1.05, 1.43), respectively. However, actual quality assessment showed no association between ES and methodological quality.Conclusion The largest study to date shows that poor quality of reporting does not reflect the actual high methodological quality. Assessment of the impact of quality on the ES based on reported quality can produce misleading results.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The method of sequence generation was not described in three quarters of randomised trial protocols approved by a research ethics committee in Denmark in 1994-95, and a US cooperative cancer research group in 1968-2006.
“The method of sequence generation was not described in three quarters of randomised trial protocols approved by a research ethics committee in Denmark in 1994-95, and a US cooperative cancer research group in 1968-2006.329 330”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [330]
Central randomisation was stated as the allocation concealment method in all phase 3 trial protocols initiated in 1968-2003 by a cooperative cancer research group that used extensive protocol review processes.
“In contrast, central randomisation was stated as the allocation concealment method in all phase 3 trial protocols initiated in 1968-2003 by a cooperative cancer research group that used extensive protocol review processes.330”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [330]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371 · see citing works
Practical Nonparametric Statistics
Groggel, D. J., & Conover, W. J. (2000) · Technometrics · doi:10.2307/1271101 · Cited by 11,071 · see citing works
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329 · see citing works
Practical methods for incorporating summary time-to-event data into meta-analysis.
Tierney, J. F., Stewart, L. A., Ghersi, D., Burdett, S., & Sydes, M. R. (2007) · Trials · doi:10.1186/1745-6215-8-16 · Cited by 6,208 · see citing works
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · see citing works
Fundamentals of Biostatistics.
(1969) · Annals of Internal Medicine · doi:10.7326/0003-4819-70-4-873 · Cited by 4,573 · see citing works
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560 · see citing works
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 1× · see citing works
“[20] Allocation concealment has shown the most consistent associations with treatment effect sizes.[21] However, all of these studies have assessed the methodological quality of reporting only.”
BackgroundImproving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127 · see citing works
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538 · see citing works
Systematic Reviews in Health Care : Meta-Analysis in Context
Egger, M., Smith, G. D., & Altman, D. G. (2000) · Cited by 2,312 · see citing works
Gluud, L. L., Villumsen, J., & Gluud, C. N. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,262 · see citing works
CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., & Moher, D. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72352 · Cited by 2,165 · see citing works
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · Trials · doi:10.1186/1745-6215-11-32 · Cited by 1,871 · see citing works
Moher, D., Jones, A., Lepage, L., & for the CONSORT Group (2001) · JAMA · doi:10.1001/jama.285.15.1992 · Cited by 1,027 · see citing works
Finding What Works in Health Care: Standards for Systematic Reviews
Eden, J., Levit, L. A., Berg, A. O., & Morton, S. C. (2013) · Cited by 943 · see citing works
Publication bias: the case for an international registry of clinical trials.
Simes, R. J. (1986) · Journal of Clinical Oncology · doi:10.1200/jco.1986.4.10.1529 · Cited by 747 · see citing works
Practical Nonparametric Statistics, 3rd Ed.
Borror, C. M. (2001) · Journal of Quality Technology · doi:10.1080/00224065.2001.11980074 · Cited by 588 · see citing works
How study design affects outcomes in comparisons of therapy. I: Medical
Colditz, G. A., Miller, J. N., & Mosteller, F. C. (1989) · Statistics in Medicine · doi:10.1002/sim.4780080408 · Cited by 520 · see citing works
Quality of Reporting of Randomized Trials as a Measure of Methodologic Quality
Huwiler-Müntener, K. (2002) · JAMA · doi:10.1001/jama.287.21.2801 · Cited by 397 · see citing works
Soares, H. P., Daniels, S., Kumar, A., Clarke, M. J., Scott, C., Swann, S., & Djulbegović, B. (2004) · BMJ · doi:10.1136/bmj.328.7430.22 · Cited by 310 · see citing works
Self-citationDevereaux, P. J., Choi, P. T.-L., El-Dika, S., Bhandari, M., Montori, V. M., Schünemann, H. J., Garg, A. X., Busse, J. W., Heels-Ansdell, D., Ghali, W. A., et al. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.03.017 · Cited by 209 · see citing works
A quality assessment of randomized control trials of primary treatment of breast cancer.
Liberati, A., Himel, H. N., & Chalmers, T. C. (1986) · Journal of Clinical Oncology · doi:10.1200/jco.1986.4.6.942 · Cited by 195 · see citing works
Randomized clinical trials in Hepatology: Predictors of quality
Gluud, L. L., Nikolova, D., & Gluud, C. N. (1999) · Hepatology · doi:10.1002/hep.510300510 · Cited by 115 · see citing works
Discrepancy between published report and actual conduct of randomized clinical trials
Hill, C. L., LaValley, M. P., & Felson, D. T. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00440-7 · Cited by 111 · see citing works
Time to register randomised trials
Horton, R., & Smith, R. W. (1999) · BMJ · doi:10.1136/bmj.319.7214.865 · Cited by 91 · see citing works
Tonks, A. (1999) · BMJ · doi:10.1136/bmj.319.7224.1565 · Cited by 70 · see citing works
Publishing study protocols: making them visible will improve registration, reporting and recruitment
Godlee, F. (2001) · BMC Meeting Abstracts · doi:10.1186/1471-8219-2-4 · Cited by 63 · see citing works
Trial sample size, but not trial quality, is associated with positive study outcome
Singh, J. A., Murphy, S., & Bhandari, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.05.007 · Cited by 17 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
SPIRIT 2013 Statement: Defining Standard Protocol Items for Clinical Trials
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., et al. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-3-201302050-00583 · Cited by 7,363 · see citing works
SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251 · see citing works
Reducing waste from incomplete or unusable reports of biomedical research
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D., & Wager, E. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62228-x · Cited by 1,319 · see citing works
Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · see citing works
Page, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0159267 · Cited by 284 · see citing works
Corbett, M. S., Higgins, J. P. T., & Woolacott, N. F. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1090 · Cited by 273 · see citing works
Dechartres, A., Trinquart, L., Atal, I., Moher, D., Dickersin, K., Boutron, I., Perrodeau, E., Altman, D. G., & Ravaud, P. (2017) · BMJ · doi:10.1136/bmj.j2490 · Cited by 171 · see citing works
SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171 · see citing works
Rafiq, S., Campodonico, C., & Varese, F. (2018) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12969 · Cited by 108 · see citing works
Empirical evaluation of which trial characteristics are associated with treatment effect estimates
Dechartres, A., Trinquart, L., Faber, T., & Ravaud, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.005 · Cited by 85 · see citing works
Review and publication of protocol submissions to Trials – what have we learned in 10 years?
Li, T., Boutron, I., Salman, R. A.-S., Cobo, E., Flemyng, E., Grimshaw, J. M., & Altman, D. G. (2017) · Trials · doi:10.1186/s13063-016-1743-0 · Cited by 66 · see citing works
Saltaji, H., Armijo-Olivo, S., Cummings, G. G., Amin, M., da Costa, B. R., & Flores-Mir, C. (2018) · Journal of Dental Research · doi:10.1177/0022034517725049 · Cited by 40 · see citing works
Tan, Z. W., Tan, A. C., Li, T., Harris, I., Naylor, J. M., Siebelt, M., van Tiel, J., Pinheiro, M., Harris, L., Chamberlain, K., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-038283 · Cited by 37 · see citing works
TARG Meta-Research Group & Collaborators (2023) · BMJ Open · doi:10.1136/bmjopen-2023-076264 · Cited by 33 · see citing works
Le Fourn, E., Giraudeau, B., Chosidow, O., Doutre, M.-S., & Lorette, G. (2013) · PLoS ONE · doi:10.1371/journal.pone.0070717 · Cited by 19 · see citing works
da Rosa Oliveira, L., Elagami, R. A., Reis, T. M., Tedesco, T. K., Mendes, F. M., Braga, M. M., Pannuti, C. M., & Raggio, D. P. (2024) · Caries Research · doi:10.1159/000542108 · Cited by 3 · see citing works
- Authors
- Mhaskar, R., Djulbegovic, B., Magazin, A., Soares, H. P., & Kumar, A.
- Affiliations
- University of South Florida (US)
- Moffitt Cancer Center (US)
- Corresponding author
- Kumar, Ambuj
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2012-06
- Volume
- 65
- Issue
- 6
- Pages
- 602-609
- ISSN
- 0895-4356 (ISSN-L)
- PMC
- PMC3637913
- OpenAlex
- W2014234827
- Cited by
- 102
- References
- 30
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 5 · 2023: 3 · 2022: 16 · 2021: 9 · 2020: 4 · 2019: 9 · 2018: 10 · 2017: 11 · 2016: 8 · 2015: 9 · 2014: 4 · 2013: 8 · 2012: 1
- FWCI
- 8.00
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (5r01ns044417-02)
- National Institute of Neurological Disorders and Stroke (R01NS044417, R01NS052956)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- methodological quality, randomized controlled trials, trial reporting quality, allocation concealment, blinding, effect size, sample size, cancer clinical trials
- MeSH terms
- Systematic Reviews as Topic, Algorithms, Clinical Protocols, Humans, Periodicals as Topic, Research Design, Bias, Confidence Intervals, Odds Ratio, Randomized Controlled Trials as Topic, Sample Size, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
