Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Objective To determine whether poor reporting of methods in randomised controlled trials reflects on poor methods. Design Observational study. Setting Reports of randomised controlled trials conducted by the Radiation Therapy Oncology Group since its establishment in 1968. Participants The Radiation Therapy Oncology Group. Outcome measures Content of reports compared with the design features described in the protocols for all randomised controlled trials. Results The methodological quality of 56 randomised controlled trials was better than reported. Adequate allocation concealment was achieved in all trials but reported in only 42% of papers. An intention to treat analysis was done in 83% of trials but reported in only 69% of papers. The sample size calculation was performed in 76% of the studies, but reported in only 16% of papers. End points were clearly defined and α and βerrors were prespecified in 76% and 74% of the trials, respectively, but only reported in 10% of the papers. The one exception was the description of drop outs, where the frequency of reporting was similar to that contained in the original statistical files of the Radiation Therapy Oncology Group. Conclusions The reporting of methodological aspects of randomised controlled trials does not necessarily reflect the conduct of the trial. Reviewing research protocols and contacting trialists for more information may improve quality assessment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755
Moher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545
Systematic Reviews in Health Care : Meta-Analysis in Context
Egger, M., Smith, G. D., & Altman, D. G. (2000) · Cited by 2,401
Systematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693
Moher, D., Schulz, K. F., Altman, D. G., & for the CONSORT Group* (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00011 · Cited by 1,426
Moher, D., Jones, A., Lepage, L., & for the CONSORT Group (2001) · JAMA · doi:10.1001/jama.285.15.1992 · Cited by 1,027
Quality of Reporting of Randomized Trials as a Measure of Methodologic Quality
Huwiler-Müntener, K. (2002) · JAMA · doi:10.1001/jama.287.21.2801 · Cited by 397
Systematic Reviews in Health Care
Loo, L. (2002) · Annals of Internal Medicine · doi:10.7326/0003-4819-137-7-200210010-00035 · Cited by 254
Research works printed but not linked: 2
- Clarke M Oxman A eds. Cochrane reviewers' handbook 4.2.0 [updated March 2003]. In Cochrane Library, issue 2. Oxford: Update Software, 2003.
- 10.1016/0197-2456(80)90013-6 · doi:10.1016/0197-2456(80)90013-6
Cited by
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
“(189, 190) that the report may not be a reasonable facsimile of the”
BackgroundSelf-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Italian Journal of Public Health · doi:10.2427/5768 · Cited by 3,374
Self-citationSystematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Interventions for preventing obesity in children
Brown, T., Moore, T. H., Hooper, L., Gao, Y., Zayegh, A., Ijaz, S., Elwenspoek, M., Foxen, S. C., Magee, L., O'Malley, C., et al. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001871.pub4 · Cited by 2,982
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments
Lo, C. K.-L., Mertz, D., & Loeb, M. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-45 · Cited by 2,477
“One possible explanation might be that NOS allows for more subjectivity than when assessing RCTs; lack of information in RCTs automatically results in higher risk of bias, whereas NOS requires the reviewer to decide subjectively on the risk of bias for each item based on the information available in the report.”
BackgroundInfluentialChan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
“Parallel-group 68 38 (4-78) 57 50 (0-100)”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546
Speech and language therapy for aphasia following stroke
Brady, M. C., Kelly, H., Godwin, J., Enderby, P., & Campbell, P. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000425.pub4 · Cited by 1,159
Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group (2013) · PLoS ONE · doi:10.1371/journal.pone.0066844 · Cited by 1,127
Savović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049
Savović, J., Jones, H., Altman, D., Harris, R., Jűni, P., Pildal, J., Als-Nielsen, B., Balk, E., Gluud, C., Gluud, L., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16350 · Cited by 668
“Blinding All 71 (36) 725 290 (40) 33 (17) 286 84 (29) Mortality 13 (5) 119 39 (33) 5 (3) 48 10 (21) Objective 20 (12) 208 81 (39) 9 (5) 63 23 (37) Subjective 38 (19) 398 170 (43) 19 (9) 175 51 (29)”
Bias in clinical intervention research.
Gluud, L. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj069 · Cited by 636
Griffin, S. J. (2004) · The Annals of Family Medicine · doi:10.1370/afm.142 · Cited by 578
Burr, J., Mowatt, G., Hernández, R., Siddiqui, M., Cook, J., Lourenco, T., Ramsay, C., Vale, L., Fraser, C., Azuara-Blanco, A., et al. (2007) · Health Technology Assessment · doi:10.3310/hta11410 · Cited by 549
Hopewell, S., Dutton, S., Yu, L.-M., Chan, A.-W., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c723 · Cited by 504
“Finally, it is important to recognise that poor reporting does not necessarily mean poor methods. Several studies”
BackgroundCheng, D. C., Bainbridge, D., Martin, J. E., & Novick, R. J. (2005) · Anesthesiology · doi:10.1097/00000542-200501000-00028 · Cited by 436
Bower, P., Gilbody, S., Richards, D., Fletcher, J., & Sutton, A. (2006) · British Journal of Psychiatry · doi:10.1192/bjp.bp.106.023655 · Cited by 429
Fostering integrity in research: Definitions, current knowledge, and future directions
Steneck, N. H. (2006) · Science and Engineering Ethics · doi:10.1007/pl00022268 · Cited by 407
Navigated Total Knee Replacement: A Meta-Analysis
Bauwens, K., Matthes, G., Wich, M., Gebhard, F., Hanson, B., Ekkernkamp, A., & Stengel, D. (2007) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200702000-00005 · Cited by 399
Dyadic coping and relationship functioning in couples coping with cancer: A systematic review
Traa, M. J., De Vries, J., Bodenmann, G., & Den Oudsten, B. L. (2015) · British Journal of Health Psychology · doi:10.1111/bjhp.12094 · Cited by 367
CEC’s Standards for Classifying the Evidence Base of Practices in Special Education
Cook, B. G., Buysse, V., Klingner, J., Landrum, T. J., McWilliam, R. A., Tankersley, M., & Test, D. W. (2015) · Remedial and Special Education · doi:10.1177/0741932514557271 · Cited by 361
Speech and language therapy for aphasia following stroke
Brady, M. C., Kelly, H., Godwin, J., & Enderby, P. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000425.pub3 · Cited by 269
A case for a duty to feed the hungry: GM plants and the third world
Carter, L. (2007) · Science and Engineering Ethics · doi:10.1007/s11948-006-0006-y · Cited by 265
Mills, E., Wu, P., Seely, D., & Guyatt, G. (2005) · Journal of Pineal Research · doi:10.1111/j.1600-079x.2005.00258.x · Cited by 256
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Burgi, E., Scherer, M., Altman, D. G., & Juni, P. (2009) · BMJ · doi:10.1136/bmj.b3244 · Cited by 248
Comparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243
Pildal, J., Chan, A.-W., Hróbjartsson, A., Forfang, E., Altman, D. G., & Gøtzsche, P. C. (2005) · BMJ · doi:10.1136/bmj.38414.422650.8f · Cited by 209
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207
Chan, A.-W., Hrobjartsson, A., Jorgensen, K. J., Gotzsche, P. C., & Altman, D. G. (2008) · BMJ · doi:10.1136/bmj.a2299 · Cited by 202
Dwan, K., Altman, D. G., Clarke, M., Gamble, C., Higgins, J. P. T., Sterne, J. A. C., Williamson, P. R., & Kirkham, J. J. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001666 · Cited by 187
“Five studies considered whether an intention-totreat analysis or per protocol analysis was specified or reported [6,20,23,25,28].”
BackgroundInfluentialSelf-citationDechartres, A., Trinquart, L., Atal, I., Moher, D., Dickersin, K., Boutron, I., Perrodeau, E., Altman, D. G., & Ravaud, P. (2017) · BMJ · doi:10.1136/bmj.j2490 · Cited by 171
Changes in attachment representations during psychological therapy
Taylor, P., Rietzschel, J., Danquah, A., & Berry, K. (2015) · Psychotherapy Research · doi:10.1080/10503307.2014.886791 · Cited by 150
Bias, Spin, and Misreporting: Time for Full Access to Trial Protocols and Results
Chan, A.-W. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050230 · Cited by 144
The relationship between dissociation and voices: A systematic literature review and meta-analysis
Pilton, M., Varese, F., Berry, K., & Bucci, S. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.06.004 · Cited by 141
Bafeta, A., Dechartres, A., Trinquart, L., Yavchitz, A., Boutron, I., & Ravaud, P. (2012) · BMJ · doi:10.1136/bmj.e813 · Cited by 129
Mhaskar, R., Djulbegovic, B., Magazin, A., Soares, H. P., & Kumar, A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.016 · Cited by 102
Self-citationArmijo-Olivo, S., Fuentes, J., Ospina, M., Saltaji, H., & Hartling, L. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-116 · Cited by 60
Are randomized trials conducted in China or India biased? A comparative empirical analysis
Zhang, D., Freemantle, N., & Cheng, K. K. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.010 · Cited by 51
“[1] found that authors of RCTs frequently used allocation concealment and blinding but failed to report these methods.”
BackgroundBüttner, F., Winters, M., Delahunt, E., Elbers, R., Lura, C. B., Khan, K. M., Weir, A., & Ardern, C. L. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-100806 · Cited by 47
Reporting of analyses from randomized controlled trials with multiple arms: a systematic review
Baron, G., Perrodeau, E., Boutron, I., & Ravaud, P. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-84 · Cited by 39
“Second, the methods may have been pre-specified but not reported in the articles [20,21].”
MethodTARG Meta-Research Group & Collaborators (2023) · BMJ Open · doi:10.1136/bmjopen-2023-076264 · Cited by 33
Liu, X., Zhang, X., Wen, S., Peng, L., Hong, Q., & Kang, D. (2015) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12173 · Cited by 22
Husson, M., Dechartres, A., Ramdjee, B., Diverres, M., Zejli, T., L'Hénaff, M., Jehanno, E., Tubach, F., & Caille, A. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.06.012 · Cited by 20
Koletsi, D., Pandis, N., Polychronopoulou, A., & Eliades, T. (2012) · Journal of Evidence Based Dental Practice · doi:10.1016/j.jebdp.2012.05.002 · Cited by 18
“Classification of clinical trials was based on reporting only; however, lack of information on the published article does not necessarily mean that correct trial procedures were not implemented.(34) It is possible that a study that followed appropriate randomization methodology failed to be classified as an RCT becaus…”
BackgroundThe quality of reporting of randomised controlled trials in asthma: a systematic review
Ntala, C., Birmpili, P., Worth, A., Anderson, N. H., & Sheikh, A. (2013) · Primary Care Respiratory Journal · doi:10.4104/pcrj.2013.00089 · Cited by 15
Allocation concealment: a methodological review
Clark, L., Schmidt, U., Tharmanathan, P., Adamson, J., Hewitt, C., & Torgerson, D. (2013) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12032 · Cited by 12
Data sharing and transparency indicators in published RCTs in Oral Health between 2017 and 2023
Mikelis, F., Karamalaki, D., Mikeli, A., Tzanetakis, G. N., & Koletsi, D. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105263 · Cited by 8
Metadata
- Authors
- Soares, H. P., Daniels, S., Kumar, A., Clarke, M., Scott, C., Swann, S., & Djulbegovic, B.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2004-01-01
- Volume
- 328
- Issue
- 7430
- Pages
- 22-24
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- OpenAlex
- W2116996240
- Cited by
- 313
- References
- 13
- FWCI
- 27.29
- Type
- journal article
- Language
- en
- Open access
- bronze
- Licence
- www.bmj.com/company/legal-information/terms-conditions/legal-information/tdm-licencepolicy (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.