Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists
Open accessAbstract
OBJECTIVES: To provide information on the frequency and reasons for outcome reporting bias in clinical trials. DESIGN: Trial protocols were compared with subsequent publication(s) to identify any discrepancies in the outcomes reported, and telephone interviews were conducted with the respective trialists to investigate more extensively the reporting of the research and the issue of unreported outcomes. PARTICIPANTS: Chief investigators, or lead or coauthors of trials, were identified from two sources: trials published since 2002 covered in Cochrane systematic reviews where at least one trial analysed was suspected of being at risk of outcome reporting bias (issue 4, 2006; issue 1, 2007, and issue 2, 2007 of the Cochrane library); and a random sample of trial reports indexed on PubMed between August 2007 and July 2008. SETTING: Australia, Canada, Germany, the Netherlands, New Zealand, the United Kingdom, and the United States. MAIN OUTCOME MEASURES: Frequency of incomplete outcome reporting-signified by outcomes that were specified in a trial's protocol but not fully reported in subsequent publications-and trialists' reasons for incomplete reporting of outcomes. RESULTS: 268 trials were identified for inclusion (183 from the cohort of Cochrane systematic reviews and 85 from PubMed). Initially, 161 respective investigators responded to our requests for interview, 130 (81%) of whom agreed to be interviewed. However, failure to achieve subsequent contact, obtain a copy of the study protocol, or both meant that final interviews were conducted with 59 (37%) of the 161 trialists. Sixteen trial investigators failed to report analysed outcomes at the time of the primary publication, 17 trialists collected outcome data that were subsequently not analysed, and five trialists did not measure a prespecified outcome over the course of the trial. In almost all trials in which prespecified outcomes had been analysed but not reported (15/16, 94%), this under-reporting resulted in bias. In nearly a quarter of trials in which prespecified outcomes had been measured but not analysed (4/17, 24%), the "direction" of the main findings influenced the investigators' decision not to analyse the remaining data collected. In 14 (67%) of the 21 randomly selected PubMed trials, there was at least one unreported efficacy or harm outcome. More than a quarter (6/21, 29%) of these trials were found to have displayed outcome reporting bias. CONCLUSION: The prevalence of incomplete outcome reporting is high. Trialists seemed generally unaware of the implications for the evidence base of not reporting all outcomes and protocol changes. A general lack of consensus regarding the choice of outcomes in particular clinical settings was evident and affects trial design, conduct, analysis, and reporting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Problems arise when trial protocols do not designate any primary outcome.
“Problems also arise when trial protocols do not designate any primary outcome, as seen in half of protocols for a sample of trials published from 2002 to 2008,254 and in 18% and 25% of randomised trial protocols that received ethics approval in Switzerland and Denmark, respectively.249 255”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [254]
Half of protocols for a sample of trials published from 2002 to 2008 did not designate any primary outcome.
“as seen in half of protocols for a sample of trials published from 2002 to 2008,254”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [254]
Numerous studies have shown substantive changes between prespecified methods and those reported in trial publications.
“Numerous studies have shown substantive changes between prespecified methods (eg, as stated in approved protocols, registries, or regulatory agency submissions) and those reported in trial publications, including changes to primary outcomes,254 256 257 475 sample size calculations,200 eligibility criteria,215 216 476 and interventions,477 as well as methods of allocation concealment,329 blinding,5”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [254]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · see citing works
Self-citationCochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670 · see citing works
Selective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435 · see citing works
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1× · see citing works
“The strength of our study rests in its use of interviews to provide valuable insight into the decision making process from the planning to the publication stage of clinical trials. Our study adds to the growing body of empirical evidence on research behaviour.”
BackgroundMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.004 · Cited by 1,858 · see citing works
Self-citationChan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · see citing works
Self-citationSystematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · see citing works
Self-citationKirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133 · see citing works
Self-citationMartinson, B. C., Anderson, M. S., & de Vries, R. (2005) · Nature · doi:10.1038/435737a · Cited by 1,096 · see citing works
Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials
Mathieu, S. (2009) · JAMA · doi:10.1001/jama.2009.1242 · Cited by 730 · see citing works
Publication bias and clinical trials
Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987) · Controlled Clinical Trials · doi:10.1016/0197-2456(87)90155-3 · Cited by 715 · see citing works
Underreporting Research Is Scientific Misconduct
Chalmers, I. (1990) · JAMA · doi:10.1001/jama.263.10.1405 · Cited by 634 · see citing works
Chan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613 · cited 1× · see citing works
“A more recent retrospective review of publications and follow-up survey of trial investigators confirmed these reasons for non-reporting.”
BackgroundSelf-citationStandardising outcomes for clinical trials and systematic reviews
Clarke, M. (2007) · Trials · doi:10.1186/1745-6215-8-39 · Cited by 483 · see citing works
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Chan, A.-W. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041086 · Cited by 444 · see citing works
Trial Publication after Registration in ClinicalTrials.Gov: A Cross-Sectional Analysis
Ross, J. S., Mulvey, G. K., Hines, E. M., Nissen, S. E., & Krumholz, H. M. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000144 · Cited by 401 · see citing works
Outcome Reporting Among Drug Trials Registered in ClinicalTrials.gov
Bourgeois, F. T., Murthy, S., & Mandl, K. D. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-153-3-201008030-00006 · Cited by 332 · see citing works
Adverse Events in Randomized Trials
Ioannidis, J. P. A. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.313 · Cited by 243 · see citing works
Bias in meta‐analysis due to outcome variable selection within studies
Hutton, J. L., & Williamson, P. (2000) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/1467-9876.00197 · Cited by 236 · see citing works
Self-citationSinha, I. P., Jones, L. V., Smyth, R. L., & Williamson, P. R. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050096 · Cited by 202 · see citing works
Self-citationOutcome selection bias in meta-analysis
Williamson, P. R., Gamble, C., Altman, D. G., & Hutton, J. L. (2005) · Statistical Methods in Medical Research · doi:10.1191/0962280205sm415oa · Cited by 188 · see citing works
Self-citationSelective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy
THIERS, B. H. (2009) · Yearbook of Dermatology and Dermatologic Surgery · doi:10.1016/s0093-3619(08)79042-7 · Cited by 157 · see citing works
Outcome measurement in clinical trials for ulcerative colitis: towards standardisation
Cooney, R. M., Warren, B. F., Altman, D. G., Abreu, M. T., & Travis, S. P. L. (2007) · Trials · doi:10.1186/1745-6215-8-17 · Cited by 128 · see citing works
Self-citationGiles, J. (2004) · Nature · doi:10.1038/news040301-9 · Cited by 123 · see citing works
Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet
Al-Marzouki, S., Roberts, I., Evans, S., & Marshall, T. (2008) · The Lancet · doi:10.1016/s0140-6736(08)61060-0 · Cited by 119 · see citing works
Blümle, A., Huwiler, K., & Witschi, M. (2008) · Swiss Medical Weekly · doi:10.4414/smw.2008.12027 · Cited by 112 · see citing works
Hahn, S., Williamson, P. R., & Hutton, J. L. (2002) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2002.00314.x · Cited by 97 · see citing works
Self-citationReporting research results: a moral obligation for all researchers.
Moher, D. (2007) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/bf03022653 · Cited by 60 · cited 1× · see citing works
“Researchers have a moral responsibility to report trial findings completely and transparently.”
BackgroundUndisclosed Changes in Outcomes in Randomized Controlled Trials: An Observational Study
Ewart, R. M., Lausen, H., & Millian, N. (2009) · The Annals of Family Medicine · doi:10.1370/afm.1017 · Cited by 50 · see citing works
Waiving confidentiality for the greater good
Chan, A., Upshur, R. E. G., Singh, J. A., Ghersi, D., Chapuis, F. R., & Altman, D. G. (2006) · BMJ · doi:10.1136/bmj.332.7549.1086 · Cited by 46 · see citing works
Self-citationFanelli, D. (2009) · PsycEXTRA Dataset · doi:10.1037/e521122012-010 · Cited by 38 · see citing works
Ghersi, D. (2006) · The Sydney eScholarship Repository (The University of Sydney) · Cited by 16 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
SPIRIT 2013 Statement: Defining Standard Protocol Items for Clinical Trials
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., et al. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-3-201302050-00583 · Cited by 7,363 · see citing works
Self-citationSPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251 · see citing works
Self-citationPublication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
Faller, H., Schuler, M., Richard, M., Heckl, U., Weis, J., & Küffner, R. (2013) · Journal of Clinical Oncology · doi:10.1200/jco.2011.40.8922 · Cited by 912 · see citing works
Why clinical trial outcomes fail to translate into benefits for patients
Heneghan, C., Goldacre, B., & Mahtani, K. R. (2017) · Trials · doi:10.1186/s13063-017-1870-2 · Cited by 404 · see citing works
Systematic review of knowledge translation strategies in the allied health professions
Scott, S. D., Albrecht, L., O’Leary, K., Ball, G. D., Hartling, L., Hofmeyer, A., Jones, C. A., Klassen, T. P., Burns, K. K., Newton, A. S., et al. (2012) · Implementation Science · doi:10.1186/1748-5908-7-70 · Cited by 331 · see citing works
SPIRIT 2013: new guidance for content of clinical trial protocols
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Dickersin, K., & Moher, D. (2013) · The Lancet · doi:10.1016/s0140-6736(12)62160-6 · Cited by 298 · see citing works
Self-citationComparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243 · see citing works
Self-citationTao, W.-W., Jiang, H., Tao, X.-M., Jiang, P., Sha, L.-Y., & Sun, X.-C. (2016) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2015.11.027 · Cited by 233 · see citing works
Saini, P., Loke, Y. K., Gamble, C., Altman, D. G., Williamson, P. R., & Kirkham, J. J. (2014) · BMJ · doi:10.1136/bmj.g6501 · Cited by 209 · see citing works
Self-citationDwan, K., Altman, D. G., Clarke, M., Gamble, C., Higgins, J. P. T., Sterne, J. A. C., Williamson, P. R., & Kirkham, J. J. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001666 · Cited by 187 · see citing works
Self-citationBias Due to Changes in Specified Outcomes during the Systematic Review Process
Kirkham, J. J., Altman, D. G., & Williamson, P. R. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009810 · Cited by 164 · see citing works
Self-citationA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165 · see citing works
SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171 · see citing works
Lasikiewicz, N., Myrissa, K., Hoyland, A., & Lawton, C. L. (2014) · Appetite · doi:10.1016/j.appet.2013.09.017 · Cited by 144 · see citing works
Wager, E., Williams, P., & on behalf of the OPEN Project (Overcome failure to Publish nEgative fiNdings) Consortium (2013) · BMJ · doi:10.1136/bmj.f5248 · Cited by 139 · see citing works
All trials must be registered and the results published
Chalmers, I., Glasziou, P., & Godlee, F. (2013) · BMJ · doi:10.1136/bmj.f105 · Cited by 131 · see citing works
Hodkinson, A., Kirkham, J. J., Tudur-Smith, C., & Gamble, C. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003436 · Cited by 111 · see citing works
Self-citationVera-Badillo, F. E., Shapiro, R., Ocana, A., Amir, E., & Tannock, I. F. (2013) · Annals of Oncology · doi:10.1093/annonc/mds636 · Cited by 117 · see citing works
Launbo, N., Zobel, E. H., von Scholten, B. J., Færch, K., Jørgensen, P. G., & Christensen, R. H. (2021) · Obesity Reviews · doi:10.1111/obr.13136 · Cited by 76 · see citing works
Harms in Systematic Reviews Paper 1: An introduction to research on harms
Qureshi, R., Mayo-Wilson, E., & Li, T. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.10.023 · Cited by 63 · see citing works
Rasmussen, K., Bero, L., Redberg, R., Gøtzsche, P. C., & Lundh, A. (2018) · BMJ · doi:10.1136/bmj.k3654 · Cited by 74 · see citing works
Out of sight but not out of mind: how to search for unpublished clinical trial evidence
Chan, A.-W. (2012) · BMJ · doi:10.1136/bmj.d8013 · Cited by 62 · see citing works
Bashir, R., Bourgeois, F. T., & Dunn, A. G. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0518-3 · Cited by 47 · see citing works
Fuller, T., Pearson, M., Peters, J., & Anderson, R. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121585 · Cited by 47 · see citing works
Publication bias in clinical trials of electronic health records
Vawdrey, D. K., & Hripcsak, G. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2012.08.007 · Cited by 38 · see citing works
Selective outcome reporting in randomized clinical trials of dental implants
Sendyk, D. I., Rovai, E. S., Souza, N. V., Deboni, M. C. Z., & Pannuti, C. M. (2019) · Journal of Clinical Periodontology · doi:10.1111/jcpe.13128 · Cited by 35 · see citing works
Kasenda, B., von Elm, E., You, J. J., Blümle, A., Tomonaga, Y., Saccilotto, R., Amstutz, A., Bengough, T., Meerpohl, J. J., Stegert, M., et al. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002046 · Cited by 33 · see citing works
Østengaard, L., Lundh, A., Tjørnhøj-Thomsen, T., Abdi, S., Gelle, M. H. A., Stewart, L. A., Boutron, I., & Hróbjartsson, A. (2020) · BMJ · doi:10.1136/bmj.m3764 · Cited by 35 · see citing works
Perlmutter, A. S., Tran, V.-T., Dechartres, A., & Ravaud, P. (2017) · Annals of Oncology · doi:10.1093/annonc/mdw682 · Cited by 31 · see citing works
Goldacre, B., Drysdale, H., Marston, C., Mahtani, K. R., Dale, A., Milosevic, I., Slade, E., Hartley, P., & Heneghan, C. (2019) · Trials · doi:10.1186/s13063-019-3172-3 · Cited by 27 · see citing works
Chan, A.-W. (2011) · BMJ · doi:10.1136/bmj.d80 · Cited by 27 · see citing works
Gryaznov, D., von Niederhäusern, B., Speich, B., Kasenda, B., Ojeda-Ruiz, E., Blümle, A., Schandelmaier, S., Mertz, D., Odutayo, A., Tomonaga, Y., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-053417 · Cited by 22 · see citing works
Speich, B., Odutayo, A., Peckham, N., Ooms, A., Stokes, J. R., Saccilotto, R., Gryaznov, D., von Niederhäusern, B., Copsey, B., Altman, D. G., et al. (2022) · Trials · doi:10.1186/s13063-022-06516-1 · Cited by 21 · see citing works
Self-citationWeston, J., Dwan, K., Altman, D., Clarke, M., Gamble, C., Schroter, S., Williamson, P., & Kirkham, J. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010075 · Cited by 16 · see citing works
Self-citation
- Authors
- Smyth, R. M. D., Kirkham, J. J., Jacoby, A., Altman, D. G., Gamble, C., & Williamson, P. R.
- Affiliations
- University of Liverpool (GB)
- Corresponding author
- Williamson, P. R.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2011-01-06
- Volume
- 342
- Issue
- jan06 1
- Pages
- c7153-c7153
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- PMC
- PMC3016816
- OpenAlex
- W2122793971
- Cited by
- 220
- References
- 33
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 10 · 2023: 14 · 2022: 13 · 2021: 8 · 2020: 10 · 2019: 13 · 2018: 11 · 2017: 14 · 2016: 18 · 2015: 30 · 2014: 14 · 2013: 17 · 2012: 24
- FWCI
- 23.50
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- BMJ (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (acceptedVersion)
- Funders
- Cancer Research UK
- Medical Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- selective reporting, clinical trials, research integrity, selective outcome reporting, research ethics training, research ethics policy, research ethics regulation
- MeSH terms
- Clinical Protocols, Clinical Trials as Topic, Research Design, Research Personnel, Treatment Outcome, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
