Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The reporting of outcomes within published randomized trials has previously been shown to be incomplete, biased and inconsistent with study protocols. We sought to determine whether outcome reporting bias would be present in a cohort of government-funded trials subjected to rigorous peer review. METHODS: We compared protocols for randomized trials approved for funding by the Canadian Institutes of Health Research (formerly the Medical Research Council of Canada) from 1990 to 1998 with subsequent reports of the trials identified in journal publications. Characteristics of reported and unreported outcomes were recorded from the protocols and publications. Incompletely reported outcomes were defined as those with insufficient data provided in publications for inclusion in meta-analyses. An overall odds ratio measuring the association between completeness of reporting and statistical significance was calculated stratified by trial. Finally, primary outcomes specified in trial protocols were compared with those reported in publications. RESULTS: We identified 48 trials with 68 publications and 1402 outcomes. The median number of participants per trial was 299, and 44% of the trials were published in general medical journals. A median of 31% (10th-90th percentile range 5%-67%) of outcomes measured to assess the efficacy of an intervention (efficacy outcomes) and 59% (0%-100%) of those measured to assess the harm of an intervention (harm outcomes) per trial were incompletely reported. Statistically significant efficacy outcomes had a higher odds than nonsignificant efficacy outcomes of being fully reported (odds ratio 2.7; 95% confidence interval 1.5-5.0). Primary outcomes differed between protocols and publications for 40% of the trials. INTERPRETATION: Selective reporting of outcomes frequently occurs in publications of high-quality government-funded trials.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our findings echo previous research that suggests that the effects of education on behaviour may be limited [5,78,80,82-85], but may represent a necessary ingredient or first step in the process of change.”
From Systematic review of knowledge translation strategies in the allied health professions (2012) · cites this paper as [84]
In the citing paper
“There is emerging evidence that this is a common problem in the literature, yet it is a difficult problem to pinpoint [84].”
From Systematic review of knowledge translation strategies in the allied health professions (2012) · cites this paper as [84]
In the citing paper
“Although up to 38% of trials defined multiple primary outcomes,249 250 251 252”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [250]
In the citing paper
“These substantive modifications are rarely acknowledged in the final trial reports.200 249 250 480”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [250]
In the citing paper
“Outcome reporting bias (i.e., within study) occurs when authors select specific outcomes to include in the report of their research; such selection is associated with those outcomes that appear to reach statistical significance.”
From Reporting research results: a moral obligation for all researchers. (2007) · cites this paper as [11]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deeks, J. J., Altman, D. G., & Bradburn, M. J. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch15 · Cited by 2,163
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Self-citationMeta‐Analysis Of Controlled Clinical Trials
Whitehead, A. (2002) · doi:10.1002/0470854200 · Cited by 827
Melander, H., Ahlqvist-Rastad, J., Meijer, G., & Beermann, B. (2003) · BMJ · doi:10.1136/bmj.326.7400.1171 · Cited by 635
Publication and Related Biases
Song, F., Eastwood, A., Gilbody, S., Duley, L., & Sutton, A. (2001) · The Advanced Handbook of Methods in Evidence Based Healthcare · doi:10.4135/9781848608344.n21 · Cited by 455
Publication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427
How important is publication bias? A synthesis of available data.
Dickersin, K. (1997) · PubMed · Cited by 311
Bias in meta-analytic research
Felson, D. T. (1992) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(92)90072-u · Cited by 231
Underpowering in randomized trials reporting a sample size calculation
Vickers, A. J. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00141-0 · Cited by 102
Hahn, S., Williamson, P. R., & Hutton, J. L. (2002) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2002.00314.x · Cited by 97 · cited 1×
“Other literature on outcome reporting bias is limited to case reports and a small pilot study that required permission from researchers to access their ethics protocols.(8) Comparisons between journal publications and final reports submitted to drug-approval agencies have also revealed discrepancies in data reporting.”
BackgroundStudy of information submitted by drug companies to licensing authorities.
Hemminki, E. (1980) · BMJ · doi:10.1136/bmj.280.6217.833 · Cited by 96
Lassere, M. N., & Johnson, K. R. (2002) · The Lancet · doi:10.1016/s0140-6736(02)11652-7 · Cited by 34
Siegel, J. P. (1990) · New England Journal of Medicine · doi:10.1056/nejm199011083231921 · Cited by 22
Editorial Review of Protocols for Clinical Trials
(1990) · New England Journal of Medicine · doi:10.1056/nejm199011083231920 · Cited by 20
Journals should see original protocols for clinical trials
Hawkey, C. J. (2001) · BMJ · doi:10.1136/bmj.323.7324.1309 · Cited by 12
Outcome reporting bias and individual patient data meta-analysis: a case study in surgery
McCormack, K., SCOTT, N. W., & Grant, A. (2001) · BMC News and views · doi:10.1186/2048-4623-1-s3-pa005 · Cited by 5
Publication and related biases: a review
Song, F., Eastwood, A., Gilbody, S., Duley, L., & Sutton, A. (2000) · UEA Digital Repository (University of East Anglia) · Cited by 5
Doctors seem not to be punished for dishonesty in their cv
Wilmshurst, P. T. (2001) · BMJ · doi:10.1136/bmj.323.7324.1309a · Cited by 4
Cited by
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
“Selective reporting of complete studies (e.g., publication bias) [28] as well as the more recently empirically demonstrated ‘‘outcome reporting bias’’ within individual studies [40,41] should be considered by authors when conducting a systematic review and reporting its results.”
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Metadata
- Authors
- Chan, A.-W.
- Affiliations
- Canadian Institutes of Health Research (CA)
- Corresponding author
- Chan, A.-W.
- Venue
- Canadian Medical Association Journal
- Publisher
- CMA Impact Inc.
- Published
- 2004-09-28
- Volume
- 171
- Issue
- 7
- Pages
- 735-740
- ISSN
- 0820-3946 (ISSN-L)
- 1488-2329
- PMC
- PMC517858
- OpenAlex
- W2159096825
- Cited by
- 444
- References
- 18
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 9 · 2023: 14 · 2022: 14 · 2021: 15 · 2020: 13 · 2019: 16 · 2018: 21 · 2017: 17 · 2016: 20 · 2015: 23 · 2014: 16 · 2013: 27 · 2012: 42
- FWCI
- 25.89
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Canadian Medical Association Journal (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- DigitalGeorgetown (Georgetown University Library) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- selective reporting, selective outcome reporting, Canadian Institutes of Health Research, primary outcome reporting, efficacy outcomes, publication bias
- MeSH terms
- Canada, Government Agencies, Humans, Research Support as Topic, Randomized Controlled Trials as Topic, Treatment Outcome, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.