Evidence for the Selective Reporting of Analyses and Discrepancies in Clinical Trials: A Systematic Review of Cohort Studies of Clinical Trials
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Most publications about selective reporting in clinical trials have focussed on outcomes. However, selective reporting of analyses for a given outcome may also affect the validity of findings. If analyses are selected on the basis of the results, reporting bias may occur. The aims of this study were to review and summarise the evidence from empirical cohort studies that assessed discrepant or selective reporting of analyses in randomised controlled trials (RCTs). METHODS AND FINDINGS: A systematic review was conducted and included cohort studies that assessed any aspect of the reporting of analyses of RCTs by comparing different trial documents, e.g., protocol compared to trial report, or different sections within a trial publication. The Cochrane Methodology Register, Medline (Ovid), PsycInfo (Ovid), and PubMed were searched on 5 February 2014. Two authors independently selected studies, performed data extraction, and assessed the methodological quality of the eligible studies. Twenty-two studies (containing 3,140 RCTs) published between 2000 and 2013 were included. Twenty-two studies reported on discrepancies between information given in different sources. Discrepancies were found in statistical analyses (eight studies), composite outcomes (one study), the handling of missing data (three studies), unadjusted versus adjusted analyses (three studies), handling of continuous data (three studies), and subgroup analyses (12 studies). Discrepancy rates varied, ranging from 7% (3/42) to 88% (7/8) in statistical analyses, 46% (36/79) to 82% (23/28) in adjusted versus unadjusted analyses, and 61% (11/18) to 100% (25/25) in subgroup analyses. This review is limited in that none of the included studies investigated the evidence for bias resulting from selective reporting of analyses. It was not possible to combine studies to provide overall summary estimates, and so the results of studies are discussed narratively. CONCLUSIONS: Discrepancies in analyses between publications and other study documentation were common, but reasons for these discrepancies were not discussed in the trial reports. To ensure transparency, protocols and statistical analysis plans need to be published, and investigators should adhere to these or explain discrepancies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A comparison of outcomes reported in protocols of RCTs submitted to ethics committees or registered in trial registries showed that scientists selectively report statistically significant outcomes (16–18).”
From Misrepresentation and distortion of research in biomedical literature (2018) · cites this paper as [18]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
Self-citationCONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329
Self-citationSPIRIT 2013 Statement: Defining Standard Protocol Items for Clinical Trials
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., et al. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-3-201302050-00583 · Cited by 7,363
Self-citationAbraham, J. (2010) · Handbook of Transnational Economic Governance Regimes · doi:10.1163/ej.9789004163300.i-1081.897 · Cited by 2,582
Selective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435 · cited 9×
“One study found that methods of handling missing data differed between documents in 12% (6/51) of RCTs [27].”
BackgroundInfluentialStatistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445 · cited 2×
“In seven studies [11,13,18,21,22,26,30], where the comparison was mostly made between the methods and results sections of the trial publication, it was found that a number of subgroup analyses conducted were not pre-specified (range: 14% [8/58] to 91% [49/54] of RCTs), pre-specified, but not reported in the publicatio…”
Subgroup analysis and other (mis)uses of baseline data in clinical trials
Assmann, S. F., Pocock, S. J., Enos, L. E., & Kasten, L. E. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02039-0 · Cited by 1,167 · cited 2×
“distinguish between these different timings for the choice of subgroup analyses [12].”
BackgroundDwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group (2013) · PLoS ONE · doi:10.1371/journal.pone.0066844 · Cited by 1,127
Self-citationDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · cited 1×
“Another review reached similar conclusions and also found that studies with significant results tended to be published earlier than studies with non-significant results [3].”
ResultBoutron, I. (2010) · JAMA · doi:10.1001/jama.2010.651 · Cited by 842 · cited 1×
“This reporting strategy of highlighting that the experimental treatment is beneficial despite a statistically non-significant difference for the primary outcome is one form of ‘‘spin’’ [32].”
BackgroundBaber, N. (1994) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.1994.tb05705.x · Cited by 759
Melander, H., Ahlqvist-Rastad, J., Meijer, G., & Beermann, B. (2003) · BMJ · doi:10.1136/bmj.326.7400.1171 · Cited by 635
Harmonised, I., & Guideline, T. (2006) · Cited by 435
Cordoba, G., Schwartz, L., Woloshin, S., Bae, H., & Gotzsche, P. C. (2010) · BMJ · doi:10.1136/bmj.c3920 · Cited by 400 · cited 2×
“Composite Outcomes One study (Table 6) investigated discrepancies in composite outcomes [16] and found changes in the specification of the composite outcome between abstracts, methods, and results sections in 33% (13/ 40) of RCTs.”
BackgroundRising, K., Bacchetti, P., & Bero, L. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050217 · Cited by 328 · cited 7×
“Five studies considered whether an intention-totreat analysis or per protocol analysis was specified or reported [6,20,23,25,28].”
BackgroundResultInfluentialSoares, H. P., Daniels, S., Kumar, A., Clarke, M., Scott, C., Swann, S., & Djulbegovic, B. (2004) · BMJ · doi:10.1136/bmj.328.7430.22 · Cited by 313 · cited 4×
“Five studies considered whether an intention-totreat analysis or per protocol analysis was specified or reported [6,20,23,25,28].”
BackgroundInfluentialSelf-citationTiming and Completeness of Trial Results Posted at ClinicalTrials.gov and Published in Journals
Riveros, C., Dechartres, A., Perrodeau, E., Haneef, R., Boutron, I., & Ravaud, P. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001566 · Cited by 269 · cited 2×
“[31], found that in 20% (9/45) of RCTs there were discrepancies due to different types of analysis (i.”
BackgroundComparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243 · cited 1×
“Furthermore, a systematic review of cohorts of RCTs comparing protocols or trial registry entries with corresponding publications found that discrepancies in methodological details, outcomes, and analyses were common [5].”
BackgroundSelf-citationFrequency and reasons for outcome reporting bias in clinical trials: interviews with trialists
Smyth, R. M. D., Kirkham, J. J., Jacoby, A., Altman, D. G., Gamble, C., & Williamson, P. R. (2011) · BMJ · doi:10.1136/bmj.c7153 · Cited by 220 · cited 1×
“In terms of selective reporting of outcomes, a previous study interviewed a cohort of trialists about outcomes that were specified in trial protocols but not fully reported in subsequent publications [33].”
ResultSelf-citationChan, A.-W., Hrobjartsson, A., Jorgensen, K. J., Gotzsche, P. C., & Altman, D. G. (2008) · BMJ · doi:10.1136/bmj.a2299 · Cited by 202 · cited 10×
“Competing Interests: Three of the authors (KD, PRW and DGA) are authors on four of the included studies [6,15,17,29].”
BackgroundResultInfluentialSelf-citationBias in Meta-Analysis Due to Outcome Variable Selection Within Studies
Hutton, J. L., & Williamson, P. R. (2000) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/1467-9876.00197 · Cited by 194 · cited 1×
“A citation search of a key article [1] was also performed.”
BackgroundSelf-citationSubgroup analyses in therapeutic cardiovascular clinical trials: Are most of them misleading?
Hernández, A. V., Boersma, E., Murray, G. D., Habbema, J. D. F., & Steyerberg, E. W. (2006) · American Heart Journal · doi:10.1016/j.ahj.2005.04.020 · Cited by 131 · cited 3×
“In seven studies [11,13,18,21,22,26,30], where the comparison was mostly made between the methods and results sections of the trial publication, it was found that a number of subgroup analyses conducted were not pre-specified (range: 14% [8/58] to 91% [49/54] of RCTs), pre-specified, but not reported in the publicatio…”
ResultSelective reporting in clinical trials: analysis of trial protocols accepted by The Lancet
Al-Marzouki, S., Roberts, I., Evans, S., & Marshall, T. (2008) · The Lancet · doi:10.1016/s0140-6736(08)61060-0 · Cited by 119 · cited 3×
“In seven studies [11,13,18,21,22,26,30], where the comparison was mostly made between the methods and results sections of the trial publication, it was found that a number of subgroup analyses conducted were not pre-specified (range: 14% [8/58] to 91% [49/54] of RCTs), pre-specified, but not reported in the publicatio…”
BackgroundSun, X., Briel, M., Busse, J. W., You, J. J., Akl, E. A., Mejza, F., Bala, M. M., Bassler, D., Mertz, D., Diaz-Granados, N., et al. (2011) · BMJ · doi:10.1136/bmj.d1569 · Cited by 99 · cited 2×
“In seven studies [11,13,18,21,22,26,30], where the comparison was mostly made between the methods and results sections of the trial publication, it was found that a number of subgroup analyses conducted were not pre-specified (range: 14% [8/58] to 91% [49/54] of RCTs), pre-specified, but not reported in the publicatio…”
BackgroundSelf-citationHahn, S., Williamson, P. R., & Hutton, J. L. (2002) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2002.00314.x · Cited by 97 · cited 4×
“Study (Maximum Number of RCTs) Quality Results Percent of RCTs (Number) Chan, 2008 [15] (n = 70) H–H Trial design: discrepancy (contradiction) between protocol and publication (equivalence trial reported as a superiority trial) 1% (1/70) Statistical test used for primary outcome: discrepancy (contradiction) between pr…”
BackgroundInfluentialSelf-citationVedula, S. S., Li, T., & Dickersin, K. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001378 · Cited by 87 · cited 4×
“Five studies considered whether an intention-totreat analysis or per protocol analysis was specified or reported [6,20,23,25,28].”
BackgroundInfluentialPage, M. J., McKenzie, J. E., & Forbes, A. B. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.10.010 · Cited by 79
Saquib, N., Saquib, J., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f4313 · Cited by 68 · cited 2×
“Study (Maximum Number of RCTs) Quality Results Percent of RCTs (Number) Chan, 2008 [15] (n = 70) H–H Adjusted analyses: discrepancy between protocol and publication (28 trials described adjusted analyses in the protocol or publication) 82% (23/28) Adjusted analyses: discrepancy (information absent) between protocol an…”
ResultSubgroups, treatment effects, and baseline risks: Some lessons from major cardiovascular trials
Parker, A. B., & Naylor, C. D. (2000) · American Heart Journal · doi:10.1067/mhj.2000.106610 · Cited by 61 · cited 2×
“In seven studies [11,13,18,21,22,26,30], where the comparison was mostly made between the methods and results sections of the trial publication, it was found that a number of subgroup analyses conducted were not pre-specified (range: 14% [8/58] to 91% [49/54] of RCTs), pre-specified, but not reported in the publicatio…”
Yu, L.-M., Chan, A.-W., Hopewell, S., Deeks, J. J., & Altman, D. G. (2010) · Trials · doi:10.1186/1745-6215-11-59 · Cited by 60
Self-citationComparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Blundell, M., Gamble, C. L., & Williamson, P. R. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031 · Cited by 56
Self-citationHernández, A. V., Steyerberg, E. W., Taylor, G. S., Marmarou, A., Habbema, J. D. F., & Maas, A. I. R. (2005) · Neurosurgery · doi:10.1227/01.neu.0000186039.57548.96 · Cited by 52 · cited 1×
“Study (Maximum Number of RCTs) Quality Results Percent of RCTs (Number) Chan, 2008 [15] (n = 70) H–H Discrepancy between protocol and publication (25 trials described subgroup analyses in the protocol or publication) 100% (25/25) Discrepancy between protocol and publication (pre-specified in protocol and only some or…”
Rosenthal, R., & Dwan, K. (2013) · Annals of Surgery · doi:10.1097/sla.0b013e318283cf7f · Cited by 50 · cited 14×
“Five studies considered whether an intention-totreat analysis or per protocol analysis was specified or reported [6,20,23,25,28].”
BackgroundMethodInfluentialSelf-citationMisuse of Baseline Comparison Tests and Subgroup Analyses in Surgical Trials
Bhandari, M., Devereaux, P. J., Li, P., Mah, D., Lim, K., Schünemann, H. J., & Tornetta, P. (2006) · Clinical Orthopaedics & Related Research · doi:10.1097/01.blo.0000218736.23506.fe · Cited by 36 · cited 2×
“In seven studies [11,13,18,21,22,26,30], where the comparison was mostly made between the methods and results sections of the trial publication, it was found that a number of subgroup analyses conducted were not pre-specified (range: 14% [8/58] to 91% [49/54] of RCTs), pre-specified, but not reported in the publicatio…”
Kasenda, B., von Elm, E. B., You, J., Blümle, A., Tomonaga, Y., Saccilotto, R., Amstutz, A., Bengough, T., Meerpohl, J., Stegert, M., et al. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-131 · Cited by 36 · cited 1×
“One study [7,9,10] included 894 protocols (and 520 related journal publications) approved by six research ethics committees from 2000 to 2003 in Switzerland, Germany, and Canada.”
BackgroundReporting on methods of subgroup analysis in clinical trials: a survey of four scientific journals
Moreira Jr., E. D., Stein, Z., & Susser, E. (2001) · Brazilian Journal of Medical and Biological Research · doi:10.1590/s0100-879x2001001100011 · Cited by 27 · cited 2×
“In seven studies [11,13,18,21,22,26,30], where the comparison was mostly made between the methods and results sections of the trial publication, it was found that a number of subgroup analyses conducted were not pre-specified (range: 14% [8/58] to 91% [49/54] of RCTs), pre-specified, but not reported in the publicatio…”
A Comparison of Subgroup Analyses in Grant Applications and Publications
Boonacker, C. W. B., Hoes, A. W., van Liere-Visser, K., Schilder, A. G. M., & Rovers, M. M. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwr075 · Cited by 11 · cited 2×
“Four studies considered discrepancies; and the discrepancy rate ranged from 61% (11/18) to 100% (25/25) of RCTs in three studies [11,14,15].”
Background
Cited by
Guidelines for the Content of Statistical Analysis Plans in Clinical Trials
Gamble, C., Krishan, A., Stocken, D., Lewis, S., Juszczak, E., Doré, C., Williamson, P. R., Altman, D. G., Montgomery, A., Lim, P., et al. (2017) · JAMA · doi:10.1001/jama.2017.18556 · Cited by 536
Self-citationWhy clinical trial outcomes fail to translate into benefits for patients
Heneghan, C., Goldacre, B., & Mahtani, K. R. (2017) · Trials · doi:10.1186/s13063-017-1870-2 · Cited by 404
“Authors of a systematic review of selective outcome reporting, including 27 analyses, found that the median proportion of trials with a discrepancy between the registered and published primary outcome was 31% [53].”
BackgroundPage, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281
Self-citationMisrepresentation and distortion of research in biomedical literature
Boutron, I., & Ravaud, P. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1710755115 · Cited by 273
Riley, R. D., Jackson, D., Salanti, G., Burke, D. L., Price, M., Kirkham, J., & White, I. R. (2017) · BMJ · doi:10.1136/bmj.j3932 · Cited by 247
Self-citation
Metadata
- Authors
- Dwan, K., Altman, D. G., Clarke, M., Gamble, C., Higgins, J. P. T., Sterne, J. A. C., Williamson, P. R., & Kirkham, J. J.
- Affiliations
- University of Liverpool (GB)
- University of Oxford (GB)
- Queen's University Belfast (GB)
- University of Bristol (GB)
- Corresponding author
- Dwan, Kerry
- ORCID
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-06-24
- Volume
- 11
- Issue
- 6
- Pages
- e1001666
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC4068996
- OpenAlex
- W2071154758
- Cited by
- 187
- References
- 37
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 14 · 2023: 16 · 2022: 14 · 2021: 11 · 2020: 16 · 2019: 20 · 2018: 15 · 2017: 25 · 2016: 18 · 2015: 21 · 2014: 5
- FWCI
- 6.30
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (publishedVersion)
- Bristol Research (University of Bristol) (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Research Explorer (The University of Manchester) (submittedVersion)
- Explore Bristol Research (submittedVersion)
- Funders
- Cancer Research UK (16895, C5529)
- National Institute for Health and Care Research (NF-SI-0611-10168)
- Medical Research Council (MR/K025643/1, MR/L004933/1, MR/K025635/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- selective reporting, reporting bias, statistical analysis plan, subgroup analysis, missing data handling, composite endpoints, randomized controlled trials
- MeSH terms
- Clinical Trials as Topic, Humans, Publishing, Research Design, Cohort Studies, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
