Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention. METHODOLOGY/PRINCIPAL FINDINGS: We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40-62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. CONCLUSIONS: Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Figure 3 Status of approved protocols for Chan 2004b study [15].](https://figures.cochat.ai/paper-figures/59b15ebd40f546c6abdc941af3f030b5/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-3.jpg)
![Figure 4 Status of approved protocols for Easterbrook 1991 study [26].](https://figures.cochat.ai/paper-figures/14a7658864a940c2b24cb84c7e8cb605/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-4.jpg)
![Figure 5 Status of approved protocols for Dickersin 1992 study [27].](https://figures.cochat.ai/paper-figures/5c6595c5a1114b54a63fd775278bc099/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-5.jpg)
![Figure 6 Status of approved protocols for Dickersin 1993 study [3].](https://figures.cochat.ai/paper-figures/aa3d5c4186b24331b2c214488355f8a7/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-6.jpg)
![Figure 7 Status of approved protocols for Stern 1997 study [4].](https://figures.cochat.ai/paper-figures/04afce37877d467bbabff840c38d7301/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-7.jpg)
![Figure 8 Status of approved protocols for Cooper 1997 study [32].](https://figures.cochat.ai/paper-figures/f0cd16c328a14425a21a07beff12b5e9/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-8.jpg)
![Figure 9 Status of trials for Wormald 1997 study [21].](https://figures.cochat.ai/paper-figures/40fd118f57954ed5a014900d0265fe95/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-9.jpg)
![Figure 10 Status of approved protocols for Ioannidis 1998 study [5].](https://figures.cochat.ai/paper-figures/404ab188570a4eb3b98992e2a1cd5fbc/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-10.jpg)
![Figure 11 Status of approved protocols for Pich 2003 study [28].](https://figures.cochat.ai/paper-figures/d57aee050aa8429e955ea17eab36a5e4/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-11.jpg)
![Figure 12 Status of approved protocols for Cronin 2004 study [31].](https://figures.cochat.ai/paper-figures/4caeeb0c2bc944ef845ddb964c8a76bd/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-12.jpg)
![Figure 13 Status of approved protocols for Decullier 2005 study [29].](https://figures.cochat.ai/paper-figures/d5e27b13a4504b068f8b813f84a5792a/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-13.jpg)
![Figure 14 Status of approved protocols for Decullier 2006 study [30].](https://figures.cochat.ai/paper-figures/d623c2abf5114e8abea683517296a756/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-14.jpg)
![Figure 15 Status of approved protocols for Hahn 2002 study [13].](https://figures.cochat.ai/paper-figures/2dd3013142574a138ea802be74b77966/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-15.jpg)
![Figure 16 Status of approved protocols for Chan 2004a study [14].](https://figures.cochat.ai/paper-figures/632e7328c62f4e1e9fdcbb1e5767e34a/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-16.jpg)
![Figure 17 Status of approved protocols for Ghersi 2006 study [17].](https://figures.cochat.ai/paper-figures/efcfe899e56047d99ddb45bbd8091c51/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-17.jpg)
![Figure 18 Status of approved protocols for von Elm 2008 study [18].](https://figures.cochat.ai/paper-figures/3e7a281f19ef40dca07cbfc23dfeaaf3/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-18.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unfortunately, attempted assessments frequently fail because authors of many trial reports neglect to provide lucid and complete descriptions of that critical information [2-4].”
From CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials (2010) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and there are also reasons to believe that publication bias is more likely to affect low powered studies (25).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical evidence suggests that authors often ignore, in the conduct and reporting of their trial, what they stated in the protocol [4,22].”
From CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. (2010) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, publication bias, selective data analysis and selective reporting of outcomes are more likely to affect low-powered studies.”
From Power failure: why small sample size undermines the reliability of neuroscience (2013) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical evidence suggests that authors often ignore, in the conduct and reporting of their trial, what they stated in the protocol. [4],[22]”
From CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials (2010) · cites this paper as [4]
Experimental — accuracy not yet measured
14 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 12×
“The HTA report of Song et al [1] led to four potentially eligible empirical studies [3,4,26,27], all of which had been identified previously.”
BackgroundMethodResultInfluentialSelf-citationSystematic Reviews in Health Care : Meta-Analysis in Context
Egger, M., Smith, G. D., & Altman, D. G. (2000) · Cited by 2,401
Self-citationPublication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 30×
“Status of approved protocols for Chan 2004b study [15].”
BackgroundMethodInfluentialSelf-citationSystematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693
Systematic Reviews in Health Care. Meta-analysis in Context.
Kleijnen, J., & Antes, G. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.3.697 · Cited by 1,363
Full publication of results initially presented in abstracts.
Scherer, R. W., Langenberg, P., & von Elm, E. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub3 · Cited by 997 · cited 1×
“The Cochrane review by Scherer et al [6] investigating the full publication of results initially presented in abstracts found that only 63% of results from abstracts describing randomized or controlled clinical trials are published in full and ‘positive’ results were more frequently published than non ‘positive’ resul…”
BackgroundPublication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898
Dickersin, K. (1992) · JAMA · doi:10.1001/jama.1992.03480030052036 · Cited by 854
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818 · cited 11×
“Stern et al [4] reported that this finding was even stronger for their subgroup of clinical trials (Hazard Ratio (HR) 3.13 (95% confidence interval (CI) 1.76, 5.58), p = 0.0001) compared to all quantitative studies (HR 2.32 (95% CI 1.47, 3.66), p = 0.0003).”
BackgroundMethodResultInfluential
Research works printed but not linked: 2
- E Von Elm, 2008, Selective reporting of outcomes of drug trials; Comparison of study protocols and published articles., Swiss Med Wkly, 138 (13–14), 197
- L Chappell, 2005, A comparison of the published version of randomized controlled trials in a specialist clinical journal with the original trial protocols.
Cited by
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
“Bias from selective outcome reporting in randomized trials has been well documented [42,43].”
BackgroundInfluentialSelf-citationPRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Self-citationCONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
Self-citationShamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Self-citation
Metadata
- Authors
- Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., Ghersi, D., Ioannidis, J. P. A., Simes, J., & Williamson, P. R.
- Affiliations
- University of Liverpool (GB)
- University of Oxford (GB)
- Hospital Clínic de Barcelona (ES)
- Moorfields Eye Hospital (GB)
- Canadian Institutes of Health Research (CA)
- King's College London (GB)
- University of Bern (CH)
- National Health and Medical Research Council (AU)
- Tufts University (US)
- The University of Sydney (AU)
- Corresponding author
- Dwan, Kerry
- ORCID
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2008-08-28
- Volume
- 3
- Issue
- 8
- Pages
- e3081
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC2518111
- OpenAlex
- W2091959199
- Cited by
- 1,546
- References
- 48
- Citations by year
- 2026: 45 · 2025: 51 · 2024: 67 · 2023: 51 · 2022: 76 · 2021: 118 · 2020: 90 · 2019: 72 · 2018: 96 · 2017: 90 · 2016: 129 · 2015: 121 · 2014: 109 · 2013: 131 · 2012: 103
- FWCI
- 20.77
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Dipòsit Digital de la Universitat de Barcelona (Universitat de Barcelona) (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- Bern Open Repository and Information System (University of Bern) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Research Portal (King's College London) (submittedVersion)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by)
- Funders
- Cancer Research UK
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- publication bias, selective reporting, randomized controlled trials, outcome reporting, meta-analysis validity, clinical trial registration
- MeSH terms
- Clinical Trials as Topic, Humans, Publishing, Cohort Studies, Randomized Controlled Trials as Topic, Treatment Outcome, Publication Bias, Patient Selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias. https://app.cochat.ai/papers/10.1371/journal.pone.0003081
- The paper
- Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., Ghersi, D., Ioannidis, J. P. A., Simes, J., & Williamson, P. R. (2008). Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias. PLoS ONE. https://doi.org/10.1371/journal.pone.0003081
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.