Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias — An Updated Review
Open accessThe DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias and outcome reporting bias have been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. METHODOLOGY/PRINCIPAL FINDINGS: In this update, we review and summarise the evidence from cohort studies that have assessed study publication bias or outcome reporting bias in randomised controlled trials. Twenty studies were eligible of which four were newly identified in this update. Only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Fifteen of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40-62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. CONCLUSIONS: This update does not change the conclusions of the review in which 16 studies were included. Direct empirical evidence for the existence of study publication bias and outcome reporting bias is shown. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Figure 3 Status of approved protocols for Chan 2004b study [17].](https://figures.cochat.ai/paper-figures/83600bbc5644401caed195b486efac8f/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-3.jpg)
![Figure 4 Status of approved protocols for Easterbrook 1991 study [24].](https://figures.cochat.ai/paper-figures/0a114dad26924772826758941018f42f/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-4.jpg)
![Figure 5 Status of approved protocols for Dickersin 1992 study [23].](https://figures.cochat.ai/paper-figures/1453f80dcd014ca3b9b32399777cc8d3/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-5.jpg)
![Figure 6 Status of approved protocols for Dickersin 1993 study [3].](https://figures.cochat.ai/paper-figures/1e175d9e8b2a4f0390577db97b820ca8/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-6.jpg)
![Figure 7 Status of approved protocols for Stern 1997 study [4].](https://figures.cochat.ai/paper-figures/876f2f4683aa40a689e73241856ecb1f/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-7.jpg)
![Figure 8 Status of approved protocols for Cooper 1997 study [19].](https://figures.cochat.ai/paper-figures/beceb71fe2a943b2ba35a9fc9d17293b/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-8.jpg)
![Figure 9 Status of trials for Wormald 1997 study [30].](https://figures.cochat.ai/paper-figures/f6a16e454252431f84d6cc28737b2e50/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-9.jpg)
![Figure 10 Status of approved protocols for Ioannidis 1998 study [5].](https://figures.cochat.ai/paper-figures/261662c83e664841aaa276cbc7c62394/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-10.jpg)
![Figure 11 Status of approved protocols for Pich 2003 study [27].](https://figures.cochat.ai/paper-figures/48fe85db48fc4b6ca22f25bb5580c0a7/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-11.jpg)
![Figure 12 Status of approved protocols for Cronin 2004 study [20].](https://figures.cochat.ai/paper-figures/8d211a89740e4b67b0c80783ab9c0582/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-12.jpg)
![Figure 13 Status of approved protocols for Decullier 2005 study [7].](https://figures.cochat.ai/paper-figures/c7868be91276448192f71187c860c7a5/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-13.jpg)
![Figure 14 Status of approved protocols for Decullier 2006 study [22].](https://figures.cochat.ai/paper-figures/4859f8920e714475a966997babd4cb7a/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-14.jpg)
![Figure 15 Status of approved protocols for Hahn 2002 study [26].](https://figures.cochat.ai/paper-figures/69f971391f504ed4bae3d24d4a05f2c1/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-15.jpg)
![Figure 16 Status of approved protocols for Chan 2004a study [18].](https://figures.cochat.ai/paper-figures/933f83ab86634ea581351235ceea1e11/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-16.jpg)
![Figure 17 Status of approved protocols for Ghersi 2006 study [25].](https://figures.cochat.ai/paper-figures/97881d16700f4ed48102dec8fc05b22b/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-17.jpg)
![Figure 18 Status of approved protocols for von Elm 2008 study [29].](https://figures.cochat.ai/paper-figures/90d1c2ab102849719682b2f8f8409206/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-18.jpg)
![Figure 19 Status of approved protocols for Turer 2007 study [28].](https://figures.cochat.ai/paper-figures/ef99b99be5ff4105970df8c239918ce4/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-19.jpg)
![Figure 20 Status of approved protocols for De Jong 2010 study [21].](https://figures.cochat.ai/paper-figures/bf16aa205a3c4592aa0e050025754624/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-20.jpg)
![Figure 21 Status of approved protocols for Blumle 2008 study [31].](https://figures.cochat.ai/paper-figures/f70bc7510f114d9ba3b2e1e1e342ba98/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-21.jpg)
![Figure 22 Status of approved protocols for Hall 2007 study [32].](https://figures.cochat.ai/paper-figures/577511f26e0441a6bbc7643a3998c183/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-22.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Clinical trial registries provide the ability to measure biases in the reporting of clinical trials that arise due to non-publication, delayed publication, or incomplete publication of results [14].”
From A systematic review of the processes used to link clinical trial registrations to their published results (2017) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many publications of randomized clinical trials of hypnosis interventions have been unclear or incomplete in reporting of the design elements that are necessary for the evaluation of the quality of evidence, 13 or have not provided sufficient details for possible direct replication.14,15”
From Best Practice Recommendations for Conducting and Reporting Controlled Trials in Clinical Hypnosis Research (2024) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“studies with statistically non-significant results may not have been submitted for publication (publication bias), or particular results that were statistically non-significant may have been omitted from study reports (selective non-reporting bias).”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [83]
The cited source says what the claim attributes to it.
In the citing paper
“Clinical trial registries are an important source of information for tracking clinical trials from their inception through to their reporting, and have been used to measure publication bias and outcome reporting bias.”
From Systematic review protocol assessing the processes for linking clinical trial registries and their published results (2016) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“Despite providing important information relevant to patients, trials with statistically non-significant results are more prone to non-publication,114 or, when published, to a long delay to publication114 115 116—often referred to as publication bias.”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [116]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 12×
“Easterbrook et al [24] found that compared with unfunded studies, government funded studies were more likely to yield statistically significant results but government sponsorship was not found to have a statistically significant effect on the likelihood of publication and company sponsored trials were less likely to b…”
BackgroundMethodResultInfluentialSelective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435 · cited 1×
“Several studies investigated a cohort of trials submitted to drug licensing authorities [38,40,41,65] and all found that many of these trials remain unpublished, with one study demonstrating that trials with positive outcomes resulted more often in submission of a final report to the regulatory authority [40].”
Systematic Reviews in Health Care : Meta-Analysis in Context
Egger, M., Smith, G. D., & Altman, D. G. (2000) · Cited by 2,412
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368 · cited 2×
“While much effort has been invested in trying to identify the former [2], it is equally important to understand the nature and frequency of missing data from the latter level.”
BackgroundChan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 29×
“The main objective of the study by Blumle et al [31] was to consider how eligibility criteria stated in protocols was reported in subsequent reports, in doing so they noted that 52% of studies in their cohort were published, decreasing to 48% for RCTs only.”
BackgroundMethodInfluentialSystematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 6×
“Since the original version of this review was published [13], a Cochrane methodology review is now available on publication bias [64].”
BackgroundMethodInfluentialSelf-citationPublication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291
Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133
Self-citationDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · cited 2×
“While much effort has been invested in trying to identify the former [1,2], it is equally important to understand the nature and frequency of missing data from the latter level.”
Background
Research works printed but not linked: 2
- Menzel S, Uebing B, Hucklenbroich P, Schober O (2007) Evaluation of clinical trials following an approval from a research ethics committee. Dtsch Med Wochenschr 132: 2313– 2317. 17957593
- Chappell L, Alfirevic Z, Chien P, Jarvis S, Thornton JG (2005) A comparison of the published version of randomized controlled trials in a specialist clinical journal with the original trial protocols; Chicago.
Cited by
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Self-citationPage, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200
“Because author decisions to report and editorial decisions to publish re-sults often depend on whether the P -value is above or below 0.05, selective reporting has been identified as a major problem in large segments of the scientific literature (Dwan et al. 2013; Page et al. 2014; You et al. 2012).”
BackgroundInfluentialCINeMA: an approach for assessing confidence in the results of a network meta-analysis
Nikolakopoulou, A., Higgins, J. P. T., Papakonstantinou, T., Chaimani, A., Del Giovane, C., Egger, M., & Salanti, G. (2020) · PLOS Medicine · doi:10.1371/journal.pmed.1003082 · Cited by 1,935
What Is the Evidence for Physical Therapy Poststroke? A Systematic Review and Meta-Analysis
Veerbeek, J. M., van Wegen, E., van Peppen, R., van der Wees, P. J., Hendriks, E., Rietberg, M., & Kwakkel, G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0087987 · Cited by 1,203
“The dominance of rather positive phase II trials in physical therapy may well reflect publication bias, since lowpowered negative trials are less likely to be published [551].”
Page, M. J., Moher, D., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/gwdhk · Cited by 916
Adiposity and cancer at major anatomical sites: umbrella review of the literature
Kyrgiou, M., Kalliala, I., Markozannes, G., Gunter, M. J., Paraskevaidis, E., Gabra, H., Martin-Hirsch, P., & Tsilidis, K. K. (2017) · BMJ · doi:10.1136/bmj.j477 · Cited by 803
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
“results [23]) has enhanced transparency in clinical trials research and has allowed probing of selective reporting biases [24,25], even if not fully remedying them.”
BackgroundSalameh, J.-P., Bossuyt, P. M., McGrath, T. A., Thombs, B. D., Hyde, C. J., Macaskill, P., Deeks, J. J., Leeflang, M., Korevaar, D. A., Whiting, P., et al. (2020) · BMJ · doi:10.1136/bmj.m2632 · Cited by 719
Risk factors for endometrial cancer: An umbrella review of the literature
Raglan, O., Kalliala, I., Markozannes, G., Cividini, S., Gunter, M. J., Nautiyal, J., Gabra, H., Paraskevaidis, E., Martin‐Hirsch, P., Tsilidis, K. K., et al. (2019) · International Journal of Cancer · doi:10.1002/ijc.31961 · Cited by 648
The Hong Kong Principles for assessing researchers: Fostering research integrity
Moher, D., Bouter, L., Kleinert, S., Glasziou, P., Sham, M. H., Barbour, V., Coriat, A.-M., Foeger, N., & Dirnagl, U. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000737 · Cited by 540
“, switched primary outcomes without disclosure, and spin) is around 30% [26].”
BackgroundEarly childhood environmental education: A systematic review of the research literature
Ardoin, N. M., & Bowers, A. W. (2020) · Educational Research Review · doi:10.1016/j.edurev.2020.100353 · Cited by 405
Constraint-induced movement therapy after stroke
Kwakkel, G., Veerbeek, J. M., van Wegen, E. E. H., & Wolf, S. L. (2015) · The Lancet Neurology · doi:10.1016/s1474-4422(14)70160-7 · Cited by 591
Tawfik, D. S., Scheid, A., Profit, J., Shanafelt, T., Trockel, M., Adair, K. C., Sexton, J. B., & Ioannidis, J. P. A. (2019) · Annals of Internal Medicine · doi:10.7326/m19-1152 · Cited by 516
“Publication bias occurs when studies with negative results are published less frequently or less rapidly than those with positive results (14).”
Characterization of PET/CT images using texture analysis: the past, the present… any future?
Hatt, M., Tixier, F., Pierce, L., Kinahan, P. E., Le Rest, C. C., & Visvikis, D. (2017) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-016-3427-0 · Cited by 475
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Prevalence, Characteristics, and Publication of Discontinued Randomized Trials
Kasenda, B., von Elm, E., You, J., Blümle, A., Tomonaga, Y., Saccilotto, R., Amstutz, A., Bengough, T., Meerpohl, J. J., Stegert, M., et al. (2014) · JAMA · doi:10.1001/jama.2014.1361 · Cited by 415
Grazing vs. mowing: A meta-analysis of biodiversity benefits for grassland management
Tälle, M., Deák, B., Poschlod, P., Valkó, O., Westerberg, L., & Milberg, P. (2016) · Agriculture, Ecosystems & Environment · doi:10.1016/j.agee.2016.02.008 · Cited by 387
Hitching Healthcare to the Chain: An Introduction to Blockchain Technology in the Healthcare Sector
Engelhardt, M. A. (2017) · Technology Innovation Management Review · doi:10.22215/timreview/1111 · Cited by 390
“Second, storage would be transparent: it would be clear to participants what data was and was not available, and replication of studies to verify results would be more straightforward, and there is good evidence that closer monitoring of studies is warranted (Chan et al., 2004; Dwan et al., 2013).”
BackgroundAssessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402
Citizens, consumers and farm animal welfare: A meta-analysis of willingness-to-pay studies
Clark, B., Stewart, G. B., Panzone, L. A., Kyriazakis, I., & Frewer, L. J. (2017) · Food Policy · doi:10.1016/j.foodpol.2017.01.006 · Cited by 374
Kim, M. S., Kim, W. J., Khera, A. V., Kim, J. Y., Yon, D. K., Lee, S. W., Shin, J. I., & Won, H.-H. (2021) · European Heart Journal · doi:10.1093/eurheartj/ehab454 · Cited by 378
Research Techniques Made Simple: Assessing Risk of Bias in Systematic Reviews
Drucker, A. M., Fleming, P., & Chan, A.-W. (2016) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2016.08.021 · Cited by 314
“Statistically significant findings are more likely to be published and are published a year earlier, on average, than studies with nonsignificant findings (Dwan et al., 2013; Hopewell et al., 2007).”
BackgroundDeVito, N. J., Bacon, S., & Goldacre, B. (2020) · The Lancet · doi:10.1016/s0140-6736(19)33220-9 · Cited by 336
E-Cigarette Use and Adult Cigarette Smoking Cessation: A Meta-Analysis
Wang, R. J., Bhadriraju, S., & Glantz, S. A. (2021) · American Journal of Public Health · doi:10.2105/ajph.2020.305999 · Cited by 326
Bano, M., Zowghi, D., Kearney, M., Schuck, S., & Aubusson, P. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.02.006 · Cited by 286
“This study publication bias makes it difficult for reviews and meta-analyses to present an accurate picture of the effectiveness of m-Learning (Dwan et al., 2013).”
BackgroundMethodInfluentialThe earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 308
“No wonder that a bias towards publishing significant results was observed for a long time and in many research areas (Sterling 1959; Csada, James & Espie 1996; Gerber & Malhotra 2008; Song et al. 2010; Dwan et al. 2013).”
Background‘Spin’ in published biomedical literature: A methodological systematic review
Chiu, K., Grundy, Q., & Bero, L. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002173 · Cited by 317
Detecting Selection Bias in Meta-Analyses with Multiple Outcomes: A Simulation Study
Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Van den Noortgate, W. (2021) · The Journal of Experimental Education · doi:10.1080/00220973.2019.1582470 · Cited by 285
Page, M. J., Sterne, J. A. C., Higgins, J. P. T., & Egger, M. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1468 · Cited by 294
Fernandes, B. S., Molendijk, M. L., Köhler, C. A., Soares, J. C., Leite, C. M. G. S., Machado-Vieira, R., Ribeiro, T. L., Silva, J. C., Sales, P. M. G., Quevedo, J., et al. (2015) · BMC Medicine · doi:10.1186/s12916-015-0529-7 · Cited by 291
“statistically nonsignificant results) are less likely to be published than studies with positive results [30, 31].”
ResultHigher number of items associated with significantly lower response rates in COS Delphi surveys
Gargon, E., Crew, R., Burnside, G., & Williamson, P. R. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.12.010 · Cited by 224
Self-citationPage, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281
Self-citationRezende, L. F. M. D., Sá, T. H. D., Markozannes, G., Rey-López, J. P., Lee, I.-M., Tsilidis, K. K., Ioannidis, J. P. A., & Eluf-Neto, J. (2018) · British Journal of Sports Medicine · doi:10.1136/bjsports-2017-098391 · Cited by 276
Considering aspects of the 3Rs principles within experimental animal biology
Sneddon, L. U., Halsey, L. G., & Bury, N. R. (2017) · Journal of Experimental Biology · doi:10.1242/jeb.147058 · Cited by 254
“Data on animal Jo u rn al o f Ex p er im en ta lB io lo g y experiments are often filed away and not published if found to be ‘non-significant’ (Dwan et al., 2013) – another example of the need to remove the focus on the P-value.”
BackgroundSchmucker, C., Schell, L. K., Portalupi, S., Oeller, P., Cabrera, L., Bassler, D., Schwarzer, G., Scherer, R. W., Antes, G., von Elm, E., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114023 · Cited by 269
“[58] In addition, a Cochrane Review concluded that trials with positive findings are published more often and more quickly than trials with negative findings.”
BackgroundSutherland, W. J., Taylor, N. G., MacFarlane, D., Amano, T., Christie, A. P., Dicks, L. V., Lemasson, A. J., Littlewood, N. A., Martin, P. A., Ockendon, N., et al. (2019) · Biological Conservation · doi:10.1016/j.biocon.2019.108199 · Cited by 232
Catalogue of bias: publication bias
DeVito, N. J., & Goldacre, B. (2019) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2018-111107 · Cited by 225
“7)’.(13) A 2014 systematic review by Schmucker and colleagues examined studies of publication and dissemination bias conducted using research approved by ethics committees or registered on a trial registry.”
BackgroundAn umbrella review of the evidence associating diet and cancer risk at 11 anatomical sites
Papadimitriou, N., Markozannes, G., Kanellopoulou, A., Critselis, E., Alhardan, S., Karafousia, V., Kasimis, J. C., Katsaraki, C., Papadopoulou, A., Zografou, M., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-24861-8 · Cited by 234
Vinkers, C. H., Tijdink, J. K., & Otte, W. M. (2015) · BMJ · doi:10.1136/bmj.h6467 · Cited by 237
Kharabian Masouleh, S., Eickhoff, S. B., Hoffstaedter, F., Genon, S., & Initiative, A. D. N. (2019) · eLife · doi:10.7554/elife.43464 · Cited by 230
“Based on our findings of infrequent significant SBB-associations, we could assume that flexible analyses routines, p-hacking and most importantly publication bias (Dwan et al., 2013) have contributed to the high number of significant SBB-reports in the literature.”
Standardized mean differences cause funnel plot distortion in publication bias assessments
Zwetsloot, P.-P., Van Der Naald, M., Sena, E. S., Howells, D. W., IntHout, J., De Groot, J. A., Chamuleau, S. A., MacLeod, M. R., & Wever, K. E. (2017) · eLife · doi:10.7554/elife.24260 · Cited by 212
“Simultaneously, meta60 analysis can be used to assess the risk of publication bias: the phenomenon that published research 61 is more likely to have positive or statistically significant results than unpublished experiments(1).”
BackgroundDunn, A. G., Coiera, E., Mandl, K. D., & Bourgeois, F. T. (2016) · Research Integrity and Peer Review · doi:10.1186/s41073-016-0006-7 · Cited by 208
“The databases of registered clinical trials are used to measure and mitigate reporting biases [64, 66].”
MethodPsychology's Replication Crisis and Clinical Psychological Science
Tackett, J. L., Brandes, C. M., King, K. M., & Markon, K. E. (2019) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-050718-095710 · Cited by 198
“…comparison of registrations to publications has not been conducted in clinical psychology, but evidence from health intervention research suggests that anywhere between 40% and 62% of RCTs contain at least one unreported change to the research design following registration (Dwan et al. 2013).”
BackgroundMobile Learning in Higher Education: A Systematic Literature Review
Naveed, Q. N., Choudhary, H., Ahmad, N., Alqahtani, J., & Qahmash, A. I. (2023) · Sustainability · doi:10.3390/su151813566 · Cited by 250
“The study employed evidence-based guidelines for the inclusion of various research studies, drawing upon established methodologies and best practices outlined in the works of Budgen and Brereton [32] and Dwan et al. [33].”
MethodMeta-Analysis of Menstrual Cycle Effects on Women’s Mate Preferences
Wood, W., Kressel, L., Joshi, P. D., & Louie, B. (2014) · Emotion Review · doi:10.1177/1754073914523073 · Cited by 193
Obesity and gynaecological and obstetric conditions: umbrella review of the literature
Kalliala, I., Markozannes, G., Gunter, M. J., Paraskevaidis, E., Gabra, H., Mitra, A., Terzidou, V., Bennett, P., Martin-Hirsch, P., Tsilidis, K. K., et al. (2017) · BMJ · doi:10.1136/bmj.j4511 · Cited by 199
Effectiveness of online and blended learning from schools: A systematic review
Topping, K. J., Douglas, W., Robertson, D. P., & Ferguson, N. (2022) · Review of Education · doi:10.1002/rev3.3353 · Cited by 220
Lasserson, T. J., Thomas, J., & Higgins, J. P. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch1 · Cited by 158
Dwan, K., Altman, D. G., Clarke, M., Gamble, C., Higgins, J. P. T., Sterne, J. A. C., Williamson, P. R., & Kirkham, J. J. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001666 · Cited by 187
Self-citationPractices for Improving Secondary School Climate: A Systematic Review of the Research Literature
Voight, A., & Nation, M. (2016) · American Journal of Community Psychology · doi:10.1002/ajcp.12074 · Cited by 155
Assessing Technical Skills in Talented Youth Athletes: A Systematic Review
Koopmann, T., Faber, I. R., Baker, J. R., & Schorer, J. (2020) · Sports Medicine · doi:10.1007/s40279-020-01299-4 · Cited by 205
“, either due to bias on the part of the journal or the authors to publish null findings) [100].”
BackgroundEstarli, M., Barrera, E. S. A., Martínez-Rodríguez, R., Baladía, E., Durán‐Agüero, S., Camacho‐López, S., Kristian Buhring, B., Herrero-López, A., & Gil‐González, D. (2016) · Revista Española de Nutrición Humana y Dietética · doi:10.14306/renhyd.20.2.223 · Cited by 208
Provencher, J. F., Borrelle, S. B., Bond, A. L., Lavers, J. L., van Franeker, J. A., Kühn, S., Hammer, S., Avery-Gomm, S., & Mallory, M. L. (2019) · FACETS · doi:10.1139/facets-2018-0043 · Cited by 163
“…ingested plastics is critical to understanding global patterns, it is also critical to use these standardized approaches when reporting “zeroes”; this is analogous to issues arising from meta-analyses and the importance of publishing non-significant results (Koricheva 2003; Dwan et al. 2013).”
MethodHagopian, L. P. (2020) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.691 · Cited by 166
Ensuring the Quality, Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors
Resnik, D. B., & Elmore, S. A. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-015-9625-5 · Cited by 161
“A systematic review and meta-analysis of twenty studies on positive biases found strong evidence for an association between statistically significant positive results and publication (Dwan et al. 2013).”
BackgroundRamsey, W. A., Heidelberg, R. E., Gilbert, A. M., Heneghan, M. B., Badawy, S. M., & Alberts, N. M. (2020) · Psycho-Oncology · doi:10.1002/pon.5280 · Cited by 172
“, publication bias).(71) Third, the change to our original inclusion and exclusion criteria limited the comprehensiveness of our assessment of perceptions, attitudes, and concerns related to eHealth and mHealth interventions.”
BackgroundSPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Migration of adult children and mental health of older parents ‘left behind’: An integrative review
Thapa, D. K., Visentin, D. C., Kornhaber, R. A., & Cleary, M. L. (2018) · PLoS ONE · doi:10.1371/journal.pone.0205665 · Cited by 174
CONSORT extension for reporting N-of-1 trials (CENT) 2015: Explanation and elaboration
Shamseer, L., Sampson, M., Bukutu, C., Schmid, C. H., Nikles, J., Tate, R., Johnston, B. C., Zucker, D., Shadish, W. R., Kravitz, R., et al. (2015) · BMJ · doi:10.1136/bmj.h1793 · Cited by 153
Ethical considerations in fish research
Sloman, K. A., Bouyoucos, I. A., Brooks, E. J., & Sneddon, L. U. (2019) · Journal of Fish Biology · doi:10.1111/jfb.13946 · Cited by 154
Prevalence and Localization of Pulmonary Embolism in Unexplained Acute Exacerbations of COPD
Aleva, F. E., Voets, L. W. L. M., Simons, S. O., de Mast, Q., van der Ven, A. J. A. M., & Heijdra, Y. F. (2016) · CHEST Journal · doi:10.1016/j.chest.2016.07.034 · Cited by 164
Kirkham, J. J., Altman, D. G., Chan, A.-W., Gamble, C., Dwan, K. M., & Williamson, P. R. (2018) · BMJ · doi:10.1136/bmj.k3802 · Cited by 144
Self-citationCONSORT extension for reporting N-of-1 trials (CENT) 2015: explanation and elaboration
Shamseer, L., Sampson, M., Bukutu, C., Schmid, C. H., Nikles, J., Tate, R., Johnston, B. C., Zucker, D., Shadish, W. R., Kravitz, R., et al. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.018 · Cited by 143
Enhancing teacher-student relationship quality: A narrative review of school-based interventions
Poling, D. V., Loan, C. L. V., Garwood, J. D., Zhang, S., & Riddle, D. B. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100459 · Cited by 155
Sun, G. H., Peress, L., & Pynnonen, M. A. (2014) · Otolaryngology · doi:10.1177/0194599814521779 · Cited by 143
Allan, G. M., Finley, C. R., Ton, J., Perry, D., Ramji, J., Crawford, K., Lindblad, A. J., Korownyk, C. S., & Kolber, M. R. (2018) · Europe PMC (PubMed Central) · Cited by 179
Lai, B. W., Young, H., Bickel, C. S., Motl, R. W., & Rimmer, J. H. (2017) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000000743 · Cited by 147
“Strong evidence shows that significant results are more likely to be published and significant outcomes have a higher chance of being fully reported.(151) Furthermore, across disability groups, randomized controlled trials recruited only 24 participants per study group, a concerningly low number that questions the gen…”
BackgroundBerger, M. L., Martin, B. C., Husereau, D., Worley, K., Allen, J. D., Yang, W., Quon, N. C., Mullins, C. D., Kahler, K. H., & Crown, W. (2014) · Value in Health · doi:10.1016/j.jval.2013.12.011 · Cited by 130
“[44] Dwan K, Gamble C, Williamson PR, et al.”
Seasonality and symptoms of depression: A systematic review of the literature
Øverland, S. N., Woicik, W., Sikora, L., Whittaker, K. J., Heli, H. A., Skjelkvåle, F. S., Sivertsen, B., & Colman, I. (2019) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796019000209 · Cited by 152
Tonelli, A. R., Zein, J., Adams, J., & Ioannidis, J. P. A. (2014) · Intensive Care Medicine · doi:10.1007/s00134-014-3272-1 · Cited by 133
“Beyond these two interventions, sporadic claims of mortality benefits seem to be spurious and reflect chance findings or selective analyses, as has been seen also in other fields [69, 70].”
BackgroundNeighborhood Socioeconomic Deprivation and Allostatic Load: A Scoping Review
Ribeiro, A. I., Amaro, J., Lisi, C., & Fraga, S. (2018) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph15061092 · Cited by 125
Chu, T. L., & Zhang, T. (2019) · International Journal of Sports Science & Coaching · doi:10.1177/1747954119858458 · Cited by 144
“Published doctoral dissertations were included because limited studies have examined social environments created by more than one social agent,(11,21) and publication bias could be slightly reduced.(29) Citations in the eligible articles and dissertations were also examined to identify potential studies that were not…”
BackgroundMayo-Wilson, E., Fusco, N., Li, T., Hong, H., Canner, J. K., & Dickersin, K. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.007 · Cited by 131
“For example, in RCTs that include many outcomes, trialists might report statistically significant results selectively [15, 16, 14].”
BackgroundMengesha, Z. B., Dune, T. M., & Perz, J. (2016) · Sexual Health · doi:10.1071/sh15235 · Cited by 136
Publication Bias in Special Education Meta-Analyses
Gage, N. A., Cook, B. G., & Reichow, B. (2017) · Exceptional Children · doi:10.1177/0014402917691016 · Cited by 119
Matthys, E., Remmen, R., & Bogaert, P. V. (2017) · BMC Family Practice · doi:10.1186/s12875-017-0698-x · Cited by 145
“Maybe we observe some publication bias here since most of the author groups included nurses [30].”
BackgroundBreet, E., Goldstone, D., & Bantjes, J. R. (2018) · BMC Public Health · doi:10.1186/s12889-018-5425-6 · Cited by 129
Yi, M., Wu, X., Zhuang, W., Xia, L., Chen, Y., Zhao, R., Wan, Q., Du, L., & Zhou, Y. (2019) · Molecular Nutrition & Food Research · doi:10.1002/mnfr.201900389 · Cited by 116
Palmer, C., & Pe’er, I. (2017) · PLOS Genetics · doi:10.1371/journal.pgen.1006916 · Cited by 121
Ayorinde, A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R. J., & Chen, Y. (2020) · PLoS ONE · doi:10.1371/journal.pone.0227580 · Cited by 127
“The validity of systematic reviews, however, can be undermined by publication bias, which occurs when the publication or non-publication of research findings is determined by the direction or strength of the evidence [6], and by outcome reporting bias whereby only a subset of outcomes, typically those most favourable,…”
BackgroundBelin, L., Tan, A. L., Rycke, Y. D., & Dechartres, A. (2020) · British Journal of Cancer · doi:10.1038/s41416-020-0805-y · Cited by 136
“Because our study focused on published studies, we cannot exclude that our results may be affected by publication bias: trials with negative results being more likely to remain unpublished and thus excluded from surrogacy measures.(34,35) Inferring the impact of taking into account unpublished trials on the estimated…”
Song, F., Loke, Y. K., & Hooper, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0110418 · Cited by 130
“In addition, empirical evidence indicates the existence of research dissemination bias, as published studies tend to be systematically different from unpublished studies [1,7].”
Erdös, J., & Wild, C. (2022) · European Journal of Paediatric Neurology · doi:10.1016/j.ejpn.2022.04.006 · Cited by 125
Mayo‐Wilson, E., Li, T., Fusco, N., Dickersin, K., & for the MUDS investigators (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1277 · Cited by 114
Emmers, D., Jiang, Q., Xue, H., Zhang, Y., Zhang, Y., Zhao, Y., Liu, B., Dill, S., Qian, Y., Warrinnier, N., et al. (2021) · BMJ Global Health · doi:10.1136/bmjgh-2021-005578 · Cited by 126
Garattini, S., Jakobsen, J. C., Wetterslev, J., Bertelé, V., Banzi, R., Rath, A., Neugebauer, E. A. M., Laville, M., Masson, Y., Hivert, V., et al. (2016) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2016.03.020 · Cited by 108
Interventions for compassionate nursing care: A systematic review
Blomberg, K., Griffiths, P., Wengström, Y., May, C., & Bridges, J. (2016) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2016.07.009 · Cited by 111
“Other studies suggest that studies with non-significant results are less likely to be published (Dwan et al., 2013).”
ResultCore outcome sets for research and clinical practice
Chiarotto, A., Ostelo, R. W., Turk, D. C., Buchbinder, R., & Boers, M. (2017) · Brazilian Journal of Physical Therapy · doi:10.1016/j.bjpt.2017.03.001 · Cited by 108
Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy
Mayo-Wilson, E., Li, T., Fusco, N., Bertizzolo, L., Canner, J. K., Cowley, T., Doshi, P., Ehmsen, J., Gresham, G., Guo, N., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.07.014 · Cited by 108
O’Keefe, D. J. (2015) · Health Communication · doi:10.1080/10410236.2014.974123 · Cited by 98
de Vries, Y. A., Roest, A. M., de Jonge, P., Cuijpers, P., Munafò, M. R., & Bastiaansen, J. A. (2018) · Psychological Medicine · doi:10.1017/s0033291718001873 · Cited by 98
“Unfortunately, up to half of all randomized controlled trials (RCTs) have never been published, and trials with statistically significant findings are more likely to be published than those without (Dwan et al., 2013).”
BackgroundBaudard, M., Yavchitz, A., Ravaud, P., Perrodeau, E., & Boutron, I. (2017) · BMJ · doi:10.1136/bmj.j448 · Cited by 107
Trepanowski, J. F., & Ioannidis, J. P. A. (2018) · Advances in Nutrition · doi:10.1093/advances/nmy014 · Cited by 105
“It is alsomore vulnerable to publication and selective-outcome reporting biases (47) compared with randomized research, because it is more difficult to ascertain how many observational data sets are available worldwide that can address a given exposure-outcome relation (48).”
Registration of published randomized trials: a systematic review and meta-analysis
Trinquart, L., Dunn, A. G., & Bourgeois, F. T. (2018) · BMC Medicine · doi:10.1186/s12916-018-1168-6 · Cited by 106
“If a trial is registered after the enrollment of the first participants, it is no longer possible to compare reported results to the original trial record in order to identify selective reporting of outcomes and analyses [81, 82].”
ResultTessema, G. A., Gomersall, J. S., Mahmood, M. A., & Laurence, C. (2016) · PLoS ONE · doi:10.1371/journal.pone.0165627 · Cited by 105
Xie, W., Halabi, S., Tierney, J. F., Sydes, M. R., Collette, L., Dignam, J. J., Buyse, M. E., Sweeney, C. J., & Regan, M. M. (2019) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkz002 · Cited by 108
“The excluded trials that did not report PFS and OS hazard ratios were more likely to have negative treatment effects on these endpoints (17,18).”
BackgroundImproving the relevance and consistency of outcomes in comparative effectiveness research
Tunis, S. R., Clarke, M., Gorst, S. L., Gargon, E., Blazeby, J. M., Altman, D. G., & Williamson, P. R. (2016) · Journal of Comparative Effectiveness Research · doi:10.2217/cer-2015-0007 · Cited by 95
Self-citation
Metadata
- Authors
- Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group
- Affiliations
- University of Liverpool (GB)
- Corresponding author
- Dwan, Kerry
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-07-05
- Volume
- 8
- Issue
- 7
- Pages
- e66844
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3702538
- OpenAlex
- W2002062153
- Cited by
- 1,127
- References
- 69
- Citations by year
- 2026: 75 · 2025: 84 · 2024: 64 · 2023: 71 · 2022: 127 · 2021: 98 · 2020: 112 · 2019: 89 · 2018: 83 · 2017: 102 · 2016: 89 · 2015: 70 · 2014: 57 · 2013: 5
- FWCI
- 15.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion)
- SERVAL (Université de Lausanne) (publishedVersion, cc-by)
- IRIS (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Tufts Medical Center
- King's College London
- University of Oxford
- University of Bristol
- University of Sydney
- Queen's University
- Women's College Hospital
- Centre Hospitalier Universitaire Vaudois
- Hospices Civils de Lyon
- Moorfields Eye Hospital NHS Foundation Trust
- School of Medicine, Stanford University
- Tufts University School of Medicine
- Medical Research Council
- National Health and Medical Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- publication bias, selective reporting, randomized controlled trials, meta-analysis validity, trial reporting quality, primary outcome reporting
- MeSH terms
- Humans, Publications, Cohort Studies, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.