Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias — An Updated Review
Open accessThe DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias and outcome reporting bias have been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. METHODOLOGY/PRINCIPAL FINDINGS: In this update, we review and summarise the evidence from cohort studies that have assessed study publication bias or outcome reporting bias in randomised controlled trials. Twenty studies were eligible of which four were newly identified in this update. Only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Fifteen of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40-62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. CONCLUSIONS: This update does not change the conclusions of the review in which 16 studies were included. Direct empirical evidence for the existence of study publication bias and outcome reporting bias is shown. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Figure 3 Status of approved protocols for Chan 2004b study [17].](https://figures.cochat.ai/paper-figures/83600bbc5644401caed195b486efac8f/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-3.jpg)
![Figure 4 Status of approved protocols for Easterbrook 1991 study [24].](https://figures.cochat.ai/paper-figures/0a114dad26924772826758941018f42f/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-4.jpg)
![Figure 5 Status of approved protocols for Dickersin 1992 study [23].](https://figures.cochat.ai/paper-figures/1453f80dcd014ca3b9b32399777cc8d3/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-5.jpg)
![Figure 6 Status of approved protocols for Dickersin 1993 study [3].](https://figures.cochat.ai/paper-figures/1e175d9e8b2a4f0390577db97b820ca8/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-6.jpg)
![Figure 7 Status of approved protocols for Stern 1997 study [4].](https://figures.cochat.ai/paper-figures/876f2f4683aa40a689e73241856ecb1f/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-7.jpg)
![Figure 8 Status of approved protocols for Cooper 1997 study [19].](https://figures.cochat.ai/paper-figures/beceb71fe2a943b2ba35a9fc9d17293b/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-8.jpg)
![Figure 9 Status of trials for Wormald 1997 study [30].](https://figures.cochat.ai/paper-figures/f6a16e454252431f84d6cc28737b2e50/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-9.jpg)
![Figure 10 Status of approved protocols for Ioannidis 1998 study [5].](https://figures.cochat.ai/paper-figures/261662c83e664841aaa276cbc7c62394/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-10.jpg)
![Figure 11 Status of approved protocols for Pich 2003 study [27].](https://figures.cochat.ai/paper-figures/48fe85db48fc4b6ca22f25bb5580c0a7/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-11.jpg)
![Figure 12 Status of approved protocols for Cronin 2004 study [20].](https://figures.cochat.ai/paper-figures/8d211a89740e4b67b0c80783ab9c0582/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-12.jpg)
![Figure 13 Status of approved protocols for Decullier 2005 study [7].](https://figures.cochat.ai/paper-figures/c7868be91276448192f71187c860c7a5/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-13.jpg)
![Figure 14 Status of approved protocols for Decullier 2006 study [22].](https://figures.cochat.ai/paper-figures/4859f8920e714475a966997babd4cb7a/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-14.jpg)
![Figure 15 Status of approved protocols for Hahn 2002 study [26].](https://figures.cochat.ai/paper-figures/69f971391f504ed4bae3d24d4a05f2c1/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-15.jpg)
![Figure 16 Status of approved protocols for Chan 2004a study [18].](https://figures.cochat.ai/paper-figures/933f83ab86634ea581351235ceea1e11/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-16.jpg)
![Figure 17 Status of approved protocols for Ghersi 2006 study [25].](https://figures.cochat.ai/paper-figures/97881d16700f4ed48102dec8fc05b22b/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-17.jpg)
![Figure 18 Status of approved protocols for von Elm 2008 study [29].](https://figures.cochat.ai/paper-figures/90d1c2ab102849719682b2f8f8409206/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-18.jpg)
![Figure 19 Status of approved protocols for Turer 2007 study [28].](https://figures.cochat.ai/paper-figures/ef99b99be5ff4105970df8c239918ce4/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-19.jpg)
![Figure 20 Status of approved protocols for De Jong 2010 study [21].](https://figures.cochat.ai/paper-figures/bf16aa205a3c4592aa0e050025754624/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-20.jpg)
![Figure 21 Status of approved protocols for Blumle 2008 study [31].](https://figures.cochat.ai/paper-figures/f70bc7510f114d9ba3b2e1e1e342ba98/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-21.jpg)
![Figure 22 Status of approved protocols for Hall 2007 study [32].](https://figures.cochat.ai/paper-figures/577511f26e0441a6bbc7643a3998c183/systematic-review-of-the-empirical-evidence-of-study-publication-bias-and-figure-22.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Clinical trial registries provide the ability to measure biases in the reporting of clinical trials that arise due to non-publication, delayed publication, or incomplete publication of results [14].”
From A systematic review of the processes used to link clinical trial registrations to their published results (2017) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many publications of randomized clinical trials of hypnosis interventions have been unclear or incomplete in reporting of the design elements that are necessary for the evaluation of the quality of evidence, 13 or have not provided sufficient details for possible direct replication.14,15”
From Best Practice Recommendations for Conducting and Reporting Controlled Trials in Clinical Hypnosis Research (2024) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“studies with statistically non-significant results may not have been submitted for publication (publication bias), or particular results that were statistically non-significant may have been omitted from study reports (selective non-reporting bias).”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [83]
The cited source says what the claim attributes to it.
In the citing paper
“Clinical trial registries are an important source of information for tracking clinical trials from their inception through to their reporting, and have been used to measure publication bias and outcome reporting bias.”
From Systematic review protocol assessing the processes for linking clinical trial registries and their published results (2016) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“Despite providing important information relevant to patients, trials with statistically non-significant results are more prone to non-publication,114 or, when published, to a long delay to publication114 115 116—often referred to as publication bias.”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [116]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 12×
“Easterbrook et al [24] found that compared with unfunded studies, government funded studies were more likely to yield statistically significant results but government sponsorship was not found to have a statistically significant effect on the likelihood of publication and company sponsored trials were less likely to b…”
BackgroundMethodResultInfluentialSelective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435 · cited 1×
“Several studies investigated a cohort of trials submitted to drug licensing authorities [38,40,41,65] and all found that many of these trials remain unpublished, with one study demonstrating that trials with positive outcomes resulted more often in submission of a final report to the regulatory authority [40].”
Systematic Reviews in Health Care : Meta-Analysis in Context
Egger, M., Smith, G. D., & Altman, D. G. (2000) · Cited by 2,412
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368 · cited 2×
“While much effort has been invested in trying to identify the former [2], it is equally important to understand the nature and frequency of missing data from the latter level.”
BackgroundChan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 29×
“The main objective of the study by Blumle et al [31] was to consider how eligibility criteria stated in protocols was reported in subsequent reports, in doing so they noted that 52% of studies in their cohort were published, decreasing to 48% for RCTs only.”
BackgroundMethodInfluentialSystematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 6×
“Since the original version of this review was published [13], a Cochrane methodology review is now available on publication bias [64].”
BackgroundMethodInfluentialSelf-citationPublication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291
Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133
Self-citationDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · cited 2×
“While much effort has been invested in trying to identify the former [1,2], it is equally important to understand the nature and frequency of missing data from the latter level.”
Background
Research works printed but not linked: 2
- Menzel S, Uebing B, Hucklenbroich P, Schober O (2007) Evaluation of clinical trials following an approval from a research ethics committee. Dtsch Med Wochenschr 132: 2313– 2317. 17957593
- Chappell L, Alfirevic Z, Chien P, Jarvis S, Thornton JG (2005) A comparison of the published version of randomized controlled trials in a specialist clinical journal with the original trial protocols; Chicago.
Cited by
Items 101–200 of 1,001
Roest, A. M., de Jonge, P., Williams, C. D., de Vries, Y. A., Schoevers, R. A., & Turner, E. H. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2015.15 · Cited by 101
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
“…et al. 2010 Selective reporting (e.g., only reporting experiments, outcomes, or analyses that achieved statistical significance) Chan et al. 2004; Dwan et al. 2011, 2013, 2014; Franco et al. 2014, 2016; Goldacre et al. 2019; John et al. 2012; Easterbrook et al. 1991 Presenting post hoc…”
Blood pressure lowering efficacy of renin inhibitors for primary hypertension
Musini, V. M., Lawrence, K. A., Fortin, P. M., Bassett, K., & Wright, J. M. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007066.pub3 · Cited by 98
The Role of Publication Selection Bias in Estimates of the Value of a Statistical Life
Viscusi, W. K. (2015) · American Journal of Health Economics · doi:10.1162/ajhe_a_00002 · Cited by 96
“Both outcome reporting bias and publication bias substantially affect the publication of results for randomized control trials of health-care interventions (Dwan et al. 2008, 2013).”
BackgroundLittell, J. H., Pigott, T., Nilsen, W., Green, S. J., & Montgomery, O. L. K. (2021) · Campbell Systematic Reviews · doi:10.1002/cl2.1158 · Cited by 115
Tong, A., Manns, B., Hemmelgarn, B. R., Wheeler, D. C., Tugwell, P., Winkelmayer, W. C., Biesen, W. V., Crowe, S., Kerr, P. G., Polkinghorne, K. R., et al. (2015) · Trials · doi:10.1186/s13063-015-0895-7 · Cited by 100
“Outcome reporting bias can also threaten the validity of meta-analysis and reliability of evidence [24, 25].”
BackgroundCybulski, L., Mayo-Wilson, E., & Grant, S. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000115 · Cited by 90
Depression and episodic memory across the adult lifespan: A meta-analytic review.
James, T., Weiss-Cowie, S., Hopton, Z. W., Verhaeghen, P., Dotson, V. M., & Duarte, A. (2021) · Psychological Bulletin · doi:10.1037/bul0000344 · Cited by 149
Pandis, N., Shamseer, L., Kokich, V. G., Fleming, P. S., & Moher, D. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.04.001 · Cited by 92
Mathur, M., & VanderWeele, T. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1464 · Cited by 59
Weather-Centric Rangeland Revegetation Planning
Hardegree, S. P., Abatzoglou, J. T., Brunson, M. W., Germino, M. J., Hegewisch, K. C., Moffet, C. A., Pilliod, D. S., Roundy, B. A., Boehm, A. R., & Meredith, G. R. (2017) · Rangeland Ecology & Management · doi:10.1016/j.rama.2017.07.003 · Cited by 98
Systematic review of Mendelian randomization studies on risk of cancer
Markozannes, G., Kanellopoulou, A., Dimopoulou, O., Kosmidis, D., Zhang, X., Wang, L., Τheodoratou, E., Gill, D., Burgess, S., & Tsilidis, K. K. (2022) · BMC Medicine · doi:10.1186/s12916-022-02246-y · Cited by 106
de Almeida, M. O., Yamato, T. P., Parreira, P. D. C. S., Costa, L. O. P., Kamper, S. J., & Saragiotto, B. T. (2019) · Brazilian Journal of Physical Therapy · doi:10.1016/j.bjpt.2019.04.004 · Cited by 96
Lifestyle Interventions in Preschool Children: A Meta-analysis of Effectiveness
Ling, J., Robbins, L. B., Wen, F., & Zhang, N. (2017) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2017.01.018 · Cited by 91
Price, J., Rushton, A., Tyros, I., Tyros, V., & Heneghan, N. R. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234511 · Cited by 104
Testing small study effects in multivariate meta‐analysis
Hong, C., Salanti, G., Morton, S. C., Riley, R. D., Chu, H., Kimmel, S. E., & Chen, Y. (2020) · Biometrics · doi:10.1111/biom.13342 · Cited by 96
“Outcome reporting bias (ORB) had not received sufficient attention until recently (Chan and Altman, 2005; Dwan et al., 2013).”
BackgroundDiabetes in Pregnancy and Childhood Cognitive Development: A Systematic Review
Adane, A. A., Mishra, G. D., & Tooth, L. R. (2016) · Pediatrics · doi:10.1542/peds.2015-4234 · Cited by 87
The misuse and abuse of statistics in biomedical research
Thiese, M. S., Arnold, Z. C., & Walker, S. D. (2015) · Biochemia Medica · doi:10.11613/bm.2015.001 · Cited by 87
“There is a publication bias, where statistically significant results are more likely to be published (31).”
BackgroundReview finds core outcome set uptake in new studies and systematic reviews needs improvement
Williamson, P. R., Barrington, H., Blazeby, J. M., Clarke, M. J., Gargon, E., Gorst, S. L., Saldanha, I. J., & Tunis, S. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.06.016 · Cited by 92
Self-citationHyaluronan for knee osteoarthritis: an updated meta-analysis of trials with low risk of bias
Richette, P., Chevalier, X., Ea, H., Eymard, F., Henrotin, Y. E., Ornetti, P., Sellam, J., Cucherat, M., & Marty, M. (2015) · RMD Open · doi:10.1136/rmdopen-2015-000071 · Cited by 94
“46 47 In addition, it should be noted that our time point assessment was 3 months, which can be considered a short period of time.”
Hall, D. A., Smith, H. L., Hibbert, A., Colley, V., Haider, H. F., Horobin, A. J., Londero, A., Mazurek, B., Thacker, B., Fackrell, K., et al. (2018) · Trends in Hearing · doi:10.1177/2331216518814384 · Cited by 95
Li, J., Li, Y., Luo, L.-J., Ye, J., Zhong, D., Xiao, Q., Zheng, H., Geng, C.-M., Jin, R., & Liang, F. (2019) · Medicine · doi:10.1097/md.0000000000016301 · Cited by 101
Guyatt, G. H., Wang, Y., Eachempati, P., Iorio, A., Murad, M. H., Hultcrantz, M., Chu, D. K., Flórez, I. D., Hemkens, L. G., Agoritsas, T., et al. (2025) · BMJ · doi:10.1136/bmj-2024-083864 · Cited by 106
Gebara, M. A., Lipsey, K. L., Karp, J. F., Nash, M. C., Iaboni, A., & Lenze, E. J. (2015) · The American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2014.11.004 · Cited by 84
“(103) Other observational study biases that may apply in the case of antidepressants and falls also include bias due to differential recall of treatment exposure(94) when patients provide reports that may not be reliable, and publication bias(104) with trends towards only reporting positive results.”
Gopal, A. D., Wallach, J. D., Aminawung, J. A., Gonsalves, G., Dal-Ré, R., Miller, J. E., & Ross, J. S. (2018) · Trials · doi:10.1186/s13063-018-2825-y · Cited by 79
Tälle, M., Déak, B., Poschlod, P., Valkó, O., Westerberg, L. M., & Milberg, P. (2018) · Biodiversity and Conservation · doi:10.1007/s10531-018-1562-6 · Cited by 100
Young, I., Waddell, L., Harding, S., Greig, J., Mascarenhas, M., Sivaramalingam, B., Pham, M. T., & Papadopoulos, A. (2015) · BMC Public Health · doi:10.1186/s12889-015-2171-x · Cited by 83
Dunn, A. G., Arachi, D., Hudgins, J., Tsafnat, G., Coiera, E., & Bourgeois, F. T. (2014) · Annals of Internal Medicine · doi:10.7326/m14-0933 · Cited by 84
Monaco, A. D., Pelargonio, G., Narducci, M. L., Manzoli, L., Boccia, S., Flacco, M. E., Capasso, L., Barone, L., Perna, F., Bencardino, G., et al. (2014) · EP Europace · doi:10.1093/europace/euu137 · Cited by 83
Best Estimate Selection Bias in the Value of a Statistical Life
Viscusi, W. K. (2018) · Journal of Benefit-Cost Analysis · doi:10.1017/bca.2017.21 · Cited by 74
IBM’s Health Analytics and Clinical Decision Support
Sun, J., Knoop, S., Shabo, A., Carmeli, B., Sow, D., Syed-Mahmood, T., Rapp, W., & Kohn, M. S. (2014) · Yearbook of Medical Informatics · doi:10.15265/iy-2014-0002 · Cited by 78
Animal models of Huntington's disease for translation to the clinic: Best practices
Menalled, L., & Brunner, D. (2014) · Movement Disorders · doi:10.1002/mds.26006 · Cited by 86
“Publications Bias An important problem is the bias toward publication of positive results attributable to either no incentive or difficulty in publishing negative results.(41-44) Lack of publication of negative results or failure to replicate apparent positive results creates the illusion of success and diverts time a…”
BackgroundBoeters, M., Garcia‐Morante, B., Schaik, G. V., Segalés, J., Rushton, J., & Steeneveld, W. (2023) · Porcine Health Management · doi:10.1186/s40813-023-00342-w · Cited by 82
“There may be an outcome reporting bias, with only the favourable interventions reported that can undermine the validity of systematic reviews [61].”
BackgroundReported Outcome Measures in Degenerative Cervical Myelopathy: A Systematic Review
Davies, B. M., McHugh, M., Elgheriani, A., Kolias, A. G., Tetreault, L. A., Hutchinson, P. J., Fehlings, M. G., & Kotter, M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157263 · Cited by 85
A survey of adaptive context-aware learning environments
Hasanov, A., Laine, T. H., & Chung, T. (2019) · Journal of Ambient Intelligence and Smart Environments · doi:10.3233/ais-190534 · Cited by 93
“[7] 2003–2016 49 Mobile learning environments (P), (RM) [16,38]”
Phillips, C. L., Iannaccone, J. A., Rooker, G. W., & Hagopian, L. P. (2017) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.376 · Cited by 81
van Lent, M., Overbeke, J., & Out, H. J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0104846 · Cited by 89
Cox, J., Varatharajan, S., Côté, P., & Optima Collaboration (2016) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2016.6270 · Cited by 91
Association of Trial Registration With Reporting of Primary Outcomes in Protocols and Publications
Chan, A.-W., Pello, A., Kitchen, J., Axentiev, A., Virtanen, J. I., Liu, A., & Hemminki, E. (2017) · JAMA · doi:10.1001/jama.2017.13001 · Cited by 78
Translational Research in Cardiovascular Repair
Chamuleau, S. A. J., van der Naald, M., Climent, A. M., Kraaijeveld, A. O., Wever, K., Duncker, D. J., Fernández‐Avilés, F., & Bolli, R. (2018) · Circulation Research · doi:10.1161/circresaha.117.311565 · Cited by 82
Questionable science and reproducibility in electrical brain stimulation research
Héroux, M. E., Loo, C. K. A., Taylor, J. L., & Gandevia, S. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175635 · Cited by 80
Null Effects and Publication Bias in Special Education Research
Cook, B. G., & Therrien, W. J. (2017) · Behavioral Disorders · doi:10.1177/0198742917709473 · Cited by 82
“Ferguson and Heene (2012) suggested that the research community is generally uncomfortable with null findings because they contradict researchers’ expectations, can be difficult to explain, and are sometimes associated with flawed studies (Heene & Ferguson, 2017; Matosin et al.”
BackgroundAssociation between trial registration and treatment effect estimates: a meta-epidemiological study
Dechartres, A., Ravaud, P., Atal, I., Riveros, C., & Boutron, I. (2016) · BMC Medicine · doi:10.1186/s12916-016-0639-x · Cited by 71
“Registration aims to make information about the existence and methods of clinical trials publicly available to limit the effect of selective publication of trials and outcomes with positive results resulting in exaggerated treatment effect estimates [8–13].”
Landais, L. L., Damman, O. C., Schoonmade, L. J., Timmermans, D. R. M., Verhagen, E. A. L. M., & Jelsma, J. G. M. (2020) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-020-00942-7 · Cited by 82
“Finally, we did not assess the risk of publication bias [132].”
BackgroundEmotional Outcomes of Casual Sexual Relationships and Experiences: A Systematic Review
Wesche, R., Claxton, S. E., & Waterman, E. A. (2020) · The Journal of Sex Research · doi:10.1080/00224499.2020.1821163 · Cited by 78
“There is a publication bias in scientific research, such that studies finding no statistically significant results are less likely to be published (Dwan et al., 2008, 2013).”
BackgroundPhysical activity and white matter hyperintensities: A systematic review of quantitative studies
Torres, E. R., Strack, E. F., Fernandez, C. E., Tumey, T. A., & Hitchcock, M. E. (2015) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2015.04.013 · Cited by 72
Prevention of orthodontic enamel demineralization: A systematic review with meta‐analyses
Tasios, T., Papageorgiou, S. N., Papadopoulos, M. A. A. P., Τσάπας, Α., & Haidich, A. (2019) · Orthodontics and Craniofacial Research · doi:10.1111/ocr.12322 · Cited by 80
Nawijn, F., Ham, W. H. W., Houwert, R. M., Groenwold, R. H. H., Hietbrink, F., & Smeeing, D. P. J. (2019) · BMC Emergency Medicine · doi:10.1186/s12873-019-0233-6 · Cited by 80
“Second, only 27% of the reviews reported an assessment of potential publication bias, despite it being known to be wide-spread within medical research [25, 27].”
BackgroundTime-to-update of systematic reviews relative to the availability of new evidence
Bashir, R., Surian, D., & Dunn, A. G. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0856-9 · Cited by 72
Thérond, A., Pezzoli, P., Abbas, M., Howard, A., Bowie, C. R., & Guimond, S. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.02.009 · Cited by 81
The value of decentralized clinical trials: Inclusion, accessibility, and innovation
Jean‐Louis, G., & Seixas, A. A. (2024) · Science · doi:10.1126/science.adq4994 · Cited by 78
“Taken together, these methodological issues point out the need for a more dynamic and inclusive approach to clinical research, one that DCTs are increasingly providing ( 11 ).”
BackgroundNaci, H., & Ioannidis, J. P. A. (2014) · The Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-010814-124614 · Cited by 74
Havlíček, J., Winternitz, J. C., & Roberts, S. C. (2020) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2019.0260 · Cited by 101
“In addition, some research areas may suffer from various types of publication bias [95].”
Willis, S., Neil, R., Mellick, M. C., & Wasley, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00393 · Cited by 55
Ivy leaf (Hedera helix) for acute upper respiratory tract infections: an updated systematic review
Sierocinski, E., Holzinger, F., & Chenot, J. (2021) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-021-03090-4 · Cited by 69
Ojerholm, E., Smith, A. M., Hwang, W., Baumann, B. C., Tucker, K. N., Lerner, S. P., Mamtani, R., Boursi, B., & Christodouleas, J. P. (2016) · Cancer · doi:10.1002/cncr.30422 · Cited by 71
“In addition to these analytic issues, biases in reporting and publication may also contribute to a surfeit of positive results.(49,50) Reporting bias includes both selective reporting and poor reporting.”
BackgroundFamily Planning Vouchers in Low and Middle Income Countries: A Systematic Review
Bellows, B., Bulaya, C., Inambwae, S., Lissner, C. L., Ali, M., & Bajracharya, A. (2016) · Studies in Family Planning · doi:10.1111/sifp.12006 · Cited by 69
Levis, B., Benedetti, A., Levis, A. W., Ioannidis, J. P. A., Shrier, I., Cuijpers, P., Gilbody, S., Kloda, L. A., McMillan, D., Patten, S. B., et al. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kww191 · Cited by 64
Li, G., Abbade, L. P. F., Nwosu, I., Jin, Y., Leenus, A., Maaz, M., Wang, M., Bhatt, M., Zielinski, L., Sanger, N., et al. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0465-7 · Cited by 70
“There were several systematic reviews assessing cohort studies that compared protocols or registrations and full reports [52, 53, 58, 59].”
BackgroundYoung, I., Greig, J., Wilhelm, B. J., & Waddell, L. A. (2019) · Journal of Food Protection · doi:10.4315/0362-028x.jfp-19-108 · Cited by 68
Miller, J. E., Ross, J. S., Wilenzick, M., & Mello, M. M. (2019) · BMJ · doi:10.1136/bmj.l4217 · Cited by 77
Melero, S., & Sánchez‐Sandoval, Y. (2017) · Children and Youth Services Review · doi:10.1016/j.childyouth.2017.05.006 · Cited by 72
Refining animal research: The Animal Study Registry
Bert, B., Heinl, C., Chmielewska, J., Schwarz, F., Grune, B., Hensel, A., Greiner, M., & Schönfelder, G. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000463 · Cited by 67
“, reporting only significant results as they are more likely to get published than negative results [26].”
BackgroundPiccoli, A., Rossettini, G., Cecchetto, S., Viceconti, A., Ristori, D., Turolla, A., Maselli, F., & Testa, M. (2018) · Journal of Functional Morphology and Kinesiology · doi:10.3390/jfmk3030040 · Cited by 74
“Furthermore, gray literature has not been included (gray literature bias), thus risking to exclude possible studies [75,76].”
BackgroundVale, C. L., Rydzewska, L. H. M., Rovers, M. M., Emberson, J. R., Gueyffier, F., Stewart, L. A., & on behalf of the Cochrane IPD Meta-analysis Methods Group (2015) · BMJ · doi:10.1136/bmj.h1088 · Cited by 66
Manera, K., Tong, A., Craig, J. C., Brown, E. A., Brunier, G., Dong, J., Dunning, T., Mehrotra, R., Naicker, S., Pecoits‐Filho, R., et al. (2017) · Peritoneal Dialysis International · doi:10.3747/pdi.2017.00022 · Cited by 72
“Another issue which often diminishes the reliability and validity of clinical trials is that of outcome reporting bias (27).”
BackgroundReporting quality of N-of-1 trials published between 1985 and 2013: a systematic review
Li, J., Gao, W., Punja, S., Ma, B., Vohra, S., Duan, N., Gabler, N. B., Yang, K., & Kravitz, R. L. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.11.016 · Cited by 68
Turner, L., Boutron, I., Hróbjartsson, A., Altman, D. G., & Moher, D. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-79 · Cited by 62
Planning and reporting of quality-of-life outcomes in cancer trials
Schandelmaier, S., Conen, K. L., von Elm, E. B., You, J. J., Blümle, A., Tomonaga, Y., Amstutz, A., Briel, M., Kasenda, B., Schandelmaier, S., et al. (2015) · Annals of Oncology · doi:10.1093/annonc/mdv283 · Cited by 72
“Failure to report pre-specified outcomes can seriously bias published evidence [26], especially because positive findings are more likely to be reported than negative findings [17].”
InfluentialEmpirical evidence about inconsistency among studies in a pair‐wise meta‐analysis
Rhodes, K. M., Turner, R. M., & Higgins, J. P. T. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1193 · Cited by 66
“…and there is inconsistency in the direction of effect, it would be misleading to give an average estimate for the intervention effect, particularly if study designs, within-study biases, and reporting biases are not taken into account (Turner et al , 2009; Welton et al , 2009; Dwan et al , 2013).”
Saqr, M., Elmoazen, R., Tedre, M., López‐Pernas, S., & Hirsto, L. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100437 · Cited by 69
Significance chasing in research practice: causes, consequences and possible solutions
Ware, J. J., & Munafò, M. R. (2015) · Addiction · doi:10.1111/add.12673 · Cited by 60
“At the same time, studies that generate null results are often never submitted for publication (6).”
Wray, F., Clarke, D. J., & Förster, A. (2017) · Disability and Rehabilitation · doi:10.1080/09638288.2017.1294206 · Cited by 74
Cognitive rehabilitation for adults with traumatic brain injury to improve occupational outcomes
Kumar, K. S., Samuelkamaleshkumar, S., Viswanathan, A., & Macaden, A. S. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007935.pub2 · Cited by 77
Electroconvulsive Therapy for Agitation and Aggression in Dementia: A Systematic Review
van den Berg, J. F., Kruithof, H. C., Kok, R., Verwijk, E., & Spaans, H. (2017) · American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2017.09.023 · Cited by 71
Wen, J., & Lei, L. (2022) · Applied Linguistics · doi:10.1093/applin/amab037 · Cited by 62
Cognitive and Affective Empathy in Autism Spectrum Disorders: A Meta-analysis
Fatima, M., & Babu, N. (2023) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-023-00364-8 · Cited by 77
“As significant results are more likely to be reported and published in the literature in contrast to non-significant findings (Dwan et al., 2013), the obtained mean effect sizes may be exaggerated, which renders it important to assess the possibility of publication bias.”
Hexavalent chromium and stomach cancer: a systematic review and meta-analysis
Suh, M., Wikoff, D. S., Lipworth, L. P., Goodman, M. G., Fitch, S., Mittal, L., Ring, C., & Proctor, D. M. (2019) · Critical Reviews in Toxicology · doi:10.1080/10408444.2019.1578730 · Cited by 67
“Meta-analyses are based on published studies, and it is known that studies with null results are less likely to be published (Thornton and Lee 2000; Dwan et al. 2013).”
BackgroundHéroux, M. E., Taylor, J. L., & Gandevia, S. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144151 · Cited by 66
Vascular Access Outcomes Reported in Maintenance Hemodialysis Trials: A Systematic Review
Viecelli, A. K., O’Lone, E., Sautenet, B., Craig, J. C., Tong, A., Chemla, E. S., Hooi, L. S., Lee, T. C., Lok, C. E., Polkinghorne, K. R., et al. (2017) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2017.09.018 · Cited by 68
Veile, A., Zimmermann, H., Lorenz, E., & Becher, H. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-016589 · Cited by 50
Brown, A. W., Aslibekyan, S. W., Bier, D. M., da Silva, R. F., Hoover, A. W., Klurfeld, D. M., Loken, E., Mayo‐Wilson, E., Menachemi, N., Pavela, G., et al. (2021) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2021.1985427 · Cited by 65
Page, M. J., McKenzie, J. E., Green, S., Beaton, D., Jain, N. B., Lenza, M., Verhagen, A. P., Surace, S. J., Deitch, J. M., & Buchbinder, R. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.006 · Cited by 67
Escribà‐Agüir, V., Rodríguez‐Gómez, M., & Ruíz-Pérez, I. (2016) · Cancer Epidemiology · doi:10.1016/j.canep.2016.07.009 · Cited by 64
Butcher, N. J., Monsour, A., Mew, E. J., Szatmari, P., Pierro, A., Kelly, L. E., Farid-Kapadia, M., Chee-a-tow, A., Saeed, L., Monga, S., et al. (2019) · Trials · doi:10.1186/s13063-019-3248-0 · Cited by 57
Self-citationRethinking the assessment of risk of bias due to selective reporting: a cross-sectional study
Page, M. J., & Higgins, J. P. T. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0289-2 · Cited by 60
“For example, participant deaths may be counted and compared between intervention groups but trialists present no data because the effect favoured the comparator or only state that the between-group difference was not statistically significant; in this case, the summary statistics needed to include the trial in a meta-…”
BackgroundWieschowski, S., Riedel, N., Wollmann, K., Kahrass, H., Müller-Ohlraun, S., Schuermann, C., Kelley, S., Kszuk, U., Siegerink, B., Dirnagl, U., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.06.002 · Cited by 60
Dechartres, A., Atal, I., Riveros, C., Meerpohl, J., & Ravaud, P. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1517 · Cited by 59
An Action Plan for Diamond Open Access
Borrell-Damián, L., & Rooryck, J. (2022) · doi:10.4000/books.oep.16147 · Cited by 60
Meerpohl, J. J., Schell, L. K., Bassler, D., Gallus, S., Kleijnen, J., Kulig, M., La Vecchia, C., Marušić, A., Ravaud, P., Reis, A., et al. (2015) · BMJ Open · doi:10.1136/bmjopen-2014-006666 · Cited by 57
Composing a new song for trials: the Standardized Outcomes in Nephrology (SONG) initiative
Tong, A., Craig, J. C., Nagler, E. V., Biesen, W. V., Manns, B., Hemmelgarn, B. R., Wheeler, D. C., Gill, J., Tugwell, P., Pecoits‐Filho, R., et al. (2017) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfx288 · Cited by 64
“Outcome reporting bias, whereby authors cherry-pick the outcomes they report on the basis of favourable results, may also occur when there is not a standardized list of outcomes measured and reported [5, 6].”
BackgroundSmartwatch interventions in healthcare: A systematic review of the literature
Triantafyllidis, A. K., Kondylakis, H., Katehakis, D. G., Kouroubali, A., Alexiadis, A., Segkouli, S., Votis, K., & Tzovaras, D. K. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105560 · Cited by 75
“Publication bias in this review is possible, mainly because studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported [50].”
BackgroundLee, H. W., Jun, J. H., Kil, K.-J., Ko, B.-S., Lee, C. H., & Lee, M. S. (2015) · Maturitas · doi:10.1016/j.maturitas.2015.11.013 · Cited by 58
Faggion, C. M., Cullinan, M. P., & Atieh, M. A. (2016) · Journal of Periodontal Research · doi:10.1111/jre.12358 · Cited by 69
Increasing the Credibility of Political Science Research: A Proposal for Journal Reforms
Nyhan, B. J. (2015) · PS Political Science & Politics · doi:10.1017/s1049096515000463 · Cited by 58
“…to ensue even when preregistration is commonly practiced, as studies frequently have found in examining results from clinical trials (see, e.g., Dwan et al. 2013).4 A better practice would be for journals to offer authors an option in which articles with prespecified analysis plans would be…”
BackgroundFusaroli, R., Grossman, R. B., Bilenberg, N., Cantio, C., Jepsen, J. R. M., & Weed, E. (2021) · Autism Research · doi:10.1002/aur.2661 · Cited by 68
“While a few different techniques have been developed to assess the heterogeneity between studies and potential publication biases (Dwan et al., 2013), they are not a solution to the issue of more reliably estimating the true effect, and the critique remains valid (Rocca & Yarkoni, 2021).”
MethodThree Decades of Climate Mitigation Policy: What Has It Delivered?
Hoppe, J., Hinder, B., Rafaty, R., Patt, A., & Grubb, M. J. (2023) · Annual Review of Environment and Resources · doi:10.1146/annurev-environ-112321-103821 · Cited by 85
Malički, M., & Marušić, A. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.002 · Cited by 62
“But even in such a group of experts, the prevalence of reported nonpublication was similar to that in several studies performed since the 1980s [14].”
ResultEconomics of Obesity — Learning from the Past to Contribute to a Better Future
Ananthapavan, J., Sacks, G., Moodie, M., & Carter, R. (2014) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph110404007 · Cited by 67
“Therefore short term summative evaluations, undertaken before the intervention has “matured” will not adequately demonstrate their true effectiveness [29]; and hence are less likely to be published [30].”
BackgroundResilient Urbanization: A Systematic Review on Urban Discourse in Pakistan
Latif, A., & Yu, T. (2020) · Urban Science · doi:10.3390/urbansci4040076 · Cited by 69
“Grey literature like unpublished reports, dissertations, and abstracts were carefully considered to minimize the research bias and maximize validity [20,21].”
Background
Metadata
- Authors
- Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group
- Affiliations
- University of Liverpool (GB)
- Corresponding author
- Dwan, Kerry
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-07-05
- Volume
- 8
- Issue
- 7
- Pages
- e66844
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3702538
- OpenAlex
- W2002062153
- Cited by
- 1,127
- References
- 69
- Citations by year
- 2026: 75 · 2025: 84 · 2024: 64 · 2023: 71 · 2022: 127 · 2021: 98 · 2020: 112 · 2019: 89 · 2018: 83 · 2017: 102 · 2016: 89 · 2015: 70 · 2014: 57 · 2013: 5
- FWCI
- 15.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion)
- SERVAL (Université de Lausanne) (publishedVersion, cc-by)
- IRIS (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Tufts Medical Center
- King's College London
- University of Oxford
- University of Bristol
- University of Sydney
- Queen's University
- Women's College Hospital
- Centre Hospitalier Universitaire Vaudois
- Hospices Civils de Lyon
- Moorfields Eye Hospital NHS Foundation Trust
- School of Medicine, Stanford University
- Tufts University School of Medicine
- Medical Research Council
- National Health and Medical Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- publication bias, selective reporting, randomized controlled trials, meta-analysis validity, trial reporting quality, primary outcome reporting
- MeSH terms
- Humans, Publications, Cohort Studies, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.