The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Publication bias: what is it? How do we measure it? How do we avoid it?
Open accessAbstract
Publication bias occurs when results of published studies are systematically different from results of unpublished studies. The term "dissemination bias" has also been recommended to describe all forms of biases in the research-dissemination process, including outcome-reporting bias, time-lag bias, gray-literature bias, full-publication bias, language bias, citation bias, and media-attention bias. We can measure publication bias by comparing the results of published and unpublished studies addressing the same question. Following up cohorts of studies from inception and comparing publication levels in studies with statistically significant or "positive" results suggested greater odds of formal publication in those with such results, compared to those without. Within reviews, funnel plots and related statistical methods can be used to indicate presence or absence of publication bias, although these can be unreliable in many circumstances. Methods of avoiding publication bias, by identifying and including unpublished outcomes and unpublished studies, are discussed and evaluated. These include searching without limiting by outcome, searching prospective trials registers, searching informal sources, including meeting abstracts and PhD theses, searching regulatory body websites, contacting authors of included studies, and contacting pharmaceutical or medical device companies for further studies. Adding unpublished studies often alters effect sizes, but may not always eliminate publication bias. The compulsory registration of all clinical trials at inception is an important move forward, but it can be difficult for reviewers to access data from unpublished studies located this way. Publication bias may be reduced by journals by publishing high-quality studies regardless of novelty or unexciting results, and by publishing protocols or full-study data sets. No single step can be relied upon to fully overcome the complex actions involved in publication bias, and a multipronged approach is required by researchers, patients, journal editors, peer reviewers, research sponsors, research ethics committees, and regulatory and legislation authorities. Keywords: publication bias, reporting bias, research-dissemination bias, evidence synthesis, systematic review, meta-analysis
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · cited 1×
“15 Furthermore, negative results being misinterpreted and reported as positive results in published clinical trials is also prevalent.”
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“For example, studies approved by a research ethics committee or institutional review board were followed up to investigate the extent of publication bias.(42,43) A systematic review of publication bias included studies that tracked a cohort of studies and reported the rate of publication by results.”
Selective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435
Avoidable waste in the production and reporting of research evidence
Chalmers, I., & Glasziou, P. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60329-9 · Cited by 2,208
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Summing up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when the publication of studies depends on the nature and direction of the results, so that published studies may be systematically different from those of unpublished studies.(7) In general, studies with statistically significant or positive results are more likely to be published than those w…”
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133
Calcium supplementation during pregnancy for preventing hypertensive disorders and related problems
Hofmeyr, G. J., Lawrie, T. A., Atallah, Á. N., & Duley, L. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001059.pub3 · Cited by 1,062
Whittington, C. J., Kendall, T., Fonagy, P., Cottrell, D., Cotgrove, A., & Boddington, E. (2004) · The Lancet · doi:10.1016/s0140-6736(04)16043-1 · Cited by 1,058
The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052
Dissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039
Self-citationFull publication of results initially presented in abstracts.
Scherer, R. W., Langenberg, P., & von Elm, E. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub3 · Cited by 997
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 1×
“…results might be underreported because the results of 97% of psychological studies published in four journals were statistically significant, 35 and the same author in 1995 concluded that the practices leading to publication bias in psychological research had not changed over a period of 30…”
Dickersin, K. (1992) · JAMA · doi:10.1001/jama.1992.03480030052036 · Cited by 854
Comparison of Treatment Effects between Animal Experiments and Clinical Trials: Systematic Review
Perel, P., Roberts, I., Sena, E., Wheble, P., Briscoe, C., Sandercock, P., Macleod, M., Mignini, L. E., Jayaram, P., & Khan, K. S. (2007) · BMJ · doi:10.1136/bmj.39048.407928.be · Cited by 827
Pooling of Animal Experimental Data Reveals Influence of Study Design and Publication Bias
Macleod, M. R., O’Collins, T., Howells, D. W., & Donnan, G. A. (2004) · Stroke · doi:10.1161/01.str.0000125719.25853.20 · Cited by 730
Sterne, J. A., Egger, M., & Moher, D. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch10 · Cited by 651
Hooper, L., Kay, C., Abdelhamid, A., Kroon, P. A., Cohn, J. S., Rimm, E. B., & Cassidy, A. (2012) · The American Journal of Clinical Nutrition · doi:10.3945/ajcn.111.023457 · Cited by 610 · cited 1×
“In a recent systematic review of RCTs, we established contact with authors of 19 trials, and included additional outcomes for twelve studies (of the 42 included RCTs).(52) However a further twelve trials had clearly measured but not reported at least one outcome of interest for our review (or reported insufficiently t…”
Self-citationAvoidable Waste in the Production and Reporting of Research Evidence
Chalmers, I., & Glasziou, P. (2009) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e3181c3020d · Cited by 600
Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children
Jefferson, T., Jones, M. A., Doshi, P., Del Mar, C. B., Heneghan, C. J., Hama, R., & Thompson, M. J. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008965.pub3 · Cited by 594
Lefebvre, C., Manheimer, E., & Glanville, J. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch6 · Cited by 559
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524
Ferguson, C. J., & Brannick, M. T. (2012) · Psychological Methods · doi:10.1037/a0024445 · Cited by 493
Publication and Related Biases
Song, F., Eastwood, A., Gilbody, S., Duley, L., & Sutton, A. (2001) · The Advanced Handbook of Methods in Evidence Based Healthcare · doi:10.4135/9781848608344.n21 · Cited by 455
Self-citationSterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
Reporting bias in medical research - a narrative review
McGauran, N., Wieseler, B., Kreis, J., Schüler, Y.-B., Kölsch, H., & Kaiser, T. (2010) · Trials · doi:10.1186/1745-6215-11-37 · Cited by 452
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Chan, A.-W. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041086 · Cited by 444
Trial Publication after Registration in ClinicalTrials.Gov: A Cross-Sectional Analysis
Ross, J. S., Mulvey, G. K., Hines, E. M., Nissen, S. E., & Krumholz, H. M. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000144 · Cited by 401
Outcome Reporting Among Drug Trials Registered in ClinicalTrials.gov
Bourgeois, F. T., Murthy, S., & Mandl, K. D. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-153-3-201008030-00006 · Cited by 332
Ioannidis, J. P. A., & Trikalinos, T. A. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.019 · Cited by 325
Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children
Jefferson, T., Jones, M. A., Doshi, P., Del Mar, C. B., Hama, R., Thompson, M., Spencer, E. A., Onakpoya, I., Mahtani, K. R., Nunan, D. N., et al. (2014) · Sao Paulo Medical Journal · doi:10.1590/1516-3180.20141324t2 · Cited by 319
Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses
Hart, B., Lundh, A., & Bero, L. (2012) · BMJ · doi:10.1136/bmj.d7202 · Cited by 255
Dickersin, K. (1992) · JAMA · doi:10.1001/jama.267.3.374 · Cited by 191
Song, F., Parekh-Bhurke, S., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., Hing, C. B., & Harvey, I. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-79 · Cited by 175
Self-citationPsaty, B. M., & Kronmal, R. A. (2008) · JAMA · doi:10.1001/jama.299.15.1813 · Cited by 173
Calcium supplementation during pregnancy for preventing hypertensive disorders and related problems
Hofmeyr, G. J., Atallah, Á. N., & Duley, L. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001059.pub2 · Cited by 162
Sterling, T. D., Rosenbaum, W. L., & Jj, W. (1995) · The American Statistician · doi:10.2307/2684823 · Cited by 158
Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy
THIERS, B. H. (2009) · Yearbook of Dermatology and Dermatologic Surgery · doi:10.1016/s0093-3619(08)79042-7 · Cited by 157
Why we need easy access to all data from all clinical trials and how to accomplish it
Gøtzsche, P. C. (2011) · Trials · doi:10.1186/1745-6215-12-249 · Cited by 148
ter Riet, G., Korevaar, D. A., Leenaars, M., Sterk, P. J., Noorden, C. J. F. V., Bouter, L. M., Lutter, R., Elferink, R. P. J. O., & Hooft, L. (2012) · PLoS ONE · doi:10.1371/journal.pone.0043404 · Cited by 134
Vera-Badillo, F. E., Shapiro, R., Ocana, A., Amir, E., & Tannock, I. F. (2013) · Annals of Oncology · doi:10.1093/annonc/mds636 · Cited by 117 · cited 1×
“”(15) Furthermore, negative results being misinterpreted and reported as positive results in published clinical trials is also prevalent.(16,17)”
Huić, M., Marušić, M., & Marušić, A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0025258 · Cited by 115
Awasthi, S., Peto, R., Read, S., Richards, S. M., Pande, V. K., Bundy, D. A. P., & the DEVTA team (2013) · The Lancet · doi:10.1016/s0140-6736(12)62126-6 · Cited by 115
Academic Freedom in Clinical Research
Nathan, D. G., & Weatherall, D. J. (2002) · New England Journal of Medicine · doi:10.1056/nejmsb020394 · Cited by 108
Hannink, G., Gooszen, H. G., & Rovers, M. M. (2013) · Annals of Surgery · doi:10.1097/sla.0b013e3182864fa3 · Cited by 103
Searching for unpublished trials in Cochrane reviews may not be worth the effort
van Driel, M., Sutter, A. D., Maeseneer, J. D., & Christiaens, T. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.09.010 · Cited by 100
Trial Registration Numbers Are Underreported in Biomedical Publications
van de Wetering, F. T., Scholten, R. J. P. M., Haring, T., Clarke, M., & Hooft, L. (2012) · PLoS ONE · doi:10.1371/journal.pone.0049599 · Cited by 79
Publication bias in psychological science: Comment on Ferguson and Brannick (2012).
Rothstein, H. R., & Bushman, B. J. (2012) · Psychological Methods · doi:10.1037/a0027128 · Cited by 74
You, B., Gan, H. K., Pond, G. R., & Chen, E. X. (2011) · Journal of Clinical Oncology · doi:10.1200/jco.2011.37.0890 · Cited by 74
Publication Bias in the Anesthesiology Literature
Oliveira, G. S. D., Chang, R., Kendall, M. C., Fitzgerald, P. C., & McCarthy, R. J. (2012) · Anesthesia & Analgesia · doi:10.1213/ane.0b013e3182468fc6 · Cited by 73
Assessing the potential for outcome reporting bias in a review: a tutorial
Dwan, K. M., Gamble, C. L., Kolamunnage‐Dona, R., Mohammed, S., Powell, C., & Williamson, P. R. (2010) · Trials · doi:10.1186/1745-6215-11-52 · Cited by 70
Rethinking credible evidence synthesis
Doshi, P., Jones, M., & Jefferson, T. (2012) · BMJ · doi:10.1136/bmj.d7898 · Cited by 60
Clinical trial data for all drugs in current use
Godlee, F. (2012) · BMJ · doi:10.1136/bmj.e7304 · Cited by 57
Academia and industry: lessons from the unfortunate events in Toronto
Nathan, D., & Weatherall, D. J. (1999) · The Lancet · doi:10.1016/s0140-6736(99)00072-0 · Cited by 56
Bhurke, S., Kwok, C. S., Pang, C. L., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., Hing, C. B., Harvey, I. M., & Song, F. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.022 · Cited by 54
Self-citationRevéiz, L., Chan, A., Krleža-Jerić, K., Granados, C., Pinart, M., Etxeandia‐Ikobaltzeta, I., Rada, D., Martínez‐Alonso, M., Bonfill, X., & Cardona, A. F. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012484 · Cited by 46
Bax, L., & Moons, K. G. M. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.003 · Cited by 43
Positive‐outcome Bias: Comparison of Emergency Medicine and General Medicine Literatures
Moscati, R. M., Jehle, D. V. K., Ellis, D. G., Fiorello, A. B., & Landi, M. (1994) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.1994.tb02443.x · Cited by 43
Sood, A., Knudsen, K., Sood, R., Wahner‐Roedler, D. L., Barnes, S. A., Bardia, A., & Bauer, B. A. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.01.009 · Cited by 41
Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children
(2012) · Prescriber · doi:10.1002/psb.878 · Cited by 40
The publication process itself was the major cause of publication bias in genetic epidemiology
Calnan, M., Smith, G. D., & Sterne, J. A. C. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.05.002 · Cited by 31
Boisen, E. (1979) · PubMed · Cited by 26
Selective Outcome Reporting as a Source of Bias in Reviews of Comparative Effectiveness
Norris, S. L., Holmer, H. K., Ogden, L. A., Fu, R. F., Abou-Setta, A. M., Viswanathan, M., & McPheeters, M. L. (2012) · Europe PMC (PubMed Central) · Cited by 25
Publication bias in five dental implant journals: an observation from 2005 to 2009.
Yuan, J. C., Shyamsunder, N., Barao, V. A. R., Lee, D. J., & Sukotjo, C. (2011) · PubMed · Cited by 23
Data on neuraminidase inhibitors were made available
Panahloo, Z. (2004) · BMJ · doi:10.1136/bmj.328.7438.523-a · Cited by 3
Cited by
Genome-wide association studies
Uffelmann, E., Huang, Q. Q., Munung, N. S., de Vries, J., Okada, Y., Martin, A. R., Martin, H. C., Lappalainen, T., & Posthuma, D. (2021) · Nature Reviews Methods Primers · doi:10.1038/s43586-021-00056-9 · Cited by 1,849
Náfrádi, L., Nakamoto, K., & Schulz, P. J. (2017) · PLOS ONE · doi:10.1371/journal.pone.0186458 · Cited by 543
Evidence-based intervention sustainability strategies: a systematic review
Hailemariam, M., Bustos, T., Montgomery, B., Barajas, R., Evans, L. B., & Drahota, A. (2019) · Implementation Science · doi:10.1186/s13012-019-0910-6 · Cited by 398
Kostyrka-Allchorne, K., Cooper, N. R., & Simpson, A. (2017) · Developmental Review · doi:10.1016/j.dr.2016.12.002 · Cited by 302
A meta-analysis of the ecological and economic outcomes of mangrove restoration
Su, J., Friess, D. A., & Gasparatos, A. (2021) · Nature Communications · doi:10.1038/s41467-021-25349-1 · Cited by 286
Metadata
- Authors
- Song, F., Hooper, & Loke, Y.
- Venue
- Open Access Journal of Clinical Trials
- Publisher
- Informa UK Limited
- Published
- 2013-07
- Pages
- 71
- ISSN
- 1179-1519 (ISSN-L)
- OpenAlex
- W1992074138
- Cited by
- 275
- References
- 70
- FWCI
- 7.89
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.