The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses.
Abstract
The issue of publication bias in psychological science is one that has remained difficult to address despite decades of discussion and debate. The current article examines a sample of 91 recent meta-analyses published in American Psychological Association and Association for Psychological Science journals and the methods used in these analyses to identify and control for publication bias. Of the 91 studies analyzed, 64 (70%) made some effort to analyze publication bias, and 26 (41%) reported finding evidence of bias. Approaches to controlling publication bias were heterogeneous among studies. Of these studies, 57 (63%) attempted to find unpublished studies to control for publication bias. Nonetheless, those studies that included unpublished studies were just as likely to find evidence for publication bias as those that did not. Furthermore, authors of meta-analyses themselves were overrepresented in unpublished studies acquired, as compared with published studies, suggesting that searches for unpublished studies may increase rather than decrease some sources of bias. A subset of 48 meta-analyses for which study sample sizes and effect sizes were available was further analyzed with a conservative and newly developed tandem procedure of assessing publication bias. Results indicated that publication bias was worrisome in about 25% of meta-analyses. Meta-analyses that included unpublished studies were more likely to show bias than those that did not, likely due to selection bias in unpublished literature searches. Sources of publication bias and implications for the use of meta-analysis are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A meta-analytic test of intergroup contact theory.
Pettigrew, T. F., & Tropp, L. R. (2006) · Journal of personality and social psychology · doi:10.1037/0022-3514.90.5.751 · Cited by 9,708
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Selective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Pharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147
Experimental disclosure and its moderators: A meta-analysis.
Frattaroli, J. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.6.823 · Cited by 1,502
meta-analysis bias in location and selection of studies
Egger, M., & Smith, G. D. (1998) · BMJ · doi:10.1136/bmj.316.7124.61 · Cited by 1,374
A comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716
Slavin, R. E. (2008) · Educational Researcher · doi:10.3102/0013189x08314117 · Cited by 560
Cited by
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
Does low self-esteem predict depression and anxiety? A meta-analysis of longitudinal studies.
Sowislo, J. F., & Orth, U. (2013) · Psychological Bulletin · doi:10.1037/a0028931 · Cited by 2,123
The consequences of perceived discrimination for psychological well-being: A meta-analytic review.
Schmitt, M. T., Branscombe, N. R., Postmes, T., & Garcia, A. (2014) · Psychological Bulletin · doi:10.1037/a0035754 · Cited by 1,804
The effect of testing versus restudy on retention: A meta-analytic review of the testing effect.
Rowland, C. A. (2014) · Psychological Bulletin · doi:10.1037/a0037559 · Cited by 1,066
Lundahl, B., Moleni, T., Burke, B. L., Butters, R., Tollefson, D., Butler, C., & Rollnick, S. (2013) · Patient Education and Counseling · doi:10.1016/j.pec.2013.07.012 · Cited by 909
The relationship between materialism and personal well-being: A meta-analysis
Dittmar, H., Bond, R., Hurst, M., & Kasser, T. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0037409 · Cited by 890
“Thus, we were successful in sampling the grey literature (Rothstein & Hopewell, 2009), thereby increasing our confidence that the results are not affected unduly by publication bias (see Ferguson & Brannick, 2012; Rothstein & Bushman, 2012; see also final section of Results).”
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Accuracy of teachers' judgments of students' academic achievement: A meta-analysis
Südkamp, A., Kaiser, J., & Möller, J. (2012) · Journal of Educational Psychology · doi:10.1037/a0027627 · Cited by 696
Gender differences in the prevalence of nonsuicidal self-injury: A meta-analysis
Bresin, K., & Schoenleber, M. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.02.009 · Cited by 678
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603
Self-citationWhy Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563
Response rates for CBT for anxiety disorders: need for standardized criteria
Loerinc, A. G., Meuret, A. E., Twohig, M. P., Rosenfield, D., Bluett, E. J., & Craske, M. G. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.08.004 · Cited by 592
Methodological guidance paper: High-quality meta- analysis in a systematic review
Pigott, T. D., & Polanin, J. R. (2020) · Review of Educational Research · doi:10.3102/0034654319877153 · Cited by 568
Ferguson, C. J. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615592234 · Cited by 524
Self-citationKöchling, A., & Wehner, M. C. (2020) · Business Research · doi:10.1007/s40685-020-00134-w · Cited by 511
Development of self-esteem from age 4 to 94 years: A meta-analysis of longitudinal studies.
Orth, U., Erol, R. Y., & Luciano, E. C. (2018) · Psychological Bulletin · doi:10.1037/bul0000161 · Cited by 438
Kühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399
The Stability and Change of Loneliness Across the Life Span: A Meta-Analysis of Longitudinal Studies
Mund, M., Freuding, M. M., Möbius, K., Horn, N., & Neyer, F. J. (2020) · Personality and Social Psychology Review · doi:10.1177/1088868319850738 · Cited by 417
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410
The influence of subjective aging on health and longevity: a meta-analysis of longitudinal data
Westerhof, G. J., Miche, M., Brothers, A. F., Barrett, A. E., Diehl, M., Montepare, J. M., Wahl, H.-W., & Wurm, S. (2014) · Psychology and Aging · doi:10.1037/a0038016 · Cited by 403
Carter, E. C., & McCullough, M. E. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00823 · Cited by 381
Effects of video-game play on information processing: a meta-analytic investigation
Powers, K. L., Brooks, P. J., Aldrich, N. J., Palladino, M. A., & Alfieri, L. (2013) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0418-z · Cited by 348
A Meta-Analytic Review of Social Identification and Health in Organizational Contexts
Steffens, N. K., Haslam, S. A., Schuh, S. C., Jetten, J., & van Dick, R. (2017) · Personality and Social Psychology Review · doi:10.1177/1088868316656701 · Cited by 311
Ferguson, C. J. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.11.002 · Cited by 295
Self-citationMeta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
Meta-analytic structural equation modeling with moderating effects on SEM parameters.
Jak, S., & Cheung, M. W. L. (2020) · Psychological Methods · doi:10.1037/met0000245 · Cited by 262
Scherrer, V., & Preckel, F. (2019) · Review of Educational Research · doi:10.3102/0034654318819127 · Cited by 251
Estimating the Difference Between Published and Unpublished Effect Sizes: A Meta-Review
Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016) · Review of Educational Research · doi:10.3102/0034654315582067 · Cited by 234
A Guide to Conducting a Meta-Analysis
Cheung, M. W.-L., & Vijayakumar, R. (2016) · Neuropsychology Review · doi:10.1007/s11065-016-9319-z · Cited by 239
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650874 · Cited by 219
Maes, M., Nelemans, S. A., Danneel, S., Fernández-Castilla, B., Van den Noortgate, W., Goossens, L., & Vanhalst, J. (2019) · Developmental Psychology · doi:10.1037/dev0000719 · Cited by 207
The role of media literacy in body dissatisfaction and disordered eating: A systematic review
McLean, S. A., Paxton, S. J., & Wertheim, E. H. (2016) · Body Image · doi:10.1016/j.bodyim.2016.08.002 · Cited by 190
Fairbairn, C. E., Briley, D. A., Kang, D., Fraley, R. C., Hankin, B. L., & Ariss, T. (2018) · Psychological Bulletin · doi:10.1037/bul0000141 · Cited by 182
Guest, E., Costa, B., Williamson, H., Meyrick, J., Halliwell, E., & Harcourt, D. (2019) · Body Image · doi:10.1016/j.bodyim.2019.04.002 · Cited by 184
Parental sensitivity and child behavioral problems: A meta-analytic review
Cooke, J. E., Deneault, A.-A., Devereux, C., Eirich, R., Fearon, R. M. P., & Madigan, S. (2022) · Child Development · doi:10.1111/cdev.13764 · Cited by 173
Is the evidence from racial bias shooting task studies a smoking gun? Results from a meta-analysis
Mekawi, Y., & Bresin, K. (2015) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.08.002 · Cited by 166
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
A meta-analysis of signaling principle in multimedia learning environments
Alpizar, D., Adesope, O. O., & Wong, R. M. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-020-09748-7 · Cited by 157
Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number
Fragkos, K. C., Tsagris, M., & Frangos, C. C. (2014) · International Scholarly Research Notices · doi:10.1155/2014/825383 · Cited by 159
Atypicality of the N170 Event-Related Potential in Autism Spectrum Disorder: A Meta-analysis
Kang, E., Keifer, C. M., Levy, E. J., Foss-Feig, J. H., McPartland, J. C., & Lerner, M. D. (2018) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2017.11.003 · Cited by 144
Technology acceptance research: Meta-analysis
Marikyan, D., Papagiannidis, S., & Stewart, G. (2026) · Journal of Information Science · doi:10.1177/01655515231191177 · Cited by 140
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122
Coronado-Montoya, S., Levis, A. W., Kwakkenbos, L., Steele, R. J., Turner, E. H., & Thombs, B. D. (2016) · PLOS ONE · doi:10.1371/journal.pone.0153220 · Cited by 107
Sirois, F. M., Molnar, D. S., & Hirsch, J. K. (2017) · European Journal of Personality · doi:10.1002/per.2098 · Cited by 108
Schäfer, S. K., Wirth, B. E., Staginnus, M., Becker, N., Michael, T., & Sopp, M. R. (2020) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2020.101280 · Cited by 85
Chow, J. C., & Ekholm, E. (2018) · Educational Psychology Review · doi:10.1007/s10648-018-9437-7 · Cited by 80
Like this meta-analysis: Screen media and mental health.
Ferguson, C. J., Kaye, L. K., Branley-Bell, D., Markey, P., Ivory, J. D., Klisanin, D., Elson, M., Smyth, M., Hogg, J. L., McDonnell, D., et al. (2022) · Professional Psychology: Research and Practice · doi:10.1037/pro0000426 · Cited by 78
Self-citationStucke, N. J., & Doebel, S. (2024) · Psychological Bulletin · doi:10.1037/bul0000445 · Cited by 69
ELICITED IMITATION TASKS AS A MEASURE OF L2 PROFICIENCY: A META-ANALYSIS
Kostromitina, M., & Plonsky, L. (2022) · Studies in Second Language Acquisition · doi:10.1017/s0272263121000395 · Cited by 61
Systematic Review of Digital Interventions for Adolescent and Young Adult Women's Body Image
Mahon, C., & Seekis, V. (2022) · Frontiers in Global Women's Health · doi:10.3389/fgwh.2022.832805 · Cited by 34
Spindle-dependent memory consolidation in healthy adults: a meta-analysis
Kumral, D., Matzerath, A., Leonhart, R., & Schönauer, M. (2023) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2023.108661 · Cited by 32
The Effects of Mentoring Programs on Emotional Well-Being in Youth: a Meta-analysis
Claro, A., & Perelmiter, T. (2022) · Contemporary School Psychology · doi:10.1007/s40688-021-00377-2 · Cited by 28
Sotola, L. K., & Credé, M. (2022) · European Journal of Social Psychology · doi:10.1002/ejsp.2858 · Cited by 28
A retrospective analysis of dissemination biases in the brief alcohol intervention literature.
Tanner-Smith, E. E., & Polanin, J. R. (2015) · Psychology of Addictive Behaviors · doi:10.1037/adb0000014 · Cited by 26
“…some researchers have recently advocated for the systematic exclusion of dissertations or other unpublished gray literature from meta-analyses (Ferguson & Brannick, 2012), we agree with other scholars (Rothstein & Bushman, 2012) who contend that systematically excluding gray literature from…”
ResultLinking attachment to empathy in childhood and adolescence: A multilevel meta-analysis
Li, S., Ran, G., & Chen, X. (2021) · Journal of Social and Personal Relationships · doi:10.1177/02654075211031006 · Cited by 23
Not Published Is Not Perished: Addressing Publication Bias in Meta-Analytic Studies in Communication
Sun, Y., & Pan, Z. (2020) · Human Communication Research · doi:10.1093/hcr/hqz015 · Cited by 18
Ferguson, C. J., & Smith, S. (2024) · Aggression and Violent Behavior · doi:10.1016/j.avb.2023.101905 · Cited by 14
Self-citation
Metadata
- Authors
- Ferguson, C. J., & Brannick, M. T.
- Venue
- Psychological Methods
- Published
- 2012
- Volume
- 17
- Issue
- 1
- Pages
- 120-128
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2124519965
- Cited by
- 493
- References
- 13
- FWCI
- 24.99
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.