Publication bias in psychological science: Comment on Ferguson and Brannick (2012).
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
It is well documented that studies reporting statistically significant results are more likely to be published than are studies reporting nonsignificant results--a phenomenon called publication bias. Publication bias in meta-analytic reviews should be identified and reduced when possible. Ferguson and Brannick (2012) argued that the inclusion of unpublished articles is ineffective and possibly counterproductive as a means of reducing publication bias in meta-analyses. We show how idiosyncratic choices on the part of Ferguson and Brannick led to an erroneous conclusion. We demonstrate that their key finding--that publication bias was more likely when unpublished studies were included--may be an artifact of the way they assessed publication bias. We also point out how the lack of transparency about key choices and the absence of information about critical features of Ferguson and Brannick's sample and procedures might have obscured readers' ability to assess the validity of their claims. Furthermore, we demonstrate that many of the claims they made are without empirical support, even though they could have tested these claims empirically, and that these claims may be misleading. With their claim that addressing publication bias introduces subjectivity and bias into meta-analysis, they ignored a large body of evidence showing that including unpublished studies that meet the inclusion criteria of a meta-analysis decreases (rather than increases) publication bias. Rather than exclude unpublished studies, we recommend that meta-analysts code study characteristics related to methodological quality (e.g., experimental vs. nonexperimental design) and test whether these factors influence the meta-analytic results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Anderson, C. A., Shibuya, A., Ihori, N., Swing, E. L., Bushman, B. J., Sakamoto, A., Rothstein, H. R., & Saleem, M. (2010) · Psychological Bulletin · doi:10.1037/a0018251 · Cited by 2,048
Self-citationThe Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Research Synthesis and Meta-Analysis: A Step-by-Step Approach
Cooper, H. (2017) · doi:10.4135/9781071878644 · Cited by 1,716
Experimental disclosure and its moderators: A meta-analysis.
Frattaroli, J. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.6.823 · Cited by 1,502
meta-analysis bias in location and selection of studies
Egger, M., & Smith, G. D. (1998) · BMJ · doi:10.1136/bmj.316.7124.61 · Cited by 1,374
A comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286
The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052
Cited by
The effect of testing versus restudy on retention: A meta-analytic review of the testing effect.
Rowland, C. A. (2014) · Psychological Bulletin · doi:10.1037/a0037559 · Cited by 1,066
The relationship between materialism and personal well-being: A meta-analysis
Dittmar, H., Bond, R., Hurst, M., & Kasser, T. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0037409 · Cited by 890
“Thus, we were successful in sampling the grey literature (Rothstein & Hopewell, 2009), thereby increasing our confidence that the results are not affected unduly by publication bias (see Ferguson & Brannick, 2012; Rothstein & Bushman, 2012; see also final section of Results).”
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603
Subjective well-being and academic achievement: A meta-analysis
Bücker, S., Nuraydin, S., Simonsmeier, B. A., Schneider, M., & Luhmann, M. (2018) · Journal of Research in Personality · doi:10.1016/j.jrp.2018.02.007 · Cited by 569
Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
Metadata
- Authors
- Rothstein, H. R., & Bushman, B. J.
- Venue
- Psychological Methods
- Published
- 2012
- Volume
- 17
- Issue
- 1
- Pages
- 129-136
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W1970092066
- Cited by
- 68
- References
- 15
- FWCI
- 7.08
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Publication bias in psychological science: Comment on Ferguson and Brannick (2012). https://app.cochat.ai/papers/10.1037/a0027128
- The paper
- Rothstein, H. R., & Bushman, B. J. (2012). Publication bias in psychological science: Comment on Ferguson and Brannick (2012). Psychological Methods. https://doi.org/10.1037/a0027128
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.