The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals
Abstract
We examine the APSR and the AJPS for the presence of publication bias due to reliance on the 0.05 significance level. Our analysis employs a broad interpretation of publication bias, which we define as the outcome that occurs when, for whatever reason, publication practices lead to bias in the published parameter estimates. We examine the effect of the 0.05 significance level on the pattern of published findings using a “caliper” test, a novel method for comparing studies with heterogeneous effects, and find that we can reject the hypothesis of no publication bias at the 1 in 32 billion level. Our findings therefore raise the possibility that the results reported in the leading political science journals may be misleading due to publication bias. We also discuss some of the reasons for publication bias and propose reforms to reduce its impact on research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Summing up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Publication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814
Time-Series Minimum-Wage Studies: A Meta-analysis
Card, D., & Krueger, A. B. (2007) · American Economic Review · Cited by 657
The Insignificance of Null Hypothesis Significance Testing
Gill, J. (1999) · Political Research Quarterly · doi:10.1177/106591299905200309 · Cited by 338
Are all Economic Hypotheses False?
De Long, J. B., & Lang, K. (1992) · Journal of Political Economy · doi:10.1086/261860 · Cited by 295
Gerber, A. S., & Malhotra, N. (2008) · Sociological Methods & Research · doi:10.1177/0049124108318973 · Cited by 246
Self-citationDoucouliagos, C. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00252.x · Cited by 213
A Review of Estimates of the Schooling/Earnings Relationship, with Tests for Publication Bias
Ashenfelter, O., Harmon, C., & Oosterbeek, H. (2000) · National Bureau of Economic Research · doi:10.3386/w7457 · Cited by 208
Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias.
Coursol, A., & Wagner, E. E. (1986) · Professional Psychology: Research and Practice · doi:10.1037//0735-7028.17.2.136 · Cited by 165
Testing for Publication Bias in Political Science
Gerber, A. S., GREEN, D. P., & Nickerson, D. W. (2001) · Political Analysis · doi:10.1093/oxfordjournals.pan.a004877 · Cited by 162
Self-citationAn Assessment of Publication Bias Using a Sample of Published Clinical Trials
Berlin, J. A., Begg, C. B., & Louis, T. A. (1989) · Journal of the American Statistical Association · doi:10.1080/01621459.1989.10478782 · Cited by 137
The illusion of learning from observational research
Gerber, A. S., GREEN, D. P., & Kaplan, E. H. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511492174.012 · Cited by 96
Self-citationRacial and Ethnic Differences in Outcome in Zidovudine-Treated Patients With Advanced HIV Disease
Easterbrook, P. (1991) · JAMA · doi:10.1001/jama.1991.03470190061030 · Cited by 93
Publication Bias in the Environmental Tobacco Smoke/Coronary Heart Disease Epidemiologic Literature
LeVois, M. E., & Layard, M. W. (1995) · Regulatory Toxicology and Pharmacology · doi:10.1006/rtph.1995.1023 · Cited by 61
Sigelman, L. (1999) · Political Analysis · doi:10.1093/oxfordjournals.pan.a029813 · Cited by 21
Diagoras of Melos (500 BC): an early analyst of publication bias
Petticrew, M. P. (1998) · The Lancet · doi:10.1016/s0140-6736(05)60377-7 · Cited by 17
Cited by
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542
Self-citationP-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
Burke, M., Hsiang, S. M., & Miguel, E. (2015) · Annual Review of Economics · doi:10.1146/annurev-economics-080614-115430 · Cited by 655
Promoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540
Self-citationTransparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
Metadata
- Authors
- Gerber, A., & Malhotra, N.
- Venue
- Quarterly Journal of Political Science
- Publisher
- Emerald
- Published
- 2008-10-24
- Volume
- 3
- Issue
- 3
- Pages
- 313-326
- ISSN
- 1554-0626 (ISSN-L)
- 1554-0634
- OpenAlex
- W1977112734
- Cited by
- 284
- References
- 18
- FWCI
- 4.50
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.