Star Wars: The Empirics Strike Back
Open accessAbstract
Using 50,000 tests published in the AER, JPE, and QJE, we identify a residual in the distribution of tests that cannot be explained solely by journals favoring rejection of the null hypothesis. We observe a two-humped camel shape with missing p-values between 0.25 and 0.10 that can be retrieved just after the 0.05 threshold and represent 10–20 percent of marginally rejected tests. Our interpretation is that researchers inflate the value of just-rejected tests by choosing “significant” specifications. We propose a method to measure this residual and describe how it varies by article and author characteristics. (JEL A11, C13)
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A large analysis of 50,000 reported p-values published between 2005 and 2011 in three widely cited general economics journals shows 'missing' p-values between .05-.20.
“A large analysis of 50,000 reported p-values published between 2005 and 2011 in thr ee widely cited general economics journals shows 'missing' p-values between .05-.20 ( 32 ).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [32]
The Introduction reports 50,078 tests from three major economics journals between 2005 and 2011. It places the missing tests at p-values between 0.10 and 0.25, not 0.05 and 0.20. The paper also describes collected test statistics, some of which were reconstructed, rather than 50,000 directly reported p-values. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · see citing works
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · see citing works
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359 · see citing works
A peculiar prevalence of <i>p</i> values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266 · see citing works
Flagged first, then by citations
Research works printed but not linked: 32
- Ashenfelter, O. and Greenstone, M.: 2004, Estimating the value of a sta- tistical life: The importance of omitted variables and publication bias, American Economic Review 94(2), 454-460.
- Ashenfelter, O., Harmon, C. and Oosterbeek, H.: 1999, A review of esti- mates of the schooling/earnings relationship, with tests for publication bias, Labour Economics 6(4), 453-470.
- Auspurg, K. and Hinz, T.: 2011, What fuels publication bias? theoretical and empirical analyses of risk factors using the caliper test, Journal of Economics and Statistic 231(5-6), 636 -660.
- Bastardi, A., Uhlmann, E. L. and Ross, L.: 2011, Wishful thinking, Psy- chological Science 22(6), 731-732.
- Berlin, J. A., Begg, C. B. and Louis, T. A.: 1989, An assessment of pub- lication bias using a sample of published clinical trials, Journal of the American Statistical Association 84(406), 381-392.
- Card, D. and DellaVigna, S.: 2013, Nine facts about top journals in eco- nomics, Journal of Economic Literature 51(1), 144-61.
- Card, D. and DellaVigna, S.: 2014, Page limits on economics articles: Ev- idence from two journals, Journal of Economic Perspectives 28(3), 149- 68.
- Card, D. and Krueger, A. B.: 1995, Time-series minimum-wage studies: A meta-analysis, The American Economic Review 85(2), 238-243.
- De Long, J. B. and Lang, K.: 1992, Are all economic hypotheses false?, Journal of Political Economy 100, 1257-1257.
- Denton, F. T.: 1985, Data mining as an industry, The Review of Economics and Statistics 67(1), 124-27.
- Dewald, W. G., Thursby, J. G. and Anderson, R. G.: 1986, Replication in empirical economics: The journal of money, credit and banking project, American Economic Review 76(4), 587-603.
- Doucouliagos, C. and Stanley, T. D.: 2011, Are all economic facts greatly exaggerated? theory competition and selectivity, Journal of Economic Surveys 27(2), 316-339.
- Doucouliagos, C., Stanley, T. and Giles, M.: 2011, Are estimates of the value of a statistical life exaggerated?, Journal of Health Economics 31(1).
- Gerber, A. and Malhotra, N.: 2008a, Do statistical reporting standards affect what is published? publication bias in two leading political science journals, Quarterly Journal of Political Science 3(3), 313-326.
- Gerber, A. S. and Malhotra, N.: 2008b, Publication bias in empirical so- ciological research: Do arbitrary significance levels distort published re- sults?, Sociological Methods & Research 37(1), 3-30.
- Gerber, A. S., Malhotra, N., Dowling, C. M. and Doherty, D.: 2010, Pub- lication bias in two political behavior literatures, American Politics Re- search 38(4), 591-613.
- Havránek, T.: 2015, Measuring intertemporal substitution: The impor- tance of method choices and selective reporting, Journal of the European Economic Association Forthcoming.
- Hendry, D. F. and Krolzig, H.-M.: 2004, We ran one regression, Oxford Bulletin of Economics and Statistics 66(5), 799-810.
- Henry, E.: 2009, Strategic disclosure of research results: The cost of proving your honesty, Economic Journal 119(539), 1036-1064.
- Leamer, E. E.: 1983, Let's take the con out of econometrics, The American Economic Review 73(1), pp. 31-43.
- Leamer, E. E.: 1985, Sensitivity analyses would help, The American Eco- nomic Review 75(3), 308-313.
- Leamer, E. and Leonard, H.: 1983, Reporting the fragility of regression estimates, The Review of Economics and Statistics 65(2), pp. 306-317.
- Lovell, M. C.: 1983, Data mining, The Review of Economics and Statistics 65(1), 1-12.
- Mahoney, M. J.: 1977, Publication prejudices: An experimental study of confirmatory bias in the peer review system, Cognitive Therapy and Re- search 1(2), 161-175.
- McCullough, B., McGeary, K. A. and Harrison, T. D.: 2008, Do economics journal archives promote replicable research?, Canadian Journal of Eco- nomics 41(4), 1406-1420.
- Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K., Gerber, A., Glennerster, R., Green, D., Humphreys, M., Imbens, G. et al.: 2014, Pro- moting transparency in social science research, Science 343(6166), 30-31.
- Nosek, B. A., Spies, J. and Motyl, M.: 2012, Scientific utopia: Ii -re- structuring incentives and practices to promote truth over publishability, Perspectives on Psychological Science 7(6), 615-631.
- Ridley, J., Kolm, N., Freckelton, R. P. and Gage, M. J. G.: 2007, An unex- pected influence of widely used significance thresholds on the distribution of reported p-values, Journal of Evolutionary Biology 20(3), 1082-1089.
- Sala-i Martin, X.: 1997, I just ran two million regressions, American Eco- nomic Review 87(2), 178-83.
- Stanley, T. D.: 2005, Beyond publication bias, Journal of Economic Sur- veys 19(3), 309-345.
- Sterling, T. D.: 1959, Publication decision and the possible effects on in- ferences drawn from tests of significance-or vice versa, Journal of The American Statistical Association 54, pp. 30-34.
- Weiss, B. and Wagner, M.: 2011, The identification and prevention of pub- lication bias in the social sciences and economics, Journal of Economics and Statistics 231(5 -6), 661 -684.
Cited by
Sorted by citations
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · see citing works
Financial Education Affects Financial Knowledge and Downstream Behaviors
Kaiser, T., Lusardi, A., Menkhoff, L., & Urban, C. (2022) · Journal of Financial Economics · doi:10.1016/j.jfineco.2021.09.022 · Cited by 508 · see citing works
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388 · see citing works
Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics
Brodeur, A., Cook, N., & Heyes, A. (2020) · American Economic Review · doi:10.1257/aer.20190687 · Cited by 333 · see citing works
Self-citationEstimating the Armington elasticity: The importance of study design and publication bias
Bajzik, J., Havranek, T., Irsova, Z., & Schwarz, J. (2020) · Journal of International Economics · doi:10.1016/j.jinteco.2020.103383 · Cited by 150 · see citing works
Havranek, T., & Sokolova, A. (2020) · Review of Economic Dynamics · doi:10.1016/j.red.2019.05.004 · Cited by 141 · see citing works
- Authors
- Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y.
- Affiliations
- University of Ottawa (CA)
- Banque de France (FR)
- Centre National de la Recherche Scientifique (FR)
- University of Bristol (GB)
- Venue
- American Economic Journal: Applied Economics
- Publisher
- American Economic Association
- Published
- 2016-01-01
- Volume
- 8
- Issue
- 1
- Pages
- 1-32
- ISSN
- 1945-7782 (ISSN-L)
- 1945-7790
- OpenAlex
- W1835208334
- Cited by
- 430
- References
- 10
- Citations by year
- 2026: 14 · 2025: 18 · 2024: 39 · 2023: 38 · 2022: 43 · 2021: 48 · 2020: 90 · 2019: 40 · 2018: 32 · 2017: 41 · 2016: 14 · 2015: 6 · 2014: 3 · 2013: 3 · 2012: 1
- FWCI
- 28.22
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Economic Journal Applied Economics (publishedVersion)
- Explore Bristol Research (publishedVersion, other-oa)
- HAL (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion, other-oa)
- Funders
- European Commission (2007–2013, FP7 2007-2013, FP7/2007)
- Conseil Régional, Île-de-France
- Social Sciences and Humanities Research Council of Canada
- FP7 Ideas: European Research Council (241114)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- significance testing, p-values, null hypothesis rejection, specification search, p-hacking, author characteristics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
