The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics
Abstract
The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal “revise and resubmit” process mitigates the problem; (iii) things are improving through time. (JEL A14, C12, C52)
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Brodeur et al. (2020) find sub-”
From Can Policies Stall the Fertility Fall? A Systematic Review of the (Quasi‐) Experimental Literature (2021) · cites this paper as Brodeur et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“18See Ioannidis et al. (2017) for an investigation of statistical power and bias in economics.”
BackgroundOptimal Bandwidth Choice for the Regression Discontinuity Estimator
Imbens, G. W., & Kalyanaraman, K. (2011) · The Review of Economic Studies · doi:10.1093/restud/rdr043 · Cited by 2,135
Understanding and misunderstanding randomized controlled trials
Deaton, A., & Cartwright, N. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.005 · Cited by 1,841
(2009) · American Economic Review · doi:10.1257/aer.99.1.i · Cited by 1,764
Angrist, J. D., & Pischke, J.-S. (2010) · Journal of Economic Perspectives · doi:10.1257/jep.24.2.3 · Cited by 1,566
Weak Instruments in Instrumental Variables Regression: Theory and Practice
Andrews, I., Stock, J. H., & Sun, L. (2019) · Annual Review of Economics · doi:10.1146/annurev-economics-080218-025643 · Cited by 849
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
Stanley, T. D. (2008) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2007.00487.x · Cited by 605
Promoting Transparency in Social Science Research
Miguel, E. A., Camerer, C. F., Casey, K., Cohen, J., Esterling, K., Gerber, A. S., Glennerster, R., GREEN, D. P., Humphreys, M., Imbens, G. W., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 559
Stanley, T. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00250.x · Cited by 489
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 484
Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan*
Casey, K., Glennerster, R., & Miguel, E. A. (2012) · The Quarterly Journal of Economics · doi:10.1093/qje/qje027 · Cited by 466
Star Wars: The Empirics Strike Back
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016) · American Economic Journal: Applied Economics · doi:10.1257/app.20150044 · Cited by 430
Self-citationA review of estimates of the schooling/earnings relationship, with tests for publication bias
Ashenfelter, O., Harmon, C. P., & Oosterbeek, H. (1999) · Labour Economics · doi:10.1016/s0927-5371(99)00041-x · Cited by 413
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388
MEASURING INTERTEMPORAL SUBSTITUTION: THE IMPORTANCE OF METHOD CHOICES AND SELECTIVE REPORTING
Havránek, T. (2015) · Journal of the European Economic Association · doi:10.1111/jeea.12133 · Cited by 369
Nine Facts about Top Journals in Economics
Card, D., & DellaVigna, S. (2013) · Journal of Economic Literature · doi:10.1257/jel.51.1.144 · Cited by 333
Reporting the Fragility of Regression Estimates
Leamer, E. E., & Leonard, H. B. “. (1983) · The Review of Economics and Statistics · doi:10.2307/1924497 · Cited by 328
ARE ALL ECONOMIC FACTS GREATLY EXAGGERATED? THEORY COMPETITION AND SELECTIVITY
Doucouliagos, H., & Stanley, T. D. (2011) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.2011.00706.x · Cited by 302
Are all Economic Hypotheses False?
Long, J. B. D., & Lang, K. (1992) · Journal of Political Economy · doi:10.1086/261860 · Cited by 291
Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)
Imbens, G. W. (2010) · Journal of Economic Literature · doi:10.1257/jel.48.2.399 · Cited by 278
Gerber, A. S., & Malhotra, N. (2008) · Quarterly Journal of Political Science · doi:10.1561/100.00008024 · Cited by 260
Consistency without Inference: Instrumental Variables in Practical Application
Young, A. (2022) · European Economic Review · doi:10.1016/j.euroecorev.2022.104112 · Cited by 215
The Loss Function Has Been Mislaid: The Rhetoric of Significance Tests
McCLOSKEY, D. N., Ames, E. S., Reiter, S., Feige, E. L., Leamer, E. E., Mayer, T., Lovell, M. C., & Denton, F. T. (1985) · American Economic Review · Cited by 201
Publication Bias in Empirical Sociological Research
Gerber, A. S., & Malhotra, N. (2008) · Sociological Methods & Research · doi:10.1177/0049124108318973 · Cited by 175
Statistical Nonsignificance in Empirical Economics
Abadie, A. (2020) · American Economic Review: Insights · doi:10.1257/aeri.20190252 · Cited by 148
Havranek, T., & Sokolova, A. (2020) · Review of Economic Dynamics · doi:10.1016/j.red.2019.05.004 · Cited by 141
What Do Editors Maximize? Evidence from Four Economics Journals
Card, D., & DellaVigna, S. (2020) · The Review of Economics and Statistics · doi:10.1162/rest_a_00839 · Cited by 123
Publication Bias and Editorial Statement on Negative Findings
Pérez, C. B., & Brodeur, A. (2019) · The Economic Journal · doi:10.1093/ej/ueaa011 · Cited by 89
Self-citationStrategic Disclosure of Research Results: The Cost of Proving Your Honesty
Henry, E. (2009) · The Economic Journal · doi:10.1111/j.1468-0297.2009.02265.x · Cited by 89
Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction MethodPreprint
Furukawa, C. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3362053 · Cited by 79
Ellison, G. (2010) · Economic Inquiry · doi:10.1111/j.1465-7295.2010.00261.x · Cited by 75
The Empirical Economist's Toolkit: From Models to Methods
Panhans, M. T., & Singleton, J. D. (2015) · History of Political Economy · doi:10.1215/00182702-4166299 · Cited by 67
Should the Randomistas (Continue to) Rule?
Ravallion, M. D. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0003 · Cited by 64
Specification Searching and Significance Inflation Across Time, Methods and Disciplines
Vivalt, E. (2019) · Oxford Bulletin of Economics and Statistics · doi:10.1111/obes.12289 · Cited by 58
Reporting errors and biases in published empirical findings: Evidence from innovation research
Bruns, S. B., Asanov, I., Bode, R., Dunger, M., Funk, C., Hassan, S. M., Hauschildt, J., Heinisch, D. P., Kempa, K., König, J., et al. (2019) · Research Policy · doi:10.1016/j.respol.2019.05.005 · Cited by 50
Havránek, T., Iršová, Z., & Vlach, T. (2018) · Land Economics · doi:10.3368/le.94.2.259 · Cited by 45
Biddle, J. E., & Hamermesh, D. S. (2017) · History of Political Economy · doi:10.1215/00182702-4166251 · Cited by 27
Data and Code for: Methods Matter: P-Hacking and Publication Bias in Causal Analysis in Economics
Brodeur, A., Cook, N., & Heyes, A. (2015) · ICPSR Data Holdings · doi:10.3886/e120246 · Cited by 2
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- Blanco-Perez Cristina, 2019, IZA World of Labor, 2019, 467
- Leamer Edward E, 1983, American Economic Review, 73, 31
- McCloskey Donald N, 1985, American Economic Review, 75, 201
Cited by
Sorted by citations
RCTs to Scale: Comprehensive Evidence From Two Nudge Units
DellaVigna, S., & Linos, E. (2022) · Econometrica · doi:10.3982/ecta18709 · Cited by 510
Financial Education Affects Financial Knowledge and Downstream Behaviors
Kaiser, T., Lusardi, A., Menkhoff, L., & Urban, C. (2022) · Journal of Financial Economics · doi:10.1016/j.jfineco.2021.09.022 · Cited by 508
Beland, L., Brodeur, A., & Wright, T. J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0270341 · Cited by 321
Self-citationCan Policies Stall the Fertility Fall? A Systematic Review of the (Quasi‐) Experimental Literature
Bergsvik, J., Fauske, A., & Hart, R. K. (2021) · Population and Development Review · doi:10.1111/padr.12431 · Cited by 259
Armstrong, C., Kepler, J. D., Samuels, D., & Taylor, D. J. (2022) · Journal of Accounting and Economics · doi:10.1016/j.jacceco.2022.101521 · Cited by 243
Metadata
- Authors
- Brodeur, A., Cook, N., & Heyes, A.
- Affiliations
- University of Ottawa (CA)
- Venue
- American Economic Review
- Publisher
- American Economic Association
- Published
- 2020-11-01
- Volume
- 110
- Issue
- 11
- Pages
- 3634-3660
- ISSN
- 0002-8282 (ISSN-L)
- OpenAlex
- W3047016911
- Cited by
- 333
- References
- 41
- Citations by year
- 2026: 44 · 2025: 50 · 2024: 91 · 2023: 49 · 2022: 47 · 2021: 41 · 2020: 7
- FWCI
- 25.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Economic Review (publishedVersion)
- Open Research Exeter (University of Exeter) (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Figshare (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- p-hacking, publication bias, credibility revolution, causal inference, instrumental variables, difference-in-differences, regression discontinuity design, randomized controlled trials, hypothesis testing, economics journals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
