The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics
Abstract
The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal “revise and resubmit” process mitigates the problem; (iii) things are improving through time. (JEL A14, C12, C52)
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Brodeur et al. (2020) find sub-”
From Can Policies Stall the Fertility Fall? A Systematic Review of the (Quasi‐) Experimental Literature (2021) · cites this paper as Brodeur et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“18See Ioannidis et al. (2017) for an investigation of statistical power and bias in economics.”
BackgroundOptimal Bandwidth Choice for the Regression Discontinuity Estimator
Imbens, G. W., & Kalyanaraman, K. (2011) · The Review of Economic Studies · doi:10.1093/restud/rdr043 · Cited by 2,135
Understanding and misunderstanding randomized controlled trials
Deaton, A., & Cartwright, N. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.005 · Cited by 1,841
(2009) · American Economic Review · doi:10.1257/aer.99.1.i · Cited by 1,764
Angrist, J. D., & Pischke, J.-S. (2010) · Journal of Economic Perspectives · doi:10.1257/jep.24.2.3 · Cited by 1,566
Weak Instruments in Instrumental Variables Regression: Theory and Practice
Andrews, I., Stock, J. H., & Sun, L. (2019) · Annual Review of Economics · doi:10.1146/annurev-economics-080218-025643 · Cited by 849
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
Stanley, T. D. (2008) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2007.00487.x · Cited by 605
Research works printed but not linked: 3
- Blanco-Perez Cristina, 2019, IZA World of Labor, 2019, 467
- Leamer Edward E, 1983, American Economic Review, 73, 31
- McCloskey Donald N, 1985, American Economic Review, 75, 201
Cited by
Sorted by citations
RCTs to Scale: Comprehensive Evidence From Two Nudge Units
DellaVigna, S., & Linos, E. (2022) · Econometrica · doi:10.3982/ecta18709 · Cited by 510
Financial Education Affects Financial Knowledge and Downstream Behaviors
Kaiser, T., Lusardi, A., Menkhoff, L., & Urban, C. (2022) · Journal of Financial Economics · doi:10.1016/j.jfineco.2021.09.022 · Cited by 508
Beland, L., Brodeur, A., & Wright, T. J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0270341 · Cited by 321
Self-citationCan Policies Stall the Fertility Fall? A Systematic Review of the (Quasi‐) Experimental Literature
Bergsvik, J., Fauske, A., & Hart, R. K. (2021) · Population and Development Review · doi:10.1111/padr.12431 · Cited by 259
Armstrong, C., Kepler, J. D., Samuels, D., & Taylor, D. J. (2022) · Journal of Accounting and Economics · doi:10.1016/j.jacceco.2022.101521 · Cited by 243
Metadata
- Authors
- Brodeur, A., Cook, N., & Heyes, A.
- Affiliations
- University of Ottawa (CA)
- Venue
- American Economic Review
- Publisher
- American Economic Association
- Published
- 2020-11-01
- Volume
- 110
- Issue
- 11
- Pages
- 3634-3660
- ISSN
- 0002-8282 (ISSN-L)
- OpenAlex
- W3047016911
- Cited by
- 333
- References
- 41
- Citations by year
- 2026: 44 · 2025: 50 · 2024: 91 · 2023: 49 · 2022: 47 · 2021: 41 · 2020: 7
- FWCI
- 25.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Economic Review (publishedVersion)
- Open Research Exeter (University of Exeter) (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Figshare (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- p-hacking, publication bias, credibility revolution, causal inference, instrumental variables, difference-in-differences, regression discontinuity design, randomized controlled trials, hypothesis testing, economics journals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
