Methods Matter: P-Hacking and Causal Inference in Economics
Open accessAbstract
The economics 'credibility revolution' has promoted the identification of causal relationships using difference-in-differences (DID), instrumental variables (IV), randomized control trials (RCT) and regression discontinuity design (RDD) methods. The extent to which a reader should trust claims about the statistical significance of results proves very sensitive to method. Applying multiple methods to 13,440 hypothesis tests reported in 25 top economics journals in 2015, we show that selective publication and p-hacking is a substantial problem in research employing DID and (in particular) IV. RCT and RDD are much less problematic. Almost 25% of claims of marginally significant results in IV papers are misleading.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is now substantial evidence that RCTs are the least biased type of study available in social sciences.
“There is now substantial evidence that RCTs are the least biased type of study available in social sciences.”
From Public sector reforms and their impact on the level of corruption: A systematic review (2021) · cites this paper as Brodeur et al. (2018)
The summary (p. 5) identifies both RCT and RDD papers as the most trustworthy among four methods studied, and the Discussion (p. 12) reports less apparent p-hacking in published RCT and RDD studies than in DID and IV studies. This supports a narrower claim about selective reporting in economics papers, not that RCTs alone are the least biased study type across the social sciences. (LLM, conf=0.88)
Experimental — accuracy not yet measuredBrodeur et al. (2018) show that selective publication and p-hacking is a substantial problem in research employing DID and IV.
“Brodeur et al. (2018) show that selective publication and p‐hacking is a substantial problem in research employing DID and (in particular) IV (which are quasi‐natural experiments).”
From Public sector reforms and their impact on the level of corruption: A systematic review (2021) · cites this paper as Brodeur et al. (2018)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 65
Research works printed but not linked: 17
- Angrist, J. D. and Pischke, J.-S. (2010). The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con Out of Econometrics. Journal of Economic Perspectives , 24(2):3-30.
- Brodeur, A., L´ e, M., Sangnier, M., and Zylberberg, Y. (2016). Star Wars: The Empirics Strike Back. American Economic Journal: Applied Economics , 8(1):1-32.
- Casey, K., Glennerster, R., and Miguel, E. (2012). Reshaping Institutions: Evidence on Aid Impacts Using a Pre-Analysis Plan. Quarterly Journal of Economics , 127(4).
- Doucouliagos, C. and Stanley, T. D. (2013). Are All Economic Facts Greatly Exaggerated? Theory Competition and Selectivity. Journal of Economic Surveys , 27(2):316-339.
- Franco, A., Malhotra, N., and Simonovits, G. (2014). Publication Bias in the Social Sciences: Unlocking the File Drawer. Science , 345(6203):1502-1505.
- Gerber, A. and Malhotra, N. (2008a). Do Statistical Reporting Standards Affect what is Published? Publication Bias in Two Leading Political Science Journals. Quarterly Journal of Political Science , 3(3):313-326.
- Gerber, A. S. and Malhotra, N. (2008b). Publication Bias in Empirical Sociological Research: Do Arbitrary Significance Levels Distort Published Results? Sociological Methods & Research , 37(1):3-30.
- Havr´ anek, T. (2015). Measuring Intertemporal Substitution: The Importance of Method Choices and Selective Reporting. Journal of the European Economic Association , 13(6):1180-1204.
- Henry, E. (2009). Strategic Disclosure of Research Results: The Cost of Proving your Honesty. Economic Journal , 119(539):1036-1064.
- Ioannidis, J. P. (2005). Why Most Published Research Findings Are False. PLoS medecine , 2(8):e124.
- Ioannidis, J. P., Stanley, T. D., and Doucouliagos, H. (2017). The Power of Bias in Economics Research. Economic Journal , 127(605):F236-F265.
- Leamer, E. E. (1983). Let's Take the Con Out of Econometrics. American Economic Review , 73(1):pp. 31-43.
- Leamer, E. E. and Leonard, H. (1983). Reporting the Fragility of Regression Estimates. Review of Economics and Statistics , 65(2):pp. 306-317.
- Ridley, J., Kolm, N., Freckelton, R. P., and Gage, M. J. G. (2007). An Unexpected Influence of Widely Used Significance Thresholds on the Distribution of Reported P-Values. Journal of Evolutionary Biology , 20(3):1082-1089.
- Rosenthal, R. (1979). The File Drawer Problem and Tolerance for Null Results. Psychological Bulletin , 86:638.
- Stanley, T. D. (2005). Beyond Publication Bias. Journal of Economic Surveys , 19(3):309-345.
- Stanley, T. D. (2008). Meta-Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection. Oxford Bulletin of Economics and Statistics , 70(1):103-127.
- Authors
- Brodeur, A., Cook, N., & Heyes, A.
- Affiliations
- University of Ottawa (CA)
- University of London (GB)
- Corresponding author
- Brodeur, Abel
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2018
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W2905572727
- Cited by
- 65
- References
- 0
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 2 · 2023: 2 · 2022: 2 · 2021: 24 · 2020: 25 · 2019: 3 · 2018: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- uO Research (University of Ottawa) (submittedVersion, public-domain)
- SSRN Electronic Journal (submittedVersion)
- Econstor (Econstor) (submittedVersion)
- University of Ottawa - Library
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Keywords
- p-hacking, selective publication, credibility revolution, causal inference, difference-in-differences, instrumental variables, randomized controlled trials, regression discontinuity design, statistical significance, research credibility, economics journals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
