Reusing Natural Experiments
Open accessAbstract
After a natural experiment is first used, other researchers often reuse the setting, examining different outcome variables. We use simulations based on real data to illustrate the multiple hypothesis testing problem that arises when researchers reuse natural experiments. We then provide guidance for future inference based on popular empirical settings including difference‐in‐differences, instrumental variables, and regression discontinuity designs. When we apply our guidance to two extensively studied natural experiments, business combination laws and the Regulation SHO pilot, we find that many results that were statistically significant using single hypothesis testing do not survive corrections for multiple hypothesis testing.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The finding that changes in trading sensitivities of stock portfolios by ESG and conventional funds during the crash do not differ statistically is unlikely to change due to the concern that reusing a natural experiment may overestimate t-statistics.
“In our case, the finding that changes in trading sensitivities of stock portfolios by ESG and conventional funds during the crash do not differ statistically is therefore unlikely to change.”
From Mutual Fund Trading, Fund Flows, and ESG Portfolios (2026) · cites this paper as Heath, Ringgenberg, Samadi, and Werner (2023)
Cites
How Much Should We Trust Differences-in-Differences Estimates?
Bertrand, M., Duflo, E., & Mullainathan, S. (2004) · The Quarterly Journal of Economics · doi:10.1162/003355304772839588 · Cited by 10,641 · see citing works
Causal diagrams for empirical research
PEARL, J. (1995) · Biometrika · doi:10.1093/biomet/82.4.669 · Cited by 2,355 · see citing works
Angrist, J. D., & Krueger, A. B. (2001) · Journal of Economic Perspectives · doi:10.1257/jep.15.4.69 · Cited by 2,355 · see citing works
Causal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336 · see citing works
… and the Cross-Section of Expected Returns
Harvey, C. R., Liu, Y., & Zhu, H. (2016) · Review of Financial Studies · doi:10.1093/rfs/hhv059 · Cited by 2,119 · see citing works
Anomalies and False Rejections
Chordia, T., Goyal, A., & Saretto, A. (2017) · doi:10.2139/ssrn.3017677 · Cited by 213 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Angrist, J. D., & Krueger, A. B. (2001) · Journal of Economic Perspectives · doi:10.1257/jep.15.4.69 · Cited by 2,355 · see citing works
Mutual Fund Trading, Fund Flows, and ESG Portfolios
Albuquerque, R., Koskinen, Y., & Santioni, R. (2026) · Journal of Financial and Quantitative Analysis · doi:10.1017/s0022109025101841 · Cited by 3 · see citing works
- Authors
- HEATH, D., RINGGENBERG, M. C., SAMADI, M., & WERNER, I. M.
- Affiliations
- Rutgers, The State University of New Jersey (US)
- Corresponding author
- WERNER, INGRID M.
- Venue
- The Journal of Finance
- Publisher
- Wiley
- Published
- 2023-06-07
- Volume
- 78
- Issue
- 4
- Pages
- 2329-2364
- ISSN
- 0022-1082 (ISSN-L)
- 1540-6261
- OpenAlex
- W4378232072
- Cited by
- 133
- References
- 6
- Citations by year
- 2026: 17 · 2025: 29 · 2024: 31 · 2023: 20 · 2022: 12 · 2021: 7 · 2020: 7 · 2019: 4 · 2018: 2 · 2017: 3 · 2016: 1
- FWCI
- 19.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- public-domain
- onlinelibrary.wiley.com/termsAndConditions#vor (vor)
- Locations
- The Journal of Finance (publishedVersion, public-domain)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Economics, Econometrics and Finance › Finance › Financial Markets and Investment Strategies
- Social Sciences › Economics, Econometrics and Finance › General Economics, Econometrics and Finance › Monetary Policy and Economic Impact
- Keywords
- natural experiment, multiple comparisons, false discovery rate, difference-in-differences, instrumental variables, regression discontinuity design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
