Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends
The DOI matches its Crossref record; 56 of the 58 references it cites were found in a registry or an index.
Abstract
This paper discusses two important limitations of the common practice of testing for preexisting differences in trends (“ pre-trends”) when using difference-in-differences and related methods. First, conventional pre-trends tests may have low power. Second, conditioning the analysis on the result of a pretest can distort estimation and inference, potentially exacerbating the bias of point estimates and under-coverage of confidence intervals. I analyze these issues both in theory and in simulations calibrated to a survey of recent papers in leading economics journals, which suggest that these limitations are important in practice. I conclude with practical recommendations for mitigating these issues. (JEL A14, C23, C51)
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, conducting statistical pre-trend tests and conditioning the analysis on passing these tests may exacerbate bias in the estimated treatment effects because the observed draws of data that pass such tests are a select sample from the true underlying data-generating process [76].”
From Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods (2025) · cites this paper as [76]
In the citing paper
“However, as several papers have noted [20, 24, 76-77], these tests can be misleading because they may not have adequate statistical power to detect a violation, even if there are non-parallel pre-intervention trend differences between treatment and comparison groups [20, 24, 76-77].”
From Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods (2025) · cites this paper as [76]
In the citing paper
“As with tests of slope differences, event study estimates should not be evaluated based on whether confidence intervals overlap zero [24, 76].”
From Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods (2025) · cites this paper as [76]
In the citing paper
“Moreover, our strategy circumvents the inference problems after pre-testing that were pointed out by Roth (2022), under spherical errors.”
From Revisiting Event-Study Designs: Robust and Efficient Estimation (2024) · cites this paper as Roth (2022)
In the citing paper
“This procedure is robust to treatment-effect heterogeneity and, under spherical errors, has attractive power properties and avoids the problem of inference after pre-testing explained by Roth (2022).”
From Revisiting Event-Study Designs: Robust and Efficient Estimation (2024) · cites this paper as Roth (2022)
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Difference-in-Differences with multiple time periods
Callaway, B., & Sant’Anna, P. H. C. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.12.001 · Cited by 9,126
Difference-in-differences with variation in treatment timing
Goodman-Bacon, A. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2021.03.014 · Cited by 8,046
Estimating dynamic treatment effects in event studies with heterogeneous treatment effects
Sun, L., & Abraham, S. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.09.006 · Cited by 5,518
Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486 · cited 1×
“…of this form (under suitable regularity conditions) include dynamic TWFE estimators, the GMM estimator of Freyaldenhoven, Hansen and Shapiro (2019), and methods for difference-in-differences conditional on covariates (Abadie, 2005; Heckman, Ichimura and Todd, 1997; Sant’Anna and Zhao, 2020).”
MethodTwo-Way Fixed Effects Estimators with Heterogeneous Treatment Effects
de Chaisemartin, C., & D’Haultfœuille, X. (2020) · American Economic Review · doi:10.1257/aer.20181169 · Cited by 5,031
Semiparametric Difference-in-Differences Estimators
Abadie, A. (2005) · The Review of Economic Studies · doi:10.1111/0034-6527.00321 · Cited by 2,286
A More Credible Approach to Parallel Trends
Rambachan, A., & Roth, J. (2023) · Review of Economic Studies · doi:10.1093/restud/rdad018 · Cited by 1,559
Self-citationEstimating the Effect of Training Programs on Earnings
Ashenfelter, O. (1978) · The Review of Economics and Statistics · doi:10.2307/1924332 · Cited by 1,515
Doubly robust difference-in-differences estimators
Sant’Anna, P. H. C., & Zhao, J. (2020) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.06.003 · Cited by 1,047 · cited 1×
“…of this form (under suitable regularity conditions) include dynamic TWFE estimators, the GMM estimator of Freyaldenhoven, Hansen and Shapiro (2019), and methods for difference-in-differences conditional on covariates (Abadie, 2005; Heckman, Ichimura and Todd, 1997; Sant’Anna and Zhao, 2020).”
MethodDid Unilateral Divorce Laws Raise Divorce Rates? A Reconciliation and New Results
Wolfers, J. (2006) · American Economic Review · doi:10.1257/aer.96.5.1802 · Cited by 950 · cited 1×
“…of parallel trends is a natural benchmark given that researchers worried about differential trends often include parametric linear controls (e.g., Wolfers (2006); Dobkin et al. (2018); Goodman-Bacon (2018)), which suggests that authors perceive linear vi-10 olations of parallel trends to be…”
Design-based analysis in Difference-In-Differences settings with staggered adoption
Athey, S., & Imbens, G. W. (2022) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.10.012 · Cited by 917
Publication Bias in Meta‐Analysis
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch1 · Cited by 603 · cited 1×
“Requiring an insignificant pre-trend to publish a paper can also be viewed as a form of publication bias, a topic which has been studied extensively (e.g., Rothstein, Sutton and Borenstein (2005), Christensen and Miguel (2016), Snyder and Zhuo (2018), Andrews and Kasy (2019)).”
BackgroundThe Economic Consequences of Hospital Admissions
Dobkin, C., Finkelstein, A., Kluender, R., & Notowidigdo, M. J. (2018) · American Economic Review · doi:10.1257/aer.20161038 · Cited by 555
Defensive Investments and the Demand for Air Quality: Evidence from the NOx Budget Program
Deschênes, O., Greenstone, M., & Shapiro, J. S. (2017) · American Economic Review · doi:10.1257/aer.20131002 · Cited by 489
Learning about an Infrequent Event: Evidence from Flood Insurance Take-Up in the United States
Gallagher, J. (2014) · American Economic Journal: Applied Economics · doi:10.1257/app.6.3.206 · Cited by 476
Revisiting Event Study Designs
Borusyak, K., & Jaravel, X. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2826228 · Cited by 463
Pre-Event Trends in the Panel Event-Study Design
Freyaldenhoven, S., Hansen, C., & Shapiro, J. M. (2019) · American Economic Review · doi:10.1257/aer.20180609 · Cited by 427
School Finance Reform and the Distribution of Student Achievement
Lafortune, J., Rothstein, J., & Schanzenbach, D. W. (2018) · American Economic Journal: Applied Economics · doi:10.1257/app.20160567 · Cited by 415
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388
Kahn-Lang, A., & Lang, K. (2020) · Journal of Business & Economic Statistics · doi:10.1080/07350015.2018.1546591 · Cited by 367
The Fiscal Cost of Hurricanes: Disaster Aid versus Social Insurance
Deryugina, T. (2017) · American Economic Journal: Economic Policy · doi:10.1257/pol.20140296 · Cited by 308
Public Insurance and Mortality: Evidence from Medicaid Implementation
Goodman-Bacon, A. (2018) · Journal of Political Economy · doi:10.1086/695528 · Cited by 270 · cited 1×
“…is a natural benchmark given that researchers worried about differential trends often include parametric linear controls (e.g., Wolfers (2006); Dobkin et al. (2018); Goodman-Bacon (2018)), which suggests that authors perceive linear vi-10 olations of parallel trends to be relevant in many cases.”
BackgroundThe League of Nations Professor Tinbergen’s Method
Keynes, J. M. (1939) · The Economic Journal · doi:10.1093/ej/49.195.558 · Cited by 266
Bailey, M. J., & Goodman-Bacon, A. (2015) · American Economic Review · doi:10.1257/aer.20120070 · Cited by 227
Economic Effects of the Abolition of Serfdom: Evidence from the Russian Empire
Markevich, A., & Zhuravskaya, E. V. (2014) · American Economic Review · doi:10.1257/aer.20160144 · Cited by 177
Valid Two-Step Identification-Robust Confidence Sets for GMM
Andrews, I. (2017) · The Review of Economics and Statistics · doi:10.1162/rest_a_00682 · Cited by 129
PRE‐TEST ESTIMATION AND TESTING IN ECONOMETRICS: RECENT DEVELOPMENTS
Giles, J. A., & Giles, D. E. A. (1993) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.1993.tb00163.x · Cited by 116
Visualization, Identification, and Estimation in the Linear Panel Event-Study Design
Freyaldenhoven, S., Hansen, C., Pérez Pérez, J., & Shapiro, J. (2021) · doi:10.3386/w29170 · Cited by 116
Nothing to See Here? A non-inferiority approach to parallel trends
Bilinski, A., & Hatfield, L. A. (2018) · arXiv · doi:10.48550/arxiv.1805.03273 · Cited by 104
Bosch, M., & Campos-Vázquez, R. M. (2014) · American Economic Journal Economic Policy · doi:10.1257/pol.6.4.71 · Cited by 98
Do College Graduates Serving as Village Officials Help Rural China?
He, G., & Wang, S. (2017) · American Economic Journal: Applied Economics · doi:10.1257/app.20160079 · Cited by 92
Business cycles in the United States of America, 1919-1932
(1939) · Cited by 91
Tewari, I. (2014) · American Economic Journal Applied Economics · doi:10.1257/app.6.4.175 · Cited by 58
The Long-Run Effects of Teacher Collective Bargaining
Lovenheim, M. F., & Willén, A. (2019) · American Economic Journal: Economic Policy · doi:10.1257/pol.20170570 · Cited by 57
Does Managed Care Widen Infant Health Disparities? Evidence from Texas Medicaid
Kuziemko, I., Meckel, K., & Rossin‐Slater, M. (2018) · American Economic Journal Economic Policy · doi:10.1257/pol.20150262 · Cited by 54
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22989 · Cited by 48
Early Retirement Incentives and Student Achievement
Fitzpatrick, M. D., & Lovenheim, M. F. (2014) · American Economic Journal: Economic Policy · doi:10.1257/pol.6.3.120 · Cited by 47
Civil Service Rules and Policy Choices: Evidence from US State Governments
Ujhelyi, G. (2014) · American Economic Journal Economic Policy · doi:10.1257/pol.6.2.338 · Cited by 42
Moments Calculation for the Doubly Truncated Multivariate Normal Density
Manjunath, B. G., & Wilhelm, S. (2021) · Journal of Behavioral Data Science · doi:10.35566/jbds/v1n1/p2 · Cited by 28
Moments Calculation For the Doubly Truncated Multivariate Normal DensityPreprint
Manjunath, B. G., & Wilhelm, S. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.5387 · Cited by 28
(2018) · arXiv (Cornell University) · Cited by 16
Snyder, C. M., & Zhuo, R. (2018) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/8vdrh · Cited by 3
Bosch, M., & Campos-Vázquez, R. M. (2014) · ICPSR Data Holdings · doi:10.3886/e114886v1 · Cited by 1
Data and code for: Pre-test with Caution: Event-study Estimates After Testing for Parallel Trends
Roth, J. (2022) · ICPSR Data Holdings · doi:10.3886/e151982 · Cited by 1
Self-citationKuziemko, I., Meckel, K., & Rossin‐Slater, M. (2018) · ICPSR Data Holdings · doi:10.3886/e114651v1
Research works printed but not linked: 3
- Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. New York: Routledge.
- Lee Jason D, 2016, Annals of Statistics, 44, 907
- Leeb Hannes, 2005, Econometric Theory, 21, 21
Cited by
Difference-in-Differences with multiple time periods
Callaway, B., & Sant’Anna, P. H. C. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.12.001 · Cited by 9,126
Estimating dynamic treatment effects in event studies with heterogeneous treatment effects
Sun, L., & Abraham, S. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.09.006 · Cited by 5,518
Revisiting Event-Study Designs: Robust and Efficient Estimation
Borusyak, K., Jaravel, X., & Spiess, J. (2024) · Review of Economic Studies · doi:10.1093/restud/rdae007 · Cited by 2,280
What's trending in difference-in-differences? A synthesis of the recent econometrics literature
Roth, J., Sant’Anna, P. H. C., Bilinski, A., & Poe, J. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.008 · Cited by 2,027
“A natural statistic to report in many contexts is the ‘‘breakdown’’ value of ¯ M using the sensitivity analysis in Rambachan and Roth (2022b) — i.e. how big would the post-treatment violation of parallel trends have to be relative to the largest pre-treatment violation to invalidate a particular conclusion?”
InfluentialSelf-citationA More Credible Approach to Parallel Trends
Rambachan, A., & Roth, J. (2023) · Review of Economic Studies · doi:10.1093/restud/rdad018 · Cited by 1,559
Self-citation
Metadata
- Authors
- Roth, J.
- Affiliations
- Brown University (US)
- Corresponding author
- Roth, Jonathan
- Venue
- American Economic Review: Insights
- Publisher
- American Economic Association
- Published
- 2022-09-01
- Volume
- 4
- Issue
- 3
- Pages
- 305-322
- ISSN
- 2640-205X (ISSN-L)
- 2640-2068
- OpenAlex
- W4293761886
- Semantic Scholar
- e997ce06ce3226f7377de5dc8d9cdd6989d501d0
- Cited by
- 769
- References
- 58
- Citations by year
- 2026: 151 · 2025: 190 · 2024: 177 · 2023: 109 · 2022: 70
- FWCI
- 100.67
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Economic Review Insights (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- difference-in-differences, parallel trends assumption, statistical power, confidence interval coverage, estimation bias, causal inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.