Nothing to See Here? A Non‐Inferiority Approach to Parallel Trends
Open accessPublished version of a preprint
The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Difference‐in‐differences is a popular method for observational health policy evaluation. It relies on a causal assumption that in the absence of intervention, treatment groups' outcomes would have evolved in parallel to those of comparison groups. Researchers frequently look for parallel trends in the pre‐intervention period to bolster confidence in this assumption. The popular “parallel trends test” evaluates a null hypothesis of parallel trends and, failing to find evidence against the null, concludes that the assumption holds. This tightly controls the probability of falsely concluding that trends are not parallel, but may have low power to detect non‐parallel trends. When used as a screening step, it can also introduce bias in treatment effect estimates. We propose a non‐inferiority/equivalence approach that tightly controls the probability of missing large violations of parallel trends, measured on the scale of the treatment effect. Our framework nests several common use cases, including linear trend tests and event studies. We show that our approach may induce no or minimal bias when used as a screening step under commonly assumed error structures and, absent violations, can offer a higher‐power alternative to testing treatment effects in more flexible models. We illustrate our ideas by reconsidering a study of the impact of the Affordable Care Act's dependent coverage provision.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Difference-in-differences with variation in treatment timing
Goodman-Bacon, A. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2021.03.014 · Cited by 8,046
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306
Estimating dynamic treatment effects in event studies with heterogeneous treatment effects
Sun, L., & Abraham, S. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.09.006 · Cited by 5,518
Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects
de Chaisemartin, C., & D’Haultfœuille, X. (2020) · American Economic Review · doi:10.1257/aer.20181169 · Cited by 5,031
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046
What's trending in difference-in-differences? A synthesis of the recent econometrics literature
Roth, J., Sant’Anna, P. H. C., Bilinski, A., & Poe, J. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.008 · Cited by 2,027 · cited 1×
“However, in addition to regression-based estimators for conditional parallel trends [37], other popular techniques use propensity score weighting or doubly robust estimators, which are beyond the scope of this paper [4, 41].”
Self-citationDuflo, E. (2001) · American Economic Review · doi:10.1257/aer.91.4.795 · Cited by 1,940
A More Credible Approach to Parallel Trends
Rambachan, A., & Roth, J. (2023) · Review of Economic Studies · doi:10.1093/restud/rdad018 · Cited by 1,559
Methods for Evaluating Changes in Health Care Policy: The Difference-in-Differences Approach
Dimick, J. B., & Ryan, A. M. (2014) · JAMA · doi:10.1001/jama.2014.16153 · Cited by 1,193
Doubly robust difference-in-differences estimators
Sant’Anna, P. H. C., & Zhao, J. (2020) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.06.003 · Cited by 1,047 · cited 1×
“However, in addition to regression-based estimators for conditional parallel trends [37], other popular techniques use propensity score weighting or doubly robust estimators, which are beyond the scope of this paper [4, 41].”
Research works printed but not linked: 31
- 10.1097/EDE.0000000000001369 · DOI
- 10.1111/biom.13862 · DOI
- E.Tchetgen Tchetgen C.Park andD.Richardson “Universal Difference‐in‐Differences for Causal Inference in Epidemiology ” arXiv:2302.00840(2023). · DOI
- 10.1177/0962280218814570 · DOI
- 10.1001/jama.2018.12345 · DOI
- 10.1001/jama.2019.9021 · DOI · Paper page
- 10.1001/jamainternmed.2019.0272 · DOI · Paper page
- 10.1257/aer.20180609 · DOI · Paper page
- 10.1080/07350015.2018.1546591 · DOI · Paper page
- 10.1016/0197‐2456(82)90024‐1 · DOI · Paper page
- 10.1111/ajps.12387 · DOI · Paper page
- 10.1017/pan.2020.43 · DOI
- 10.1080/07474938.2017.1348683 · DOI
- A.TazhitdinovaandG.Vazquez‐Bare “Difference‐in‐Differences With Unequal Baseline Treatment Status ”2023. · DOI
- A.Strezhnev “Group‐Specific Linear Trends and the Triple‐Differences in Time Design. Preprint(2024). · DOI
- 10.7326/M16‐2607 · DOI · Paper page
- 10.1086/230638 · DOI · Paper page
- 10.1111/j.1541‐0420.2008.01192.x · doi:10.1111/j.1541-0420.2008.01192.x
- 10.1016/j.jeconom.2020.12.001 · DOI · Paper page
- 10.1016/j.jeconom.2020.10.012 · DOI · Paper page
- 10.1186/s12874‐021‐01471‐y · DOI
- P. H. C.Sant'AnnaandQ.Xu “Difference‐in‐Differences with Compositional Changes ”2023 arXiv:2304.13925. · arXiv
- 10.1111/1475‐6773.13666 · DOI · Paper page
- 10.2307/1913827 · DOI · Paper page
- 10.1257/aer.20120070 · DOI · Paper page
- 10.1080/01621459.2017.1281813 · DOI
- Cohen R. A., 2012, Health Insurance Coverage: Early Release of Estimates From the National Health Interview Survey Technical Report
- 10.1016/j.jhealeco.2014.12.004 · DOI · Paper page
- 10.1377/hlthaff.2012.0552 · DOI · Paper page
- 10.1111/j.1475‐6773.2012.01458.x · DOI · Paper page
- 10.1162/003355304772839588 · DOI · Paper page
Cited by
Difference-in-differences with variation in treatment timing
Goodman-Bacon, A. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2021.03.014 · Cited by 8,046
Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey
de Chaisemartin, C., & D’Haultfœuille, X. (2023) · The Econometrics Journal · doi:10.1093/ectj/utac017 · Cited by 522
Liu, L., Wang, Y., & Xu, Y. (2024) · American Journal of Political Science · doi:10.1111/ajps.12723 · Cited by 390
Liu, L., Wang, Y., & Xu, Y. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3555463 · Cited by 142
Two-Way Fixed Effects and Differences-in-Differences with Heterogeneous Treatment Effects: A Survey
de Chaisemartin, C., & D'Haultfoeuille, X. (2022) · National Bureau of Economic Research · doi:10.3386/w29734 · Cited by 70
Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow
del Rio-Chanona, M., Laurentsyeva, N., & Wachs, J. (2023) · arXiv · doi:10.48550/arxiv.2307.07367 · Cited by 62
Testing for Equivalence of Pre-Trends in Difference-in-Differences Estimation
Dette, H., & Schumann, M. (2024) · Journal of Business & Economic Statistics · doi:10.1080/07350015.2024.2308121 · Cited by 30
“Asweuseequivalence tests, our article is closely related to Bilinski and Hatfield (2020), who provide a discussion on the benefits of using equivalence (or “non-inferiority”) tests when testing for violations of modeling assumptions.”
Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods
Feng, S., Ganguli, I., Lee, Y., Poe, J., Ryan, A., & Bilinski, A. (2025) · Statistics in Medicine · doi:10.1002/sim.70247 · Cited by 16
Self-citation
Metadata
- Authors
- Bilinski, A., & Hatfield, L. A.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2026-02
- Volume
- 45
- Issue
- 3-5
- Pages
- e70296-e70296
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2989928112
- Cited by
- 102
- References
- 22
- FWCI
- 26.81
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- onlinelibrary.wiley.com/termsAndConditions#vor (vor, 4 d)
- doi.wiley.com/10.1002/tdm_license_1.1 (tdm)
- Funders
- Agency for Healthcare Research and Quality (R01HS028985, LH)
- National Institute of General Medical Sciences (1R35GM155224, AB)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.