Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 32 of the 34 references it cites were found in a registry or an index; 9 of the 27 sources its claims cite were checked for support.

Advances in Difference-in-differences Methods for Policy Evaluation Research

Open access
Wang, Guangyi, Hamad, Rita, White, Justin S. · 2024 · Epidemiology

Abstract

Difference-in-differences (DiD) is a powerful, quasi-experimental research design widely used in longitudinal policy evaluations with health outcomes. However, DiD designs face several challenges to ensuring reliable causal inference, such as when policy settings are more complex. Recent economics literature has revealed that DiD estimators may exhibit bias when heterogeneous treatment effects, a common consequence of staggered policy implementation, are present. To deepen our understanding of these advancements in epidemiology, in this methodologic primer, we start by presenting an overview of DiD methods. We then summarize fundamental problems associated with DiD designs with heterogeneous treatment effects and provide guidance on recently proposed heterogeneity-robust DiD estimators, which are increasingly being implemented by epidemiologists. We also extend the discussion to violations of the parallel trends assumption, which has received less attention. Last, we present results from a simulation study that compares the performance of several DiD estimators under different scenarios to enhance understanding and application of these methods.

Overview

CoChat Summary

## TL;DR

This paper is a methodologic primer on difference-in-differences (DiD) for health policy evaluation, focusing on why two-way fixed effects can be biased under heterogeneous treatment effects and on newer heterogeneity-robust alternatives. It also discusses violations of the parallel trends assumption and reports a Monte Carlo simulation comparing estimator performance across several scenarios.

## Key findings

  • Two-way fixed effects DiD can produce biased estimates when treatment effects are heterogeneous across groups or over time; the paper explains this via “negative weights” and “forbidden comparisons.”
  • In the simulation study, with constant and homogeneous effects, two-way fixed effects had the lowest bias (−0.01%) and RMSE (0.12).
  • With dynamic treatment effects, two-way fixed effects had much larger bias (−66.40% to −78.62%), while Callaway and Sant’Anna had the lowest bias (0.64%–4.14%); Borusyak et al., Sun and Abraham, and Wooldridge were around −6% bias.
  • Under parallel trends violations, bias and RMSE increased across all estimators; among heterogeneity-robust methods, Callaway and Sant’Anna and Sun and Abraham outperformed Borusyak et al. and Wooldridge.

## Methods

  • Methodologic primer/review of DiD designs, including the basic 2 × 2 setup, generalized/two-way fixed effects DiD for staggered adoption, heterogeneity-robust estimators, and approaches to parallel trends violations.
  • Monte Carlo simulation study comparing two-way fixed effects with four heterogeneity-robust estimators across 500 runs per estimator in each scenario.
  • Simulation scenarios varied along three dimensions: constant vs dynamic treatment effects, homogeneous vs heterogeneous treatment effects, and presence vs absence of parallel trends violations; performance was assessed by percent bias and RMSE.

## Limitations

  • The paper notes that traditional pretrend testing has low power and is not a reliable indicator of parallel trends violations.
  • The paper states that some covariate-adjustment solutions are still under development; for example, code for conditioning on the untreated potential value of covariates is “still under development.”
  • Evidence evident from the text: the simulation results are based on specified scenarios and a data-generating process described in the supplement, so their applicability depends on those assumptions and may not generalize to all empirical settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1097/ede.0000000000001755

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

FIGURE 1. Monte Carlo simulation results for scenarios 1 and 2 (no PTA violations). Scenarios 1a–1c have constant effects, and scenarios 2a–2c have dynamic (linear trend) effects. Scenarios 1a and 2a have homogeneous effects across groups; scenarios 1b and 2b have heterogeneous (at random) effects across groups; and scenarios 1c and 2c have heterogeneous (large first) effects across groups. Each scenario is listed in Table 4. BJS Indicates Borusyak-Jaravel-Spiess; CS, Callaway-Sant’Anna; JW, Wooldridge; PTA, parallel trends assumption; SA, Sun-Abraham; TWFE, Two-way fixed effects.
FIGURE 1. Monte Carlo simulation results for scenarios 1 and 2 (no PTA violations). Scenarios 1a–1c have constant effects, and scenarios 2a–2c have dynamic (linear trend) effects. Scenarios 1a and 2a have homogeneous effects across groups; scenarios 1b and 2b have heterogeneous (at random) effects across groups; and scenarios 1c and 2c have heterogeneous (large first) effects across groups. Each scenario is listed in Table 4. BJS Indicates Borusyak-Jaravel-Spiess; CS, Callaway-Sant’Anna; JW, Wooldridge; PTA, parallel trends assumption; SA, Sun-Abraham; TWFE, Two-way fixed effects.CC BY-NC-ND · PubMed Central
FIGURE 2. Monte Carlo simulation result for scenario 3 (with PTA violations). Scenarios 3a–3c have constant effects, and Scenarios 3d,e have dynamic (linear trend) effects. Scenarios 3a and 3d have homogeneous effects across groups; Scenarios 3b and 3e have heterogeneous (at random) effects across groups; and scenarios 3c and 3f have heterogeneous (large first) effects across groups. Each scenario is listed in Table 4. BJS Indicates Borusyak-Jaravel-Spiess; CS, Callaway-Sant’Anna; JW, Wooldridge; PTA, parallel trends assumption; SA, Sun-Abraham; TWFE, Two-way fixed effects.
FIGURE 2. Monte Carlo simulation result for scenario 3 (with PTA violations). Scenarios 3a–3c have constant effects, and Scenarios 3d,e have dynamic (linear trend) effects. Scenarios 3a and 3d have homogeneous effects across groups; Scenarios 3b and 3e have heterogeneous (at random) effects across groups; and scenarios 3c and 3f have heterogeneous (large first) effects across groups. Each scenario is listed in Table 4. BJS Indicates Borusyak-Jaravel-Spiess; CS, Callaway-Sant’Anna; JW, Wooldridge; PTA, parallel trends assumption; SA, Sun-Abraham; TWFE, Two-way fixed effects.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

4 Valid · 4 Not valid · 9 Not checked · 31 Own results and interpretations

30 more claims · 1 with their cited sources checked

Create a free account to see all 48 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 10
  • Probably not a substantial claim. 3

Create a free account to see these claims.

Citations

Cites

Show all 34 references

Cited by

See all 117

Metadata

Authors
Wang, G., Hamad, R., & White, J. S.
Affiliations
  • Harvard University (US)
  • Boston University (US)
Corresponding author
White, Justin S.
Venue
Epidemiology
Published
2024-07-05
Volume
35
Issue
5
Pages
628-637
ISSN
  • 1044-3983 (ISSN-L)
  • 1531-5487
OpenAlex
W4400349012
Cited by
117
References
34
Citations by year
2026: 57 · 2025: 56 · 2024: 4
FWCI
40.15
Type
journal article
Language
en
Open access
Locations
  • Epidemiology (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
Funders
  • National Institutes of Health
  • National Heart, Lung, and Blood Institute (R01HL151638)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Social Sciences › Public Administration › Evaluation and Performance Assessment
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
difference-in-differences, policy evaluation, staggered treatment effects, treatment effect heterogeneity, parallel trends assumption, causal inference, health outcomes, simulation study
MeSH terms
Computer Simulation, Health Policy, Humans, Research Design, Models, Statistical, Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Advances in Difference-in-differences Methods for Policy Evaluation Research. https://app.cochat.ai/papers/10.1097/ede.0000000000001755
The paper
Wang, G., Hamad, R., & White, J. S. (2024). Advances in Difference-in-differences Methods for Policy Evaluation Research. Epidemiology. https://doi.org/10.1097/ede.0000000000001755

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.