Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods

May combine two claims.

Conformal inference assumes that the proportion of treated units in each cluster varies little over time and imposes strong parallel trends over all pre- and post-treatment periods.

“However, it assumes that the proportion of treated units in each cluster to vary little over time and imposes a strong parallel trends assumption over all pre- and post-treatment periods [42].”

• cites [42] sourceDisputed

  The Introduction (p. 2) describes the method in terms of stationary, weakly dependent errors whose distribution is invariant under intervention. The Related Literature discussion (p. 5) says it does not model treatment assignment and requires a suitable model for the treated unit’s potential outcome. These stated requirements run counter to the attributed requirements about treated-unit proportions within clusters and strong parallel trends. (LLM, conf=0.84)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

• AssumptionThe authors state this as an assumption: Conformal inference assumes that the proportion of treated units in each cluster varies little over time and imposes strong parallel trends over all pre- and post-treatment periods.

May combine two claims.

The wider use of DiD has coincided with major methodological advances over the past decade that improved the method while making it harder to implement and interpret.

“Broader DiD adoption has been accompanied by significant methodological advancements over the past decade, which have refined the approach but rendered its implementation and interpretation more challenging [17-21].”

• cites [17] sourcePartly supported

  The Introduction describes DiD as popular and says a rapid wave of methodological papers has made it challenging for practitioners to keep up and even for experts to understand how the developments fit together. It does not say the advances made DiD results harder to interpret or the methods harder to implement; the Introduction says new methods can improve interpretability, and Section 3.5 says software packages make implementation straightforward. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [18] sourceNot checked — no full text of the source
• cites [19] sourceNot checked — no full text of the source
• cites [20] sourcePartly supported

  Section 4, “Testing for Preexisting Trends” (page unavailable), says DiD has grown in popularity and that researchers are now effectively required to test for pre-trends when data permit. The paper supports broader adoption and more demanding implementation, but it does not attribute those demands to major methodological advances over the past decade or say that such advances refined DiD while making interpretation harder. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Many US health care policies were introduced at state or local levels, which left comparable states or localities untreated and enabled DiD designs.

“Broadly, these DiDs leveraged the fact that many US health care policies—such as phased insurance expansion [58], elimination of asset tests [65], and antibullying policies [64]—were introduced at the state- or local-level, leaving comparable states or localities untreated.”

• cites [58] sourceCould not verify

  The citation is attached to “antibullying policies,” but no passage in our copy addresses such policies. The Methods (Statistical Analysis) instead describe a difference-in-differences study of Massachusetts Medicaid ACO implementation, comparing Medicaid-insured with privately insured children—not untreated states or localities. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [65] sourceNot checked — no full text of the source
• cites [64] sourceSupported

Classified as probably not a substantial claim.

Difference-in-differences is a widely used observational causal inference method in health policy for evaluating the real-world effects of policies and programs.

“Difference-in-differences (DiD) is a popular observational causal inference method in health policy, employed to evaluate the real-world impact of policies and programs.”

• cites [1] sourcePartly supported

  The Conclusions describe difference-in-differences as a method for estimating the causal impact of population-level changes and discuss evaluating policies to improve population health. But the Introduction says it is used relatively infrequently by epidemiologists, and the Conclusions locate its widespread use in economics; the paper does not support calling it popular in health policy. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

When formal theory is well developed, researchers can translate parallel trends into domain-specific conditions and use context to choose the outcome's functional form or scale, to which parallel trends is usually sensitive.

“Where formal theory is well-developed, researchers may be able to translate the parallel trends assumption into domain-specific conditions and use context to help select functional form or scale of the outcome, to which the parallel trends assumption is usually sensitive [17, 32, 66].”

• cites [17] sourcePartly supported

  Section 4.1 (page unavailable) says parallel trends is generally sensitive to whether outcomes are measured in levels or logs, and the paper advises researchers to justify their choice of functional form. Section 4.6 recommends using context-specific knowledge to assess parallel trends. The passages do not say that well-developed formal theory lets researchers translate parallel trends into domain-specific conditions or use context to select the outcome scale. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [32] sourceSupported
• cites [66] sourcePartly supported

  The Methods section (Review tool for data collection; page unavailable) says the functional form of counterfactual infectious-disease trends should be justified and appropriate to the outcome. The Results section (Strength of methods assessment; page unavailable) identifies inappropriate outcome scales and missing justifications as common problems. These passages support using domain context to choose a form or scale, but do not address translating parallel trends into domain-specific conditions when formal theory is developed or say that parallel trends is usually sensitive to outcome scale. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

• AssumptionThe authors state this as an assumption: Treatment and comparison groups would have had parallel average outcome trajectories absent the intervention.

May combine two claims.

Recent work has described epidemiological conditions needed for parallel trends with infectious disease outcomes under different transformations and specifications and has proposed estimators for that setting.

“Recent work has described epidemiological conditions required for parallel trends to hold with infectious disease outcomes under different outcome transformations and specifications, and proposed estimators for this purpose [27, 69].”

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [69] sourcePartly supported

  In ‘Using difference-in-differences to evaluate policy effects on Covid-19 cases’ (page unavailable), Theorem 1 examines parallel trends under a stochastic SIRD model and finds that the assumption generally fails. The Introduction proposes an alternative estimator for policy effects. The state-level Results compare outcome transformations and specifications, but do not set out conditions under which parallel trends holds for each; they say none of the four TWFE specifications is compatible with the SIRD model. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

• AssumptionThe authors state this as an assumption: Treatment and comparison groups would have had parallel average outcome trajectories absent the intervention.

Classified as probably not a substantial claim.

Conditional parallel trends implies that parallel trends holds within covariate-defined strata, such as within smokers and within non-smokers.

“This implies parallel trends within populations in covariate-defined strata (e.g., parallel trends within smokers and non-smokers): [17, 34]”

• cites [17] sourceSupported
• cites [34] sourceSupported

• AssumptionThe authors state this as an assumption: Parallel trends holds within covariate-defined strata.

May combine two claims.

In DiD, covariates that create only time-invariant level differences between treatment and comparison outcomes are not confounders, and omitting them does not bias ATT estimates.

“By contrast, in DiD, covariates that only produce time-invariant level differences between treatment and comparison outcomes are not confounders [89], and excluding them does not bias ATT estimates (Table 2, row 1).”

• cites [89] sourceSupported

Classified as probably not a substantial claim. May combine two claims.

Non-inferiority tests set a null hypothesis that the parallel trends violation exceeds a threshold and allow DiD only when there is enough evidence to reject that null.

“One alternative is to consider non-inferiority tests [24, 78], which specify a null hypothesis that the violation of parallel trends exceeds some threshold (instead of no violation) and proceed with DiD only if there is sufficient evidence to reject the null [17, 24].”

• cites [24]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [78] sourceNot checked — no full text of the source
• cites [17] sourcePartly supported

  Section 4.4.1 describes non-inferiority tests as setting a null of a large pre-treatment trend and rejecting it only with strong evidence that the trend is small. It does not prescribe proceeding with DiD only after rejection; the same section says these approaches provide no clear guidance when the test fails to reject. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

One reviewed study found that the safer supply policy had an increasing treatment effect on opioid prescriptions over time.

“For instance, one study found an increasing treatment effect of the safer supply policy in opioid prescription over time [84].”

• cites [84] sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Cross-jurisdiction shopping can create spillover violations in soft drink tax evaluations, making neighboring jurisdictions unsuitable as comparison units.

“For example, if individuals can easily avoid a soft drink tax by shopping in a neighboring jurisdiction, researchers might want to remove these jurisdictions from the set of comparison groups [72].”

• cites [72] sourceSupported

• AssumptionThe authors state this as an assumption: One unit's potential outcomes do not depend on other units' treatment status, implying no spillovers.

May combine two claims.

Many newer DiD specifications are saturated, estimating an ATT for each post-intervention time period and sometimes for specific units or groups.

“Many new specifications are “saturated,” estimating an ATT specific to each post-intervention time period and sometimes to specific units or groups of units.”

• cites [34] sourceSupported
• cites [35] sourceNot checked — no full text of the source
• cites [45] sourceSupported
• cites [46] sourceSupported
• cites [47]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [48] sourcePartly supported

  Section II.B estimates effects by event time, and Section III.C estimates effects separately for each minimum-wage event. The robustness checks in Section III call one model “highly saturated.” These passages support the paper as an example of time- and event-specific estimation, but they do not establish the broader claim that many new DiD specifications are saturated or describe those estimates as ATTs. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

When a policy is announced or enacted long before implementation, individuals may anticipate its adoption, so the pre-intervention period may need to end before that anticipation.

“Similarly, if a policy is announced or enacted in legislation for an extended period prior to implementation, researchers may end the pre-intervention period prior to the time at which individuals likely anticipated its adoption [73-75].”

• cites [73]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [74] sourceSupported
• cites [75] sourceNot checked — no full text of the source

• AssumptionThe authors state this as an assumption: The treatment has no effect on treated units before the intervention begins.

Classified as probably not a substantial claim.

There is growing consensus that researchers should default to saturated group-time models, including estimators by Callaway and Sant'Anna, Sun and Abraham, and Borusyak, Jaravel, and Spiess.

“There is growing consensus that researchers should default to these saturated models, including estimators proposed by Callaway and Sant’Anna, Sun and Abraham, and Borusyak, Jaravel, and Spiess (Table 3, row 3) [34, 35, 46, 106].”

• cites [34] sourcePartly supported

  The Introduction and Section 4 describe the paper’s group-time treatment-effect estimators, and the Introduction discusses related estimators by Sun and Abraham. Section 3 explains problems with conventional two-way fixed-effects estimates. The paper does not establish a growing consensus that researchers should default to saturated models or recommend the full set of named estimators as a default. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [35] sourceNot checked — no full text of the source
• cites [46] sourcePartly supported

  The Introduction describes the Callaway–Sant’Anna and Sun–Abraham estimators as robust to arbitrary treatment-effect heterogeneity and proposes the authors’ own robust imputation estimator. It does not claim a growing consensus that researchers should default to saturated group-time models. The reference to a “default” in §4.3 concerns grouping for standard-error estimation in the authors’ Stata command, not a default choice of model. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [106] sourcePartly supported

  Section 2 (p. 4) identifies the Callaway–Sant’Anna, Sun–Abraham, and Borusyak et al. estimators in its discussion of flexible difference-in-differences methods. Section 5.4 (p. 14) compares some of these methods with TWFE. The passages do not say there is a growing consensus that researchers should default to saturated group-time models. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Callaway and Sant'Anna described approaches to aggregate individual group-time average treatment effects into an overall ATT estimate.

“They then described several approaches to aggregate individual group-time average treatment effects into an overall ATT estimate [34, 106].”

• cites [34] sourceSupported
• cites [106] sourceSupported

May combine two claims.

In a large empirical reanalysis, staggered-timing estimators sometimes produced substantially smaller treatment effect estimates and often larger standard errors, requiring more power to reject the null than TWFE.

“In a large empirical reanalysis [21], estimators that account for staggered treatment timing were found to sometimes produce substantially smaller treatment effect estimates. These estimators also often produced larger standard errors and would require more power to reject the null hypothesis than TWFE [21].”

• cites [21] sourceSupported

May combine two claims.

Stacked regression allows both time and cohort heterogeneity and requires parallel trends only over the horizon the researcher specifies.

“This approach allows both time and cohort heterogeneity, requiring parallel trends over only the horizon of time that the researcher specifies.”

• cites [48] sourceSupported

May combine two claims.

These alternatives provide well-defined causal parameters with transparent weights over 2x2 comparisons and avoid using treated units as comparison groups.

“Overall, these alternatives provide well-defined causal parameters with transparent weights over the 2 × 2 comparisons and avoid using treated units as comparison groups [17].”

• cites [17] sourceSupported

May combine two claims.

Dynamic TWFE yields a sensible ATT estimand if heterogeneity exists only in time since treatment, but it requires the sequence of treatment effects to be the same across all cohorts regardless of calendar adoption time.

“The dynamic TWFE specification yields a sensible estimand for the ATT if heterogeneity only exists in the number of time periods since treatment, but it still requires the sequence of treatment effects to be the same across all cohorts regardless of calendar adoption time (Table 3, row 2) [35, 46].”

• Also stated in: § Recommendation 3: When Treatment Adoption Is Staggered, Use Saturated Estimators With Clean Comparison Groups, Particularly Those That Estimate Group-Time Effects · plain

  “For example, dynamic TWFE allows treatment effects to be different in year 2 compared to year 3 following treatment adoption, but the year 2 effect is assumed to be the same across all adoption cohorts [17].”

• cites [35] sourceNot checked — no full text of the source
• cites [46] sourceSupported
• cites [17] sourceSupported

May combine two claims.

Rambachan and Roth proposed a method to bound how large parallel trends violations must be to shift effect estimates and to construct valid confidence intervals under certain violations.

“Rambachan and Roth [91] proposed an approach to bound how large violations of parallel trends must be to meaningfully shift effect estimates and developed methods to construct valid confidence intervals for the treatment effect under certain violations.”

• cites [91] sourceSupported

May combine two claims.

Other confounding-adjustment innovations include Bayesian trajectory modeling, machine learning for IPW and doubly robust estimators, and IPW using latent variables or marginal structural models.

“Other innovations in confounding adjustment methods include Bayesian approaches for trajectory modeling [96, 97], machine learning to estimate weights for IPW and doubly robust estimators [98-100], and IPW approaches driven by latent variables or marginal structure models to facilitate a better understanding of selection into treatment [100-102].”

• cites [96] sourceNot checked — no full text of the source
• cites [97] sourceCould not verify

  The cited paper falls within the citation groups for machine-learning-based IPW and doubly robust estimators, and for IPW using latent variables or marginal structural models. Its Materials and Methods instead describes Bayesian estimation of a negative binomial model; searches for inverse-probability weighting, machine learning, doubly robust estimation, latent variables, and marginal structural models found no passage addressing those methods. The Bayesian trajectory-modeling clause is attributed to different citations. (LLM, conf=0.74)

  Experimental — accuracy not yet measured
• cites [98] sourceNot checked — no full text of the source
• cites [99] sourcePartly supported

  The sections on predicting treatment weights and doubly robust methods (pages not available in our copy) describe machine-learning-generated IPW weights, their use in marginal structural models, and machine learning with doubly robust estimators. The paper does not support the cited clause’s reference to latent-variable-driven IPW or its stated purpose of clarifying selection into treatment. The Bayesian trajectory-modeling clause cites other sources, not this one. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [100] sourceNot checked — no full text of the source
• cites [101] sourceSupported
• cites [102] sourcePartly supported

  The Introduction and Section 2 describe data-adaptive ensemble learning to estimate inverse-probability weights for marginal structural models, supporting those parts of the claim. The paper does not describe its approach as a doubly robust estimator or as latent-variable-driven IPW. The Bayesian trajectory-modeling clause is cited to other sources, not this paper. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

The Callaway and Sant'Anna estimator uses only the last pre-treatment period for reference, invoking a weaker parallel trends assumption but possibly having lower power than estimators using all pre-intervention periods.

“The Callaway and Sant’Anna [34] estimator uses only the last pre-treatment period for reference, which invokes a weaker version of the parallel trends assumption”

• Also stated in: § Recommendation 3: When Treatment Adoption Is Staggered, Use Saturated Estimators With Clean Comparison Groups, Particularly Those That Estimate Group-Time Effects · hedged

  “but may have correspondingly lower power [108] than estimators that use all pre-intervention periods as reference (e.g., [46]).”

• cites [34] sourceSupported
• cites [108]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [46] sourcePartly supported

  The Introduction (page unavailable) says the authors’ estimator uses all pre-treatment periods while alternatives use more limited information, and reports efficiency gains over versions equivalent to the Callaway and Sant’Anna estimator without extra controls. Section 2 (page unavailable) discusses weaker parallel-trends assumptions and says the Callaway and Sant’Anna assumptions are equivalent to the authors’ assumptions in a one-pre-period case. Our copy does not establish the specific claim that the Callaway and Sant’Anna estimator uses only the last pre-treatment period as its reference. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

• AssumptionThe authors state this as an assumption: Conditional parallel trends: for t1<T*, t2≥T*, and covariates Xi, the expected untreated outcome change is equal across treated and comparison groups conditional on Xi.

Classified as probably not a substantial claim.

Techniques for conditional parallel trends fall into trajectory modeling and propensity score weighting.

“These fall into two main categories, which we denote trajectory modeling [28, 29, 86, 88, 95] and propensity score weighting [17, 30].”

• cites [28] sourceNot checked — no full text of the source
• cites [29] sourceNot checked — no full text of the source
• cites [86] sourceNot checked — no full text of the source
• cites [88] sourceNot checked — no full text of the source
• cites [95] sourceNot checked — no full text of the source
• cites [17] sourceSupported
• cites [30] sourceNot checked — no full text of the source

• AssumptionThe authors state this as an assumption: Conditional parallel trends: for t1<T*, t2≥T*, and covariates Xi, the expected untreated outcome change is equal across treated and comparison groups conditional on Xi.

May combine two claims.

With few clusters, normal-based clustered standard errors are typically anti-conservative, leading to inappropriately small p values and an inflated Type I error rate.

“If there are only a few clusters in the data, normal-based clustered standard errors are typically anti-conservative [114, 115], leading to inappropriately small p values and an inflated Type I error rate.”

• cites [114] sourceNot checked — no full text of the source
• cites [115] sourcePartly supported

  The Introduction (page unavailable) reports that, with few treated units, standard-normal tests using heteroskedasticity-robust variance estimates can over-reject: in one simulation, the rejection rate was 15% at a nominal 5% level. It also says unit-clustered standard errors can massively underestimate uncertainty when there is one treated cluster. This supports the concern about small p values and inflated Type I error, but the cited analysis concerns few treated clusters, not few clusters in general, and does not establish that the problem is typical across all such settings. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

May combine two claims.

Conley and Taber proposed estimating the distribution of errors in treated clusters using untreated clusters; this may be powerful with few treated but many untreated clusters.

“Conley and Taber proposed an approach to estimate the distribution of errors in the treated clusters using that from the untreated clusters [40]. This approach may be powerful when there are only a few treated clusters but many untreated ones from which to estimate the error distribution.”

• cites [40] sourceSupported

May combine two claims.

Normal-based inference requires a substantial number of both treated and untreated clusters, such as at least 25 or 30 in each class, which may not exist in many DiD applications.

“While normal-based inference accounts for error correlation within clusters, it requires a substantial number of both treated and untreated clusters (e.g., at least 25 or 30 clusters in each class), which may not exist in many DiD applications (Table 4, row 1) [17, 114].”

• cites [17] sourcePartly supported

  Section 5.1 says standard DiD inference relies on many treated and untreated clusters for an asymptotic normal approximation, and that many applications have too few, especially treated clusters. It does not specify a threshold of 25 or 30 clusters in each class. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [114] sourceNot checked — no full text of the source

May combine two claims.

Two reviewed papers implemented bootstrap algorithms for inference, one traditional and one wild cluster.

“Two papers implemented bootstrap algorithms for inference, one with traditional bootstrap [118] and another one with wild cluster bootstrap [119].”

• cites [118] sourceNot checked — no full text of the source
• cites [119] sourceSupported

May combine two claims.

Remaining adjustment methods included dynamic TWFE (2), two-stage DiD by Gardner (1), and the d'Chaisemartin and D'Haultfoeuille estimator (1).

“The remaining adjustment methods included dynamic TWFE (n = 2), two-stage DiD proposed by Gardner [47] (n = 1), and the d’Chaisemartin and D’Haultfoeuille estimator [33] (n = 1).”

• cites [47]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [33] sourceSupported

Classified as probably not a substantial claim.

It is recommended to estimate standard errors clustered at the level of treatment assignment.

“In all scenarios, it is recommended to estimate the standard errors clustered at the level of treatment assignment [17, 110-111].”

• cites [17] sourcePartly supported

  Section 5.2 recommends clustering standard errors at the level where treatment is independently assigned, but calls this a rule of thumb from a design-based perspective. The citing sentence’s “in all scenarios” overstates that qualified recommendation; Section 5.1 also discusses alternative inference procedures for settings with few clusters. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites [110]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [111] sourceNot checked — no full text of the source

May combine two claims.

The review surveyed DiD analyses published in three JAMA network journals from January 2022 through November 2024: JAMA had 13, JAMA Internal Medicine had 16, and JAMA Pediatrics had 13.

“we surveyed DiD analyses published in three Journal of American Medical Association (JAMA) network journals between January 2022 and November 2024: JAMA (n = 13, 31%), JAMA Internal Medicine (n = 16, 38%), and JAMA Pediatrics (n = 13, 31%)”

May combine two claims.

None of the reviewed articles discussed non-inferiority tests or calculated test power from pre-intervention data.

“None discussed non-inferiority tests or calculated test power based on pre-intervention data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two reviewed studies used a doubly robust estimator.

“and two used a doubly robust estimator.”

May combine two claims.

Of the 34 reviewed studies that adjusted for covariates, 28 (82%) included only baseline covariates, while six (18%) adjusted for both baseline and time-varying covariates.

“Twenty-eight out of these 34 studies that adjusted for covariates (82%) included only baseline covariates in their model specifications, while the other six (18%) adjusted for both baseline and time-varying covariates.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One reviewed study did not clearly specify its estimator.

“One study did not clearly specify its estimator.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Five reviewed papers (12%) studied an intervention assigned at the individual patient or physician level.

“Five papers (12%) studied an intervention that was assigned at an individual patient or physician level.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

DiD and two-way fixed effects are not interchangeable terms, because the TWFE estimator applies only to some DiD analyses.

“Importantly, the terms “DiD” and “TWFE” are not interchangeable; the TWFE estimator is only used in a subset of DiD analyses.”

May combine two claims.

Saturated models avoid several problems that arise with staggered treatment timing and time-varying covariates and permit more deliberate aggregation of treatment effects across groups.

“Saturated models, which will be discussed further throughout the text, both prevent several problems that can arise when incorporating staggered treatment timing and time-varying covariates and allow researchers to more intentionally aggregate ATTs across different groups.”

Classified as probably not a substantial claim. May combine two claims.

Pre-intervention event study effects can be read as placebo effects that should be small with narrow confidence intervals and should not show trends or anticipation.

“The pre-intervention effects can be interpreted as placebo effects that should be small in magnitude with narrow confidence intervals, without evidence of trends or anticipation effects.”

Probably not a substantial claim: may not pass S2 (checkable).

When confounders vary over time, other study designs may be more appropriate than DiD.

“As a result, in this case, other study designs may be more appropriate than DiD.”

Classified as probably not a substantial claim.

With cross-sectional data, adjusting for covariates that shift the outcome level may be needed because changing sample composition can drive differential trends.

“First, suppose researchers have cross-sectional data, meaning that different individuals are sampled at each observed time point. In this case, researchers may need to account for covariates that shift an outcome’s level, even if they do not affect an individual’s trajectory (Table 2, row 2, Case 1) because changing composition may drive differential trends.”

• AssumptionThe authors state this as an assumption: Conditional parallel trends: for t1<T*, t2≥T*, and covariates Xi, the expected untreated outcome change is equal across treated and comparison groups conditional on Xi.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Post-intervention changes in the comparison group may indicate spillover effects.

“They should also consider whether changes in the comparison group post-intervention may suggest spillover effects.”

• AssumptionThe authors state this as an assumption: One unit's potential outcomes do not depend on other units' treatment status, implying no spillovers.

Classified as probably not a substantial claim.

Event study estimates should not be judged by whether their confidence intervals overlap zero.

“As with tests of slope differences, event study estimates should not be evaluated based on whether confidence intervals overlap zero [24, 76].”

• cites [24]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [76] sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Adding covariates directly into a TWFE specification may not be appropriate.

“However, this approach may not be appropriate.”

Classified as probably not a substantial claim.

With cross-sectional or longitudinal/panel data, adjusting for covariates that shift the outcome trend may be needed.

“Second, in either cross-sectional or longitudinal/panel data (in which the same individuals are sampled at each time point), researchers may adjust for covariates that shift the outcome trend (Table 2, row 3, Case 2).”

• AssumptionThe authors state this as an assumption: Conditional parallel trends: for t1<T*, t2≥T*, and covariates Xi, the expected untreated outcome change is equal across treated and comparison groups conditional on Xi.

Classified as probably not a substantial claim.

No anticipation and SUTVA together allow potential outcomes to be linked to observed outcomes, a condition sometimes called consistency.

“Together, no anticipation and SUTVA allow us to link the potential outcomes to the observed outcomes as follows, which is also sometimes referred to as consistency [44]:”

• cites [44] sourceNot checked — no full text of the source

• AssumptionThe authors state this as an assumption: The treatment has no effect on treated units before the intervention begins.
• AssumptionThe authors state this as an assumption: One unit's potential outcomes do not depend on other units' treatment status, implying no spillovers.

May combine two claims.

With staggered treatment adoption, event study interpretation depends on the specification, and pre-intervention points may not show secular trends unless all are constructed relative to the same pre-intervention period.

“Further, when treatment adoption is staggered, the interpretation of event study plots depends on the specification [79], and pre-intervention points may not display secular trends unless they are all constructed relative to the same pre-intervention period (i.e., base period = “universal” in R or long2 in Stata).”

• cites [79]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Domain knowledge can help identify possible violations of the no-spillover and no-anticipation assumptions.

“Last, domain knowledge may help to inform potential violations of spillover and no-anticipation assumptions.”

• AssumptionThe authors state this as an assumption: The treatment has no effect on treated units before the intervention begins.
• AssumptionThe authors state this as an assumption: One unit's potential outcomes do not depend on other units' treatment status, implying no spillovers.

Classified as probably not a substantial claim.

For treated units, the untreated potential outcome is not observed.

“Note that the untreated potential outcome is unobserved for treated units.”

Probably not a substantial claim: may not pass S2 (checkable).

Conditioning parallel trends on a time-varying variable makes it hard to define underlying parallel trends, because trends may not be parallel in any well-defined subgroup.

“Second, conditioning the parallel trends assumption on a time-varying variable makes it difficult to conceptualize a cohesive notion of underlying parallel trends—if trends are only parallel conditional on a time-varying variable, they may not be, in fact, believed to be parallel in any well-defined subgroup.”

• cites [93]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: Conditional parallel trends: for t1<T*, t2≥T*, and covariates Xi, the expected untreated outcome change is equal across treated and comparison groups conditional on Xi.

Classified as probably not a substantial claim.

Historically, medical researchers mainly used adjustment for observed confounders and epidemiological designs to address selection bias in non-randomized treatment effect estimation.

“Historically, medical researchers relied primarily on adjustment for observed confounders (e.g., regression adjustment, stratification, propensity score weighting, or matching) and epidemiological designs (e.g., case control studies) to address selection bias when estimating treatment effects in non-randomized settings.”

Classified as probably not a substantial claim.

Event study plots are a useful diagnostic for assessing pre-intervention trends.

“Event study plots (e.g., Figure 2) provide another useful diagnostic for evaluating pre-intervention trends.”

May combine two claims.

Econometric causal inference methods such as DiD can adjust for some unobserved confounders and, in some cases, support rigorous causal interpretation of observational results.

“econometric causal inference methods, including difference-in-differences, that can adjust for some unobserved confounders and, in certain cases, allow for rigorous causal interpretations of results from observational studies.”

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [14] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The conditions under which the traditional DiD estimator is valid are well established in the literature.

“The conditions under which the DiD estimator derived above is valid are well-established in the literature.”

Classified as probably not a substantial claim. May combine two claims.

These models can invoke different versions of the parallel trends assumption, and researchers often need to specify the time periods and units over which parallel trends is believed to hold.

“These models can invoke different versions of the parallel trends assumption, and researchers often need to specify the time periods and units over which the parallel trends assumption is believed to hold.”

• AssumptionThe authors state this as an assumption: Conditional parallel trends: for t1<T*, t2≥T*, and covariates Xi, the expected untreated outcome change is equal across treated and comparison groups conditional on Xi.

Probably not a substantial claim: may not pass S3 (load-bearing).

Chen et al. proposed methods to bound violation impact under different assumptions about the magnitude of anticipation effects.

“researchers may consider methods proposed by Chen et al. [73] to bound violation impact under different assumptions about the magnitude of anticipation effects.”

• cites [73]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: No anticipation: treatment has no effect before implementation.

May combine two claims.

In case 2, bias may persist after adding covariates to TWFE because of the time-heterogeneity issue and because naive covariate addition typically does not allow trends to differ by covariate value.

“In Case 2 (cross-sectional or longitudinal data, covariates affect outcome’s trajectory), bias may persist after adding covariates to a TWFE specification both because of the above issue and because naïvely adding covariates typically does not allow trends to differ by covariate value (Table 2, row 3).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Software for group-time estimators often allows user customization; the Sun and Abraham estimator can be implemented in R fixest with choices of reference periods and comparison group.

“In practice, software to implement these estimators also often allows for user customization. For example, the Sun and Abraham [35] estimator, motivated by strong parallel trends (Assumption 4), can be implemented with software (R fixest) that allows users to pick one or multiple reference periods and either never-treated or last-treated units as the comparison group [109].”

• cites [35] sourceNot checked — no full text of the source
• cites [109]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: Strong parallel trends (Assumption 4) holds.

Classified as probably not a substantial claim. May combine two claims.

When parallel trends are violated, researchers might consider potential confounders and use trajectory modeling, propensity score weighting, or doubly robust estimators to account for them.

“In summary, when faced with violations of parallel trends, researchers might carefully consider potential confounders and use trajectory modeling, propensity score weighting, or combined doubly robust estimators to account for them (Table 2, rows 2 and 3).”

• AssumptionThe authors state this as an assumption: Conditional parallel trends: for t1<T*, t2≥T*, and covariates Xi, the expected untreated outcome change is equal across treated and comparison groups conditional on Xi.

May combine two claims.

The dynamic TWFE assumption may not be plausible if early adopters had a different policy experience or if there was selection into treatment.

“This may not be plausible if early adopters had a different experience of the policy (e.g., a pilot phase) or if there was selection into treatment.”

Classified as probably not a substantial claim.

Sensitivity analyses such as bounding can help quantify the robustness of substantive conclusions when causal assumptions are violated.

“Sensitivity analyses like bounding can be helpful to quantify the robustness of substantive conclusions when causal assumptions are violated.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sensitivity analyses can help when researchers suspect violations of the no anticipation assumption.

“Sensitivity analyses can also be helpful when researchers suspect violations of the no anticipation assumption.”

• AssumptionThe authors state this as an assumption: No anticipation: treatment has no effect before implementation.

Classified as probably not a substantial claim.

When treated units initiate treatment at different times, group-time treatment effect estimation methods perform well with heterogeneous treatment effects across time and cohorts.

“When treated units initiate treatment at different times, we recommend that researchers default to group-time treatment effect estimation methods that perform well with heterogeneous treatment effects across time and cohorts.”

Classified as probably not a substantial claim.

Researchers can relax assumptions by assuming parallel trends only in subgroups or conditional on covariates.

“They can also relax assumptions, for example by assuming parallel trends only in subgroups of the data or conditional on covariates.”

• AssumptionThe authors state this as an assumption: Conditional parallel trends: for t1<T*, t2≥T*, and covariates Xi, the expected untreated outcome change is equal across treated and comparison groups conditional on Xi.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

When treatment is assigned at an aggregate level, analyzing individual-level data is unlikely to substantially increase power.

“When treatment is assigned at an aggregate level, analyzing individual-level data is unlikely to substantially increase power.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Normal-based clustered robust inference is one of the most popular methods for clustering standard errors.

“Normal-based clustered robust inference (e.g., cluster robust in Stata) is one of the most popular methods for clustering standard errors.”

May combine two claims.

Wild cluster bootstrap invokes an assumption of a balanced proportion of treated units among clusters, which may not be plausible in many DiD applications.

“However, despite strong simulation performance, the wild cluster bootstrap technically invokes an assumption of a balanced proportion of treated units among clusters, which may not be plausible in many DiD applications where the treatment is assigned at the cluster level (see Canay et al. [117]).”

• cites [117] sourceNot checked — no full text of the source

• AssumptionThe authors state this as an assumption: Wild cluster bootstrap assumes a balanced proportion of treated units among clusters.

Probably not a substantial claim: may not pass S4 (based).

Normal-based clustered inference requires both many treated and many untreated clusters.

“Finally, normal-based clustered inference requires both many treated and many untreated clusters.”

May combine two claims.

The Conley-Taber approach requires errors in treated units to have the same distribution as untreated units, which may be violated under treatment effect heterogeneity.

“However, it requires that errors in the treated units have the same distribution as those in the untreated units, which may be violated under treatment effect heterogeneity.”

• AssumptionThe authors state this as an assumption: The Conley-Taber approach requires that errors in treated units have the same distribution as errors in untreated units.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Researchers should cluster standard errors at the level of treatment assignment and be wary of normal-based cluster inference if there are insufficient treatment or comparison clusters, applying alternatives.

“Taken together, researchers should cluster standard errors at the level of treatment assignment and be wary of normal-based cluster inference if there are insufficient numbers of treatment or comparison clusters and apply alternatives.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

DiD is one of the most popular observational causal inference tools in health policy and medical research.

“DiD is one of the most popular observational causal inference tools in health policy and medical research.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

When an intervention is introduced first in a few cities or states, DiD can support evaluation and learning from early adopters before scaling to additional locations.

“When an intervention is introduced first in a few cities or states, DiD can support evaluation and learning from early adopters before scaling to additional locations.”

May combine two claims.

Simulations suggest wild cluster bootstrap methods can perform well with about five clusters, though estimates may be conservative when the proportion of treated units is small.

“Simulations suggest that these methods can perform well with about five clusters, although estimates may be conservative when the proportion of treated units is small (Table 4, row 2) [39, 114, 116].”

• cites [39] sourceNot checked — no full text of the source
• cites [114] sourceNot checked — no full text of the source
• cites [116] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In DiD studies, treatment generally occurs at an aggregate level, but data may be collected at the same or a more granular level.

“In DiD studies, treatment generally occurs at an aggregate level (e.g., state), but data may be collected either at the same aggregated or a more granular level.”

Probably not a substantial claim: may not pass S2 (checkable).

When there are not many treated and untreated clusters, researchers should choose alternatives that require weaker assumptions.

“When this is not the case, researchers should opt for alternatives that require weaker assumptions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The wild cluster bootstrap, or wild score bootstrap for generalized linear models, is a popular approach for small-cluster inference.

“One popular approach is the wild cluster bootstrap (or wild score bootstrap for generalized linear models) [41].”

• cites [41] sourceNot checked — no full text of the source