Guide for comparing estimators of policy change effects on health.

Abstract

To the Editor: The effects of policy changes on human health are of great interest to epidemiologists. Policies are often introduced in a nonrandomized fashion, and researchers must use observational data to estimate the effects of the policy change. Around 2020, new estimators were proposed mostly in the econometric literature (with one article in the epidemiologic literature) to estimate the average treatment effect on the treated (ATT) for difference-in-differences (DID) settings where polices are introduced at different times across treated units. The two-way fixed-effect estimator, which had previously been the standard approach for these settings, was shown to be biased and/or to estimate an unintuitive target parameter when treatment effects were dynamic or heterogeneous (when effects differed over time or across adoption cohorts, respectively).1 Newly introduced estimators that addressed the limitations of the two-way fixed-effects estimator include the Group-Time ATT estimator introduced by Callaway and Sant’Anna,2 the Cohort ATT estimator introduced by Sun and Abraham,3 and the Target Trial estimator developed for this context by Ben-Michael et al.4 Another tool, the Goodman Bacon decomposition,1 allows researchers to see how the traditional two-way fixed-effects estimator can be biased as a result of dynamic effects, or can estimate a nonintuitive parameter by aggregating effects across heterogeneous effects using weights that epidemiologists would be unlikely to choose. To illustrate the need for new estimators and facilitate their use by epidemiologists, we created an online guide (https://rpubs.com/corinne-riddell/guide-to-did-estimators) that uses R code to compute the ATT estimates across nine scenarios of increasing complexity and describes the findings. A policy can be rolled out in a multitude of ways, and how the policy is introduced and the shape of its effect on the health outcome matter in determining which estimator is appropriate for the analysis. For any policy analysis, the following characteristics can be observed by the researcher and can affect the performance of the two-way fixed-effects estimator and the parameter being estimated: Timing of policy adoption: When do the policy changes occur in reference to the total observation window? Staggered treatment adoption: A policy that is introduced at the same time by all treated units does not have staggered adoption. A policy change that is introduced at different time points by different units has staggered adoption. In addition, the following characteristics cannot be known a priori but also contribute to whether a traditional two-way fixed-effects analysis will produce an unbiased estimate of the policy effect or if a new estimator is more appropriate: Heterogenous treatment effects: If there are multiple adoption cohorts (groups of states that introduce the policy change at the same time), the effect of the policy change is heterogeneous if the magnitude of the effect varies across adoption cohorts. Dynamic treatment effects: A policy has a dynamic treatment effect if its effect on the health outcomes varies as a function of time since the policy’s introduction. These four characteristics, in varying combination, can affect the two-way fixed-effects estimator’s ability to produce an unbiased effect as well as the underlying parameter being estimated.1 For example, heterogeneity of the treatment’s effect across adoption cohorts (characteristic no. 3) changes the parameter being estimated because the two-way fixed-effects estimate is a weighted average of all ATTs, where the weights are a function of both the number of units treated at each time and the treatment variance, which is influenced by the time of policy adoption during the study period (characteristic no. 1).1 This matters because units treated closer to the middle of the study period have larger weights. This is an important aspect of the new literature, as it shows how the two-way fixed-effects estimate can be affected by the timing of policy introductions in terms of the total length of the panel analyzed by the researcher.4 A researcher can obtain a different two-way fixed-effects estimate of the policy’s effect on the outcome solely by adding more prepolicy time (or postpolicy time) to the analysis, because of the contribution of the treatment variance to the weights. Taking a weighted average is common in epidemiology, but the weights chosen by the two-way fixed-effects estimator are hidden by the model, and do not correspond to weights that any “reasonable” epidemiologist would choose. In this setting, the two-way fixed-effects estimate is providing a valid estimate but to the wrong question. Fortunately, the new estimators overcome this issue. This is illustrated in the guide, where we demonstrate how decreasing the amount of time observed before the policy change shifts the two-way fixed-effects estimate but not the other estimates. A separate issue arises wi