Policy evaluation during a pandemic

Using variations of difference-in-differences, the authors find a hard-to-interpret mix of policy effects, including large reductions, no effect, or relatively large increases in Covid-19 cases due to SIPOs.

“using variations of DID, we tend to find a hard-to-interpret mix of policy effects that include estimates indicating large reductions in Covid-19 cases due to SIPOs, no effect of SIPOs, or even relatively large increases in Covid-19 cases due to SIPOs.”

The difference-in-differences strategy that includes current Covid-19 cases as a regressor fails when the policy changes the number of cases.

“The second strategy breaks down when the policy affects the number of Covid-19 cases (which is the goal of the policy).”

• AssumptionThe authors state this as an assumption: For economic outcomes, untreated potential outcomes follow a two-way fixed effects model that also depends on the current number of Covid-19 cases in the location.

Unconfoundedness is generally compatible with SIRD model parameters varying over time.

“First, we show that unconfoundedness is generally compatible with SIRD model parameters varying over time; e.g., βlt=βt.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

Additional Covid-19 related policies such as emergency declarations, school closures, and business closures were often implemented around the same time as SIPOs.

“a number of additional Covid-19 related policies such as emergency declarations, school closures, and business closures were often implemented around the same time.”

• Also stated in: § County-level results · plain

  “there are several other Covid-19 policies that were being implemented in a number of states around the same time.”

In general, it is not possible to sign the bias from using standard difference-in-differences or regression difference-in-differences.

“Remark 4In general, it is not possible to sign the bias from using standard DID or regression DID”

In the Monte Carlo simulations for economic outcomes reported in Table 2, the adjusted regression difference-in-differences approach performs well uniformly across data-generating processes.

“the approach suggested in the paper (adjusted regression DID) performs well uniformly across DGPs.”

The standard difference-in-differences strategy for economic outcomes fails because treated and untreated locations differ in current case counts.

“The first strategy breaks down because of differences in the current number of cases across treated and untreated locations.”

• AssumptionThe authors state this as an assumption: For economic outcomes, untreated potential outcomes follow a two-way fixed effects model that also depends on the current number of Covid-19 cases in the location.

The sign of the difference-in-differences bias cannot generally be determined from the expressions in Theorem 1.

“Second, the sign of the bias cannot generally be determined from these expressions.”

The magnitude of difference-in-differences bias is sensitive to the timing of the policy.

“The magnitude of the bias of difference-in-differences is also sensitive to the timing of the policy”

Early treated states, particularly the March 19 group, experienced very large increases in their number of cases relative to later- and never-adopters of the policy.

“early treated states (particularly, the March 19 group) experienced very large increases in their number of cases relative to later- and never-adopters of the policy.”

Imposing the stochastic SIRD model for untreated potential outcomes does not restrict how treated potential outcomes are generated.

“This setup does not place restrictions on how treated potential outcomes (i.e., Covid-19 cases under the policy) are generated.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes for Covid-19 cases follow a stochastic SIRD model.

The setup allows policy effects to be heterogeneous across locations.

“Perhaps more importantly, this setup also allows for heterogeneous effects of policies across different locations.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes for Covid-19 cases follow a stochastic SIRD model.

The unconfoundedness approach is not compatible with model parameters that are specific to particular locations.

“On the other hand, we also show that the unconfoundedness approach is not compatible with model parameters that are specific to particular locations.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

Difference-in-differences is severely biased when the timing of initial cases differs between treated and untreated locations.

“Difference-in-differences is severely biased when the timing of initial cases is different between treated and untreated locations”

States that adopted shelter-in-place orders tended to have many Covid-19 cases earlier than states that did not adopt them or adopted them later.

“states that implemented these policies also tended to have a large number of Covid-19 cases earlier than states that did not implement this type of policy (or implemented it later).”

Under the Stochastic SIRD Model for Untreated Potential Outcomes and Assumption 1, for any post-treatment period, expected untreated cumulative cases for treated locations given pre-treatment characteristics equal those for untreated locations with the same characteristics.

“Proposition 1 In the Stochastic SIRD Model for Untreated Potential Outcomes and under Assumption 1 , and for any t≥t∗ , E[Ct(0)|Ft∗−1,D=1]=E[Ct(0)|Ft∗−1,D=0]”

• AssumptionThe authors state this as an assumption: There is overlap: the probability of treatment is bounded away from zero, and the propensity score is bounded away from one almost surely.
• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

Standard difference-in-differences performs less well, particularly when the timing of initial cases is systematically different across treated and untreated locations.

“standard DID performs less well particularly in cases where the timing of initial cases is systematically different across treated and untreated locations.”

Neither standard difference-in-differences nor difference-in-differences with current Covid-19 cases as a regressor generally recovers the average treatment effect on economic outcomes across treated locations.

“We show that, generally, neither approach can deliver the average effect of the treatment on economic outcomes across treated locations.”

• AssumptionThe authors state this as an assumption: For economic outcomes, untreated potential outcomes follow a two-way fixed effects model that also depends on the current number of Covid-19 cases in the location.

In the stochastic SIRD model, the parallel trends assumption does not generally hold.

“The next result makes explicit that the parallel trends assumption is generally violated in stochastic SIRD models”

• Also stated in: § Using difference-in-differences to evaluate policy effects on Covid-19 cases · plain

  “The parallel trends assumption does not generally hold”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

There were large decreases in travel across all states regardless of their shelter-in-place policies.

“there were large decreases in travel across all states regardless of their shelter-in-place policies.”

The doubly robust unconfoundedness approach performs well across each specification.

“the doubly robust unconfoundedness approach performs much better with good performance across each specification.”

When first cases appear at the same average time across treated and untreated locations, difference-in-differences appears unbiased.

“even in the case where the first cases show up, on average, at the same time across treated and untreated locations (in this case, as expected, DID appears to be unbiased)”

In the economic-outcome simulation, differences in first-case timing violate parallel trends and produce poor estimates of the policy effect on the economic outcome.

“As before, and even in this very simple example, differences in the timing of first cases lead to violations of parallel trends that lead to poor estimates of the effect of the policy on the economic outcome of interest.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes for Covid-19 cases follow a stochastic SIRD model.

Under the model in Eq. (11), the Stochastic SIRD Model for Untreated Potential Outcomes, and Assumption 1, for any post-treatment period, the expected change in untreated active cases for treated locations is identified by Eq. (13).

“Theorem 3In the model for untreated potential outcomes in Eq. (11) and the Stochastic SIRD Model for Untreated Potential Outcomes and under Assumption 1 , and for any t≥t∗ , (13)E[Δ(t∗−1,t)It(0)|D=1]=EDE[D]Δ(t∗−1,t)It−ω(D,Ft∗−1)(It−m0,tI(Ft∗−1))”

• AssumptionThe authors state this as an assumption: There is overlap: the probability of treatment is bounded away from zero, and the propensity score is bounded away from one almost surely.
• AssumptionThe authors state this as an assumption: Untreated potential economic outcomes follow the model in Eq. (11), with a common macro shock, a location fixed effect, a term for untreated active Covid-19 cases, and idiosyncratic time-varying unobservables.
• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

The difference-in-differences approach is quite sensitive to seemingly minor modifications to the specification, such as using the logarithm or level of the outcome or including location-specific linear trends.

“the DID approach is quite sensitive to seemingly minor modifications to the specification such as using the logarithm or level of the outcome or whether or not one includes location-specific linear trends.”

• Also stated in: § State-level results · plain

  “the qualitative conclusion as to whether or not SIPOs affected the number of Covid-19 cases is highly sensitive to functional form assumptions made by the researcher.”

Using county-level data, the authors tend to find relatively small reductions in travel due to SIPOs.

“we tend to find relatively small reductions in travel due to SIPOs.”

• cites Goolsbee and Syverson (2021) sourceNot checked yet

  Not checked in the free run.

Under the model in Eq. (11), standard difference-in-differences and regression difference-in-differences generally deliver biased estimates of the average treatment effect on the treated for economic outcomes.

“We show that both of these approaches generally deliver biased estimates of ATTtY under the model in Eq. (11).”

• AssumptionThe authors state this as an assumption: Untreated potential economic outcomes follow the model in Eq. (11), with a common macro shock, a location fixed effect, a term for untreated active Covid-19 cases, and idiosyncratic time-varying unobservables.
• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

In that same case, the unconfoundedness approach has notably smaller root mean squared error than difference-in-differences.

“the unconfoundedness approach suggested in the paper has notably smaller root mean squared error.”

Difference-in-differences bias for policy evaluation becomes more severe as first-case timing differs more between treated and untreated locations.

“generally, the bias of difference-in-differences approaches for policy evaluation becomes more severe as the timing of first cases becomes more different between treated and untreated locations.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes for Covid-19 cases follow a stochastic SIRD model.

The paper's approach compares locations that had similar pre-treatment pandemics but differed in whether they implemented the policy.

“our approach essentially compares locations that have been experiencing similar pandemics in pre-treatment periods but where some locations implement the policy and others do not.”

Difference-in-differences approaches can, in some cases, lead to unreasonable estimates that SIPOs increased Covid-19 cases.

“We find that DID approaches can, in some cases, lead to unreasonable estimates that SIPOs increased Covid-19 cases.”

In the Monte Carlo simulations for cumulative cases reported in Table 1, the unconfoundedness approach performs better than difference-in-differences across all specifications.

“The high level takeaway from this table is that the unconfoundedness approach uniformly appears to perform better than difference-in-differences.”

Unconfoundedness-type strategies that compare locations with the same pre-treatment Covid-19 variables avoid the drawbacks of difference-in-differences for evaluating policy effects on cases.

“we first show that unconfoundedness-type identification strategies (i.e., strategies that compare locations that have the same pre-treatment values of key Covid-19 related variables) do not suffer from the same drawbacks as difference-in-differences approaches.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes for Covid-19 cases follow a stochastic SIRD model.

Under the Stochastic SIRD Model for Untreated Potential Outcomes and Assumption 1, for any post-treatment period, the average treatment effect on the treated for cumulative cases is identified by the expression in Eq. (9).

“Theorem 2 In the Stochastic SIRD Model for Untreated Potential Outcomes and under Assumption 1 , and for any t≥t∗ , (9)ATTtC=Eω(D,Ft∗−1)Ct−m0,tC(Ft∗−1)”

• AssumptionThe authors state this as an assumption: There is overlap: the probability of treatment is bounded away from zero, and the propensity score is bounded away from one almost surely.
• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

There are substantial differences in pre-treatment characteristics such as the number of Covid-19 cases between states that were early adopters of SIPOs, late adopters, or never implemented a SIPO.

“we document substantial differences in important pre-treatment characteristics such as the number of Covid-19 cases between states that were early adopters of SIPOs, late adopters of SIPOs, or never implemented a SIPO.”

• Also stated in: § State-level results · plain

  “early in the pandemic, the number of cases were substantially different for states that adopted shelter-in-place orders earlier relative to states that adopted them later or that did not adopt them at all.”

The bias from incorrectly imposing the parallel trends assumption in a stochastic SIRD model is given by the difference between sums of expected changes in untreated cases for treated and untreated groups.

“Further, the bias from incorrectly imposing the parallel trends assumption is given by E[Δ(t∗−1,t)Ct|D=1]−E[Δ(t∗−1,t)Ct|D=0]−ATTtC=∑s=t∗tE[E[ΔCs(0)|Ft∗−1,D=1]|D=1]−∑s=t∗tE[E[ΔCs(0)|Ft∗−1,D=0]|D=0]”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

For economic outcomes, the proposed two-step approach identifies the parameters of a two-way fixed effects model that allows current cases to affect the outcome, using untreated locations in the first step.

“we propose a two-step approach where the parameters of a two way fixed effects model that additionally allows for current cases to affect the outcome are identified using untreated locations in the first step.”

• AssumptionThe authors state this as an assumption: For economic outcomes, untreated potential outcomes follow a two-way fixed effects model that also depends on the current number of Covid-19 cases in the location.

When the policy effect is evaluated on impact, the difference-in-differences bias is the difference between expected new cases treated locations would have had absent the policy and expected new cases for untreated locations.

“In that case, the bias from using DID is given by EβIt∗−1(0)NSt∗−1(0)|D=1−EβIt∗−1(0)NSt∗−1(0)|D=0”

• Also stated in: § Using difference-in-differences to evaluate policy effects on Covid-19 cases · plain

  “This bias is the difference between the expected number of new cases that treated locations would have experienced in the absence of the policy relative to the expected number of new cases for untreated locations.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

Across simulations, difference-in-differences tends to over-reject.

“Across simulations, difference-in-differences also tends to over-reject.”

The direction of the difference-in-differences bias is sensitive to the timing of first cases in treated and untreated locations and to the timing of the policy.

“the direction of the bias is sensitive to both (i) timing of first cases in treated and untreated locations, and (ii) the timing of the policy itself”

In the second step, the proposed approach recovers the average path of active cases that treated locations would have had absent treatment, under unconfoundedness-type arguments.

“Then, in the second step, we recover the path of active cases that treated locations would have experienced on average if they had not been treated (this follows under similar unconfoundedness type arguments used for cumulative cases above).”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes for Covid-19 cases follow a stochastic SIRD model.

Under Theorem 3 conditions, the bias of regression difference-in-differences equals negative alpha times the average treatment effect on the treated for active cases.

“and the bias of regression DID (i.e., including current cases as a covariate) is given by E[Δ(t∗−1,t)Yt|D=1]−(τ~t+αE[Δ(t∗−1,t)It|D=1])−ATTtY=−αATTtI where ATTtI≔E[It(1)−It(0)|D=1] .”

• AssumptionThe authors state this as an assumption: Untreated potential economic outcomes follow the model in Eq. (11), with a common macro shock, a location fixed effect, a term for untreated active Covid-19 cases, and idiosyncratic time-varying unobservables.
• AssumptionRests on an assumption: The policy can change the path of active Covid-19 cases and is likely targeted at changing it.

Difference-in-differences sometimes yields meaningfully large placebo shelter-in-place order effects in states that did not implement such an order.

“Using difference-in-differences, we also sometimes spuriously estimate meaningfully large effects of placebo SIPOs in states that did not actually implement a SIPO.”

Difference-in-differences estimates of shelter-in-place order effects are highly sensitive to minor specification changes.

“We additionally show that difference-in-differences estimates are very sensitive to minor changes in how they are specified.”

The proposed estimator is doubly robust: it gives consistent policy effect estimates if either the propensity score or the outcome regression is correctly specified.

“Our approach is doubly robust in the sense that it delivers consistent estimates of policy effects if either the propensity score or outcome regression is correctly specified.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes for Covid-19 cases follow a stochastic SIRD model.

Using the unconfoundedness approach, the authors estimate relatively small and statistically insignificant effects of SIPOs on Covid-19 cases.

“Using the unconfoundedness approach, we estimate relatively small and statistically insignificant effects of SIPOs.”

When the outcome is the level of the number of cases and the specification includes a linear trend, the estimates indicate that SIPOs may have reduced the number of Covid-19 cases, with results closer to zero and only marginally statistically significant.

“The results in panel (c), which also include a linear trend but where the outcome is the level of the number of cases, also indicate that SIPOs may have reduced the number of Covid-19 cases (these results are closer to zero and only marginally statistically significant).”

For Minnesota, the DID estimates are quite similar to the unconfoundedness results.

“For Minnesota, the DID estimates are quite similar to the unconfoundedness results presented here.”

All states (not just those who implemented the policy) were experiencing massive decreases in travel over the period considered.

“This can be seen to be clearly relevant from the summary statistics in Table 3 where all states (not just those who implemented the policy) were experiencing massive decreases in travel over the period that we consider.”

Using DID, the placebo policy in South Dakota is spuriously estimated to have reduced Covid-19 cases.

“More notably, using DID, we spuriously estimate that the placebo policy in South Dakota reduced the number of Covid-19 cases in South Dakota.”

For travel, the estimates are more broadly similar than for cumulative Covid-19 cases.

“In this case, the estimates are more broadly similar than they were for cumulative Covid-19 cases.”

Using standard DID, the overall estimate of the effect of SIPOs on travel across post-treatment periods is −1.30 (p=0.16), i.e., a reduction of about 1.3 percent.

“Using standard DID, the overall estimate of the effect of SIPOs across post-treatment time periods is −1.30 (p-value: 0.16); i.e., across the first several weeks of the SIPO, the policy reduced travel by about 1.3 percent relative to what travel would have been if the policy had not been implemented.”

Without a location-specific linear trend, the estimates suggest that SIPOs led to an increase in the number of Covid-19 cases.

“the results in panels (a) and (b) of Fig. 5, neither of which include a location-specific linear trend, are much different and suggest that SIPOs led to an increase in the number of Covid-19 cases.”

On April 25, Minnesota's SIPO reduced Covid-19 cases by about 0.46 cases per 1000 people in the state.

“For example, on April 25, we estimate that Minnesota’s SIPO reduced Covid-19 cases by about 0.46 cases per 1000 people in the state.”

None of the four specifications in Fig. 5 result in either large or statistically significant violations of the parallel trends assumption in pre-treatment periods.

“none of the four specifications used in the figure result in either large or statistically significant violations of the underlying parallel trends assumption in pre-treatment periods”

Using adjusted regression DID, the estimated overall effect of SIPOs on travel is −1.25 (p=0.19).

“Finally, the adjusted regression DID overall estimated effect of SIPOs is −1.25 (p-value: 0.19) when we allow for the policy to have had an effect on cases.”

There is more variation in non-essential business closures and gathering restrictions across states.

“There is more variation with respect to non-essential business closures and gathering restrictions”

Using regression DID, the estimated overall effect of SIPOs on travel is −1.63 (p=0.08).

“The point estimates from regression DID (these estimates just include observed current cases as a covariate) are somewhat larger in magnitude; in this case, the estimated overall effect of SIPOs on travel is −1.63 (p-value: 0.08).”

There is variation in treatment timing for SIPOs.

“There is certainly variation in treatment timing for SIPOs”

Results that include location-specific linear trends are notably different from the results in Fig. 5.

“the results that include location-specific linear trends are notably different from the results in Fig. 5.”

For Minnesota, under unconfoundedness, the SIPO decreased Covid-19 cases at least in some periods.

“On the other hand, for Minnesota, we estimate that its SIPO decreased Covid-19 cases at least in some periods.”

The primary difference is that Iowa closed its schools somewhat later than surrounding states, with the exception of Nebraska.

“The primary difference is that Iowa closed its schools somewhat later than surrounding states with the exception of Nebraska.”

Another large number of states from the South are dropped because the timing of their policies is very similar, making it challenging to find reasonable comparison states when region is included as a conditioning variable.

“Another large number of states from the South are dropped due to the timing of their policies being very similar which makes it challenging to find reasonable comparison states when region is included as a conditioning variable.”

When the outcome is the logarithm of the number of Covid-19 cases and the specification includes a location-specific linear time trend, the estimates indicate a large reduction in Covid-19 cases due to SIPOs.

“when the outcome is the logarithm of the number of Covid-19 cases and the specification includes a location-specific linear time trend, then the estimates indicate a large reduction in Covid-19 cases due to shelter-in-place policies”

Arkansas tended to implement other policies with very similar timing as all of its surrounding states, especially with respect to emergency declaration and school closures.

“Overall, Arkansas tended to implement other policies with very similar timing as all of its surrounding states especially with respect to the timing of the emergency declaration and school closures.”

Standard DID estimates indicate a relatively small but persistent negative effect of SIPOs on retail and recreation travel.

“Standard DID estimates indicate a relatively small but persistent negative effect of SIPOs on retail and recreation travel.”

The authors do not find large or any statistically significant estimates of Covid-19 policies in pre-treatment periods.

“we do not find large or any statistically significant estimates of Covid-19 policies in pre-treatment periods.”

The timing of Iowa's policies is somewhat more different from its surrounding states.

“The timing of Iowa’s policies are somewhat more different from its surrounding states.”

For Missouri, under unconfoundedness, there is no evidence of SIPO effects on Covid-19 cases.

“For Missouri, we do not find evidence of policy effects on Covid-19 cases.”

Difference-in-differences generally delivers potentially severely biased estimates of a policy's effect on cumulative Covid-19 cases.

“Theorem 1 shows that difference-in-differences generally delivers (potentially severely) biased estimates of the effect of a policy on cumulative Covid-19 cases.”

• AssumptionRests on an assumption: Locations generally do not differ by time-invariant unobservables that make them more or less likely to have many Covid-19 cases; the key differences are the timing of initial cases and the pandemic response.
• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

Difference-in-differences is less suitable for pandemic policy evaluation than it usually is in economics applications.

“In the current paper, we argue that difference-in-differences has properties that make it relatively less attractive for conducting policy evaluation during a pandemic than it typically would be for most applications in economics.”

If the joint distribution of currently infected and susceptible individuals differs between treated and untreated locations, they would have experienced different numbers of new Covid-19 cases even without the policy.

“if the joint distribution of currently infected and number of susceptible individuals is different among treated and untreated locations, then they would have experienced a different number of new Covid-19 cases even if the policy had not been implemented.”

The paper's proposed alternative approaches can avoid the issues identified with difference-in-differences.

“We propose alternative approaches that are able to circumvent these issues.”

In the context of Covid-19, difference-in-differences biases could be substantial.

“In the context of Covid-19, there are some cases where these biases could be substantial.”

The doubly robust approach allows the authors to avoid estimating a full structural epidemic model while still obtaining policy effect estimates compatible with the epidemic model.

“This is important as it implies that we can circumvent having to estimate a full structural epidemic model while still delivering estimates of policy effects that are compatible with the epidemic model.”

Pre-policy differences in pandemic state are a major concern for early pandemic policy evaluation because policy choices often depended on local pandemic conditions.

“Moreover, these differences are a major concern for evaluating early pandemic policies where different locations’ policy choices often depended on the state of the pandemic in those locations.”

The expression for the average treatment effect on the treated for cumulative cases in Eq. (9) possesses the double robustness property.

“However, the expression for ATTtC in Eq. (9) possesses the double robustness property.7”

• cites Bang and Robins (2005) sourceNot checked yet

  Not checked in the free run.

• cites Słoczyński and Wooldridge (2018) sourceNot checked yet

  Not checked in the free run.

The identification arguments for fixed policy timing can be extended to settings where policy timing varies across locations.

“However, it is straightforward to extend these arguments to the case where the timing of the policy varies across locations”

For difference-in-differences, the same problem persists for economic outcomes when those outcomes depend on the number of Covid-19 cases.

“For difference-in-differences, we further show that a version of this problem continues to exist even when one is interested in understanding the effect of a policy on other economic outcomes when those outcomes also depend on the number of Covid-19 cases.”

• AssumptionThe authors state this as an assumption: For economic outcomes, untreated potential outcomes follow a two-way fixed effects model that also depends on the current number of Covid-19 cases in the location.

It is not feasible to select among alternative difference-in-differences specifications based on their pre-treatment performance.

“This suggests that it is not feasible to choose between alternative difference-in-differences specifications based on their performance in pre-treatment periods.”

This correlation mechanically produces larger case increases among early-treated states than among untreated or later-treated states.

“This correlation mechanically leads to larger increases in Covid-19 cases among early treated states relative to untreated and later-treated states.”

Under certain conditions, the Alemán et al. approach can evaluate nationwide pandemic policies with common timing by using sub-national variation in pandemic stage, a setting where the paper's approach would not work.

“under certain conditions, the approach in Alemán et al. (2020) can be used to evaluate nationwide pandemic-related policies with common timing across locations by exploiting sub-national variation in the “stage” of the pandemic. This is a setting where our approach would not be viable.”

• cites Alemán et al. (2020)Not checked yet

  Not checked in the free run.

Reducing the infection rate through non-pharmaceutical interventions lowers the total number of cases needed before herd immunity is reached.

“decreasing the infection rate through non-pharmaceutical interventions decreases the total number of cases that need to occur before reaching herd immunity”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes for Covid-19 cases follow a stochastic SIRD model.

The bias of regression difference-in-differences comes from incorrectly setting expected untreated active case changes for treated locations equal to observed active case changes for treated locations.

“The bias of regression DID (that directly includes current cases as a covariate) comes from (incorrectly) setting E[Δ(t∗−1,t)It(0)|D=1]=E[Δ(t∗−1,t)It|D=1].”

• AssumptionThe authors state this as an assumption: Untreated potential economic outcomes follow the model in Eq. (11), with a common macro shock, a location fixed effect, a term for untreated active Covid-19 cases, and idiosyncratic time-varying unobservables.
• AssumptionRests on an assumption: The policy can change the path of active Covid-19 cases and is likely targeted at changing it.

It will be challenging for difference-in-differences to perform well at evaluating the effect of SIPOs on the number of cases.

“This immediately suggests that it will be challenging for DID to perform well at evaluating the effect of shelter-in-place orders on the number of cases.”

Unconfoundedness approaches that condition on the pre-treatment pandemic state are likely more useful for evaluating Covid-19 policies than difference-in-differences approaches, because case spread during a pandemic is highly nonlinear.

“We argue that unconfoundedness type approaches, that condition on the pre-treatment “state” of the pandemic, are likely to be more useful for evaluating policies than difference-in-differences type approaches due to the highly nonlinear spread of cases during a pandemic.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes for Covid-19 cases follow a stochastic SIRD model.

The expected number of new Covid-19 cases from one period to the next depends on the infected fraction, the number of susceptible people, and the infection rate parameter.

“The expected number of new cases from one period to the next depends on three things. First, it depends on Ilt−1/Nl which is the fraction of individuals that are infected in period t−1.”

• Also stated in: § Notation: · plain

  “Second, the expected number of new cases depends on the number of susceptible individuals in the population.”

• Also stated in: § Notation: · plain

  “Finally, the change in the number of cases depends on the parameter β which is called the infection rate.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes for Covid-19 cases follow a stochastic SIRD model.

In regression difference-in-differences, bias arises because the approach does not accommodate the possibility that the policy affects current cases, which in turn affects outcomes.

“In the regression DID case, biased estimates arise because the approach does not accommodate the possibility that the policy has an effect on the current number of cases (which in turn has an effect on outcomes).”

• AssumptionThe authors state this as an assumption: Untreated potential economic outcomes follow the model in Eq. (11), with a common macro shock, a location fixed effect, a term for untreated active Covid-19 cases, and idiosyncratic time-varying unobservables.
• AssumptionRests on an assumption: The policy can change the path of active Covid-19 cases and is likely targeted at changing it.

In standard difference-in-differences, bias arises because the researcher does not account for current cases having a direct effect on outcomes.

“In the standard DID case, biased estimates arise because the researcher does not account for current cases in a particular location having a direct effect on outcomes.”

• AssumptionThe authors state this as an assumption: Untreated potential economic outcomes follow the model in Eq. (11), with a common macro shock, a location fixed effect, a term for untreated active Covid-19 cases, and idiosyncratic time-varying unobservables.

The key reason difference-in-differences breaks down is that the distribution of pandemic-related variables immediately before the policy is not generally the same across treated and untreated locations.

“First, the key reason why the difference-in-differences strategy breaks down is that, in general, the distribution of pandemic related variables immediately before the policy (contained in Ft∗−1) is not the same across treated and untreated locations.”

• AssumptionRests on an assumption: Locations generally do not differ by time-invariant unobservables that make them more or less likely to have many Covid-19 cases; the key differences are the timing of initial cases and the pandemic response.
• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

The same arguments for policy effects on cumulative cases apply to other variables such as the current number of infections.

“However, the same sorts of arguments apply to other possible variables of interest such as the current number of infections.”

The parallel trends assumption underlying difference-in-differences is likely violated in this application.

“this suggests that the parallel trends assumption underlying the DID approach is likely to be violated.”

The bias of standard difference-in-differences arises from incorrectly setting expected untreated outcome changes for treated locations equal to observed outcome changes for untreated locations.

“The bias of standard DID arises from (incorrectly) setting E[Δ(t∗−1,t)Yt(0)|D=1]=E[Δ(t∗−1,t)Yt|D=0].”

• AssumptionThe authors state this as an assumption: Untreated potential economic outcomes follow the model in Eq. (11), with a common macro shock, a location fixed effect, a term for untreated active Covid-19 cases, and idiosyncratic time-varying unobservables.

A doubly robust estimator provides consistent estimates of the average treatment effect on the treated for cumulative cases if either the propensity score model or the outcome regression model is correctly specified.

“A main advantage of a doubly robust estimator is that it provides consistent estimates of ATTtC if either the propensity score model or the outcome regression model is correctly specified”

• cites Bang and Robins (2005) sourceNot checked yet

  Not checked in the free run.

• cites Słoczyński and Wooldridge (2018) sourceNot checked yet

  Not checked in the free run.

Proposition 1 implies that, on average, the unobserved cumulative cases treated locations would have experienced absent treatment equal the cumulative cases untreated locations actually experienced among locations with the same pre-treatment characteristics.

“This is an important result and implies that, on average, the unobserved number of cumulative cases that locations that participated in the treatment would have experienced if they had not participated in the treatment is the same as the cumulative number of cases that untreated locations actually did experience among locations that had the same pre-treatment characteristics.”

A leading example of substantial difference-in-differences bias is when initial Covid-19 cases varied across locations and policies were implemented earlier in locations that tended to have cases earlier.

“Perhaps the leading example is when the timing of initial Covid-19 cases varied across locations and Covid-19 related policies were implemented earlier in locations that tended to have cases earlier.”

Over relatively short time horizons, under the stated conditions, both standard difference-in-differences and regression difference-in-differences will overstate the magnitude of the policy effect.

“In this case, both standard DID and regression DID (that adjusts for the actual number of cases) will both overstate the magnitude of the effect of the policy.”

• AssumptionThe authors state this as an assumption: Untreated potential economic outcomes follow the model in Eq. (11), with a common macro shock, a location fixed effect, a term for untreated active Covid-19 cases, and idiosyncratic time-varying unobservables.
• AssumptionRests on an assumption: The policy can change the path of active Covid-19 cases and is likely targeted at changing it.

Due to the nonlinearity of the SIRD model, differences in pre-policy pandemic variables lead to violations of the parallel trends assumption.

“Due to the nonlinearity of the SIRD model, this leads to violations of the parallel trends assumption.”

• AssumptionThe authors state this as an assumption: Untreated potential outcomes are generated by the Stochastic SIRD Model for Untreated Potential Outcomes.

Leading epidemiological models of Covid-19 transmission are highly nonlinear and depend on the number of currently infected people, the number of susceptible people, and transmission properties.

“the main epidemiological models for Covid-19 transmission are highly nonlinear and depend on (i) the number of currently infected individuals in a particular location, (ii) the number of susceptible individuals in a location, and (iii) the transmission properties of Covid-19.”

The proposed approach uses Eq. (13) to recover untreated active cases by subtracting the policy effect on active cases from observed active cases.

“By contrast, our approach uses the expression in Eq. (13) for E[Δ(t∗−1,t)It(0)|D=1]. This expression takes the observed path of active cases and subtracts from it the effect of the policy on active cases”

• AssumptionThe authors state this as an assumption: Untreated potential economic outcomes follow the model in Eq. (11), with a common macro shock, a location fixed effect, a term for untreated active Covid-19 cases, and idiosyncratic time-varying unobservables.

Under a stochastic SIRD model, a policy effect can be evaluated using an unconfoundedness strategy that compares cases in treated locations to cases in untreated locations with the same pre-policy Covid-19 characteristics.

“Theorem 2 says that, under a stochastic SIRD model, we can evaluate the effect of a policy using an unconfoundedness strategy that compares the number of cases in locations that participated in the treatment to the number of cases in locations that did not participate in the treatment and which had the same Covid-19 related characteristics in the period before the policy was implemented.”

Two common ways to evaluate policies (DID directly or including cases as a covariate in a DID setup) do not generally deliver an average effect of the policy when the policy can affect cases and cases can affect the economic outcome.

“For this case, we also showed that two of the most common ways to evaluate these policies (difference-in-differences directly or including the number of cases as a covariate in a DID setup) do not generally deliver an average effect of the policy.”

For these two placebo policies, the unconfoundedness approach performs better than DID.

“These results suggest that, at least for these two policies, the unconfoundedness approach performs better than DID.”

Our results on the effect of SIPOs on travel are more similar to existing estimates (e.g., Goolsbee and Syverson (2021)).

“First, our results on the effect of SIPOs on travel are more similar to existing estimates (e.g., Goolsbee and Syverson (2021)).”

• cites Goolsbee and Syverson (2021) sourceNot checked yet

  Not checked in the free run.

The proposed alternative estimator is valid in this case.

“We proposed an alternative estimator that is valid in this case.”

None of the TWFE event study specifications discussed are compatible with the SIRD model discussed in the paper.

“none of the TWFE event study specifications discussed here are compatible with the SIRD model that we have discussed in the paper.”

Our results on the effect of SIPOs on the number of Covid-19 cases are substantially different from existing estimates.

“Our results, especially those about the effect of SIPOs on the number of Covid-19 cases, are substantially different from existing estimates, and it is worth making a few additional comments.”

A better explanation for the differences is that they are driven by pre-treatment differences in the state of the pandemic between counties in states that implemented SIPOs relative to counties in neighboring states that did not.

“Instead, a better explanation seems to be that these differences are driven by pre-treatment differences in the state of the pandemic between counties in states that implemented SIPOs relative to counties in neighboring states that did not.”

Difference-in-differences estimates are likely to be positive because early policy adopters were tending to experience larger increases in cases.

“DID estimates are likely to be positive as early policy adopters were tending to experience larger increases in cases.”

It is reasonable to consider the mix and timing of other policies as being quite similar to all of Arkansas's neighboring states.

“Our interpretation is that it is reasonable to consider the mix and timing of other policies as being quite similar to all of its neighboring states.”

Difference-in-differences would be susceptible to attributing faster growth in Covid-19 cases to the policy rather than to simply a larger number of early cases in treated locations.

“It is exactly in this case where DID would be susceptible to attributing faster growth in Covid-19 cases to the policy rather than to simply a larger number of early cases in treated locations.”

It is hard to find reasonable comparisons for New York and California.

“It is, therefore, hard to find reasonable comparisons for these states.”

Many of our estimates, though not statistically significant, are large enough to be compatible with a wide variety of possible effects of SIPOs on Covid-19 cases.

“And even though a number of our estimates are not statistically significant, many of them are large enough to be compatible with a wide variety of possible effects of SIPOs on Covid-19 cases.”

Heterogeneity robust difference-in-differences estimation strategies provide a number of advantages relative to TWFE event studies.

“these estimation strategies provide a number of advantages relative to the TWFE event studies”

• cites Callaway and Sant’Anna (2021) sourceNot checked yet

  Not checked in the free run.

• cites Gardner (2022)Not checked yet

  Not checked in the free run.

It is not possible to use a purely data-driven or reduced-form model selection procedure to infer that one specification is likely to perform better than the others, despite the choice of model largely driving the results.

“This suggests that it is not possible to use a purely data-driven/reduced form model selection procedure to infer that one specification is likely to perform better than the others, despite the choice of the model largely driving the results.”

The large magnitude of differences in estimated policy effects cuts against heterogeneous policy effects being a full explanation of the differences.

“Arguably, these differences could be explained by heterogeneous effects of policies implemented in different states, though, in our view, the large magnitude of the differences in estimated policy effects cuts against heterogeneous policy effects being a full explanation of the differences.”

For identifying direct effects of policies on Covid-19 cases, strategies based on unconfoundedness type assumptions are likely to perform better than difference-in-differences type strategies due to the highly nonlinear nature of Covid-19 spread.

“For identifying the direct effects of policies on the number of Covid-19 cases, our results suggest that strategies based on unconfoundedness type assumptions are likely to perform better than difference-in-differences type strategies due to the highly nonlinear nature of the spread of Covid-19.”