Unobservable Selection and Coefficient Stability: Theory and Evidence

The study conducted an assessment of selection on unobservables and a falsification test for the main effects estimates.

“We conducted an assessment of selection on unobservables (Oster, 2019) and a falsification test for our main effects estimates, which we describe further in the results and in Appendix C, available on the journal website.”

An Oster (2019) assessment indicated that unobservables would need to be at least three times as related to selection as observables to invalidate the main virtual charter findings.

“In our analysis, we found that unobservables are unlikely to invalidate our main findings for students attending virtual charter high schools because the ratio of unobservables to observables needed to invalidate our findings is by at least 3 times (see Appendix Table C1, available on the journal website).”

The coefficient stability approach proposed by Oster (2019) evaluates how sensitive the estimated DiD treatment effect is to potential omitted-variable bias.

“To assess the robustness of our estimated treatment effects on PR rework, we implement the coefficient stability approach proposed by Oster (2019). This method evaluates how sensitive the estimated DiD treatment effect is to potential omitted-variable bias by comparing changes in the coefficient of interest and the explanatory power of the model when moving from a baseline specification (with only”

For the PR rework outcome, the Oster-adjusted coefficient remains positive and statistically meaningful across all delta values greater than 1.

“For the PR rework outcome, the Oster-adjusted coefficient remains positive and statistically meaningful across all delta values greater than 1”

Oster (2019) and Altonji, Elder and Taber (2005) provide conditions under which stability of coefficients to observable controls can help rule out selection on unobservables.

“Moreover, Oster (2019) and Altonji, Elder and Taber (2005) provide conditions under which the stability of coefficients to the inclusion of observable controls can also help rule out selection on unobservables.”

The estimates are robust to Oster (2019) correction for omitted variable bias.

“We investigate the stability of the treatment effect coefficients and conclude they unlikely suffer from an omitted variable bias.”

The value δ0 of δ that would be required to find an estimate of the treatment effect of zero conditional on setting Rmax=1.5R~ ranges from 5.7 to 16.2, substantially above the value considered appropriate by Oster (2019).

“We find that this value ranges from 5.7 to 16.2, which are substantially above the value considered appropriate by Oster (2019).”

Oster's unrestricted estimator yields estimates largely in line with the regression estimates.

“The method yields estimates largely in line with the regression estimates.”

The omitted variables bias in our setting is negligible.

“In this section, we adopt the approach of Oster (2019) to argue that the omitted variables bias in our setting is negligible.”

Setting δ=1, the value of Rmax that would be needed to estimate a treatment effect of 0 is substantially above those considered appropriate by Oster (2019), and in some cases a negative treatment effect is still found for Rmax=1.

“We obtain values again substantially above those considered appropriate by Oster (2019) and, in some cases, still find a negative treatment effect for R max = 1 .”

Selection-on-unobservables bounding methods show that an implausibly large degree of sorting on unobservables would be needed to explain away the estimated effects of absences on high school completion and college enrollment.

“We apply selection-on-unobservables bounding methods to our baseline value-added models to show that an implausibly large degree of sorting on unobservables is needed to explain away the estimated effects of absences on high school completion and college enrollment”