Sensitivity to Exogeneity Assumptions in Program Evaluation

Abstract

In many empirical studies of the effect of social programs researchers assume that, conditional on a set of observed covariates, assignment to the treatment is exogenous or unconfounded (aka selection on observables). Often this assumption is not realistic, and researchers are concerned about the robustness of their results to departures from it. One approach (e.g., Charles Manski, 1990) is to entirely drop the exogeneity assumption and investigate what can be learned about treatment effects without it. With unbounded outcomes, and in the absence of alternative identifying assumptions, there are no restrictions on the set of possible values for average treatment effects. This does not mean, however, that all evaluations are equally sensitive to departures from the exogeneity assumption. In this paper I explore an alternative approach, developed by Paul Rosenbaum and Donald Rubin (1983), where the assumption of exogeneity is explicitly relaxed by allowing for a limited amount of correlation between treatment and unobserved components of the outcomes. The starting point of the sensitivity analysis is the assumption that the exogeneity assumption is satisfied only conditional on an additional unobserved covariate. Making assumptions about the effect of the unobserved covariate on the outcome and its correlation with the treatment, I trace out the set of possible values for the treatment effect of interest. By considering a sufficiently large set of possible correlations with outcomes and treatment, one can recover the bounds on the treatment effect derived by Manski (1990). The approach here, in the spirit of Rosenbaum and Rubin (1983) and Rosenbaum (1995), is to allow only a limited amount of correlation and to judge the sensitivity of average treatment-effect estimates to such correlations. There are two novel features of the proposed analysis. First, rather than formulate the sensitivity in terms of coefficients on the unobserved covariate, the sensitivity results are presented in terms of partial R values, which may be easier to interpret. Second, the partial R values of the unobserved covariates are compared to those for the observed covariates in order to facilitate judgments regarding the plausibility of values necessary to substantially change results obtained under exogeneity. The proposed sensitivity analysis is conceptually related to the practice of assessing sensitivity of estimates by comparisons with results obtained by discarding one or more observed covariates (James Heckman and V. Joseph Hotz, 1989; Rajeev Dehejia and Sadek Wahba, 1999; Jeffrey Smith and Petra Todd, 2001). The attraction of the sensitivity analysis is that it is more directly relevant: one is not interested in what would have happened in the absence of covariates actually observed, but in biases that are the result from not observing all relevant covariates.