On two approaches to weighting in causal inference

Abstract

There are two general approaches to weighting in causal inference. One of them centers on modeling the data—on accurately and flexibly modeling the probabilities of receiving treatment. The other focuses on diagnostics—on directly minimizing and/or constraining covariate imbalances and the dispersion of the weights. In this issue of the Journal, Setodji et al1 offer an interesting comparison of two weighting methods for estimating average treatment effects with time-invariant binary treatments: the use of generalized boosted models to estimate the probability of receiving treatment,2 and the use of the covariate balancing propensity score3 to promote covariate balance between the treatment group and the weighted control group. Claiming at the outset that neither the covariate balancing propensity score nor generalized boosted models should be preferred in all circumstances, Setodji et al1 aim to provide guidance on how to choose between these two methods. In a sense, each stands as a proxy for one of the more general approaches described above: the described motivations for using the covariate balancing propensity score apply just as well to the balancing methods of Hainmueller4 or Chan et al,5 and the motivations for using generalized boosted models apply as well to the use of any flexible nonparametric regression technique to estimate the propensity score. To get to the heart of the issue, we look at the error in estimating the average treatment effect on the treated, , when treatment assignment is unconfounded. This estimand is used frequently in settings with time-invariant binary treatments in which we expect treatment to be made available to a new population similar to the one studied. Our remarks largely apply to average treatment effects over other subsets of the population, including the whole population, with slight changes in the details. Let represents the outcome; , the treatment assignment; and , the observed covariates in an iid sample with treated units and controls. We write , where and . Consider the weighting estimator with weights . The estimators considered by Setodji et al1 are instances of this one using estimated inverse probability weights. Its error in estimating can be decomposed as follows: We focus our attention on the first term in parenthesis—the imbalance in the function , which we will write . This is what we can control and what we need to control for accurate estimation. As long as the weights are bounded and do not depend on the outcomes , the second term converges to zero at rate. Therefore, our primary goal, and the motivation for essentially all weighting methods, is to ensure that the imbalance is small without inflating the variance of the second term. See the appendix, https://links.lww.com/EDE/B258, for details. If we know the treatment assignment probabilities, inverse probability weighting (IPW) does this fairly well: the imbalance decays like under essentially no assumptions on . This suggests that IPW with estimates of the treatment assignment probability, like the generalized boosted models method discussed by the authors, will perform well. However, there are two major criticisms of this approach: (1) IPW estimators are very sensitive to error in estimating the propensity score – in practice, we are often unable to estimate the propensity score well enough to control the bias as if we knew the propensity score, and small propensity score estimates can considerably inflate the variance; (2) or slightly faster decay of the imbalance is not optimal—it is possible to shrink the imbalance much faster. Methods that choose weights to explicitly minimize or constrain the imbalance, like the covariate balancing propensity score or entropy balancing, aim to address both. We will focus on the second criticism. The ability to shrink the imbalance much faster than in some circumstances is a strong reason to prefer these methods. This motivates us to try to detect these favorable circumstances. In order to balance the unknown function , balancing methods choose weights that minimize or bound the level of imbalance uniformly over a class of functions , i.e., weights that control . These methods are clearly preferable to model-based IPW approaches when we are confident that is in or near a set for which we can guarantee that decays fast. However, these classes are small and therefore not likely to contain the true in practice. Often the weights are chosen so that .4,5 We call these exact balancing methods. In order for exact balancing methods to work at all, it is necessary that has dimension no larger than . Perhaps more importantly, unless the dimension of is much smaller than , it is likely that we will need very large weights to balance uniformly; these methods force us to make a tradeoff between misspecification of and large weights that inflate the variance dramatically. In practice, these methods often suffer from both problems at once: we see both excessive variance