How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It

10.7% (8 of 75) of the reviewed experimental studies engaged in both controlling for a posttreatment covariate and subsetting based on posttreatment criteria.

“almost 1 in 10 engaged in both (10.7%, eight studies)”

The discipline's most prestigious journals frequently publish experimental studies using posttreatment conditioning practices.

“which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals”

14.7% (11 of 75) of the reviewed experimental studies subsetted the data based on potential posttreatment criteria.

“14.7% of studies subsetted the data based on potential posttreatment criteria (11 of 75 studies reviewed)”

Problems associated with conditioning on posttreatment variables remain largely unrecognized in the field.

“problems associated with conditioning on posttreatment variables remain largely unrecognized in the field”

21.3% (16 of 75) of the reviewed experimental studies controlled for a posttreatment covariate in a statistical model.

“21.3%(16 of 75) controlled for a posttreatment covariate in a statistical model”

Among the reviewed studies that controlled for a posttreatment variable, six used a mediation technique (8%).

“Among the studies that controlled for a posttreatment variable, six used a mediation technique (8%).”

12% (9 of 75) of the reviewed studies conditioned on a variable shown to be affected by the experimental treatment in analyses contained within the article itself.

“Most strikingly, 12% of studies conditioned on a variable shown to be affected by the experimental treatment in analyses contained within the article itself (9 of 75).”

Few of the reviewed studies acknowledge potential concerns regarding the bias that posttreatment conditioning can introduce.

“few acknowledge potential concerns regarding the bias that posttreatment conditioning can introduce”

Among respondents who perceive themselves as further from Eid, the magnitude of the negative coefficient approximately doubles relative to the unconditional estimate (-.460; 95% CI: -0.567, -0.353).

“the magnitude of the negative coefficient approximately doubles relative to the unconditional estimate among respondents who perceive themselves as further from Eid (-.460; 95% CI: -0.567, -0.353).”

Trump's endorsement reduced the likelihood of supporting Eid's retention by -0.214 (p < .01, 95% CI: -0.301, -0.127) on the four-point scale.

“Trump's endorsement reduced the likelihood of supporting Eid's retention by -0.214 (p < .01,95% CI: -0.301, -0.127) on the four-point scale.”

Conditioning on ideological distance unbalances the sample by respondents' self-reported ideology even though self-reported ideology is measured pretreatment.

“As described in the subsection “Practices That Lead to Posttreatment Bias,” conditioning on ideological distance actually unbalances the sample by respondents' self-reported ideology even though self-reported ideology is measured pretreatment.”

Among respondents who perceive themselves as ideologically close to Eid, treatment group respondents are significantly more conservative on the 7-point ideology scale than control group respondents (4.633 versus 3.652, p < .01).

“For instance, among respondents who perceive themselves as ideologically close to Eid, treatment group respondents are significantly more conservative on our 7-point ideology scale than are control group respondents (4.633 versus 3.652, p < .01 in a t-test).”

The treatment increases perceptions of Eid's conservatism (from 3.821 in the control group to 4.781 in the treatment group, p < .01).

“The reason is that the treatment increases perceptions of Eid's conservatism (from 3.821 in the control group to 4.781 in the treatment group, p < .01 in a t-test).”

This effect becomes significant at the p < .05 level in the fourth column when cases are dropped and lagged controls are included.

“This effect becomes significant at the p < .05level in the fourth column when we drop cases and include lagged controls.”

These results are largely unchanged when lagged behavioral controls are included in the second column.

“These results are largely unchanged when we include lagged behavioral controls in the second column.”

In the first column omitting posttreatment controls or conditioning, the appropriations treatment is significant only in the full information condition.

“The first column, which omits any posttreatment controls or conditioning, shows that the appropriations treatment is significant only in the full information condition.”

When cases based on contributor targeting are dropped in the third column, the limited information treatment becomes significant at the p < .10 level in the salary condition.

“However, when we instead drop cases based on contributor targeting in the third column, the limited information treatment becomes significant at the p < .10 level in the salary condition.”

The magnitude of the effect estimates varies substantially when conditioning on posttreatment variables.

“In addition, we find that the magnitude of the effect estimates varies substantially when we condition on posttreatment variables.”

Posttreatment conditioning induces substantial differences in the estimated effects of DGH's treatments.

“Table 3 demonstrates that posttreatment conditioning induces substantial differences in the estimated effects of DGH's treatments.”

The paper replicates the appropriations treatment effect for full information groups.

“We replicate the appropriations treatment effect for full information groups, but our analysis raises concerns about posttreatment bias for both the limited information effect in the salary condition and the appropriation effect in the limited information condition.”

The lagged measures used by DGH are themselves affected by the manipulations.

“Unfortunately, the lagged measures are themselves affected by the manipulations.”

A significant fraction of the experimental studies published in the discipline's most prestigious journals drop observations based on posttreatment variables or control for posttreatment variables in their statistical analysis.

“We find that a significant fraction of the experimental studies published in the discipline's most prestigious journals drop observations based on posttreatment variables or control for posttreatment variables in their statistical analysis.”

52.0% of the sample have no conditioning on posttreatment variables.

“| No conditioning on posttreatment variables | 52.0% |”

21.3% of the sample control for or interact with a posttreatment variable.

“| Controls for/interacts with a | 21.3% |”

14.7% of the sample drop cases based on posttreatment criteria.

“| Drops cases based on posttreatment | 14.7% |”

In the sample of 75 experimental studies from 2012-14 in APSR, AJPS, and JOP, 46.7% engage in posttreatment conditioning.

“| Engages in posttreatment conditioning | 46.7% |”

The evidence provided demonstrates that posttreatment conditioning is a frequent and significant problem in political science.

“In total, the evidence we provide demonstrates that posttreatment conditioning is a frequent and significant problem in political science.”

10.7% of the sample have both types of posttreatment conditioning present.

“| Both types of posttreatment conditioning | 10.7% |”

Scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation.

“However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation.”

Conditioning on posttreatment variables, including controlling for them, eliminating observations based on posttreatment criteria, or subsetting data based on them, can bias estimates of causal effects.

“Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects.”

Any method that generates an unbiased estimate of the DCEV will also generate an unbiased estimate of the ATE.

“Any method that generates an unbiased estimate of the DCEV ( ) will then also generate an unbiased estimate of the ATE ( ).”

Conditioning on posttreatment variables eliminates the advantages of randomization because it compares dissimilar groups.

“Conditioning on posttreatment variables eliminates the advantages of randomization because we are now comparing dissimilar groups.”

The discipline does not understand posttreatment bias.

“Empirically, then, the answer to the question of whether the discipline already understands posttreatment bias is clear: It does not.”

By conditioning on a posttreatment variable, treatment and control groups become unbalanced with respect to every other possible confounder.

“The problem is that by conditioning on a posttreatment variable, we have unbalanced the treatment and control groups with respect to every other possible confounder.”

If Assumption (1) is violated, one would also need to adjust for bias resulting from imbalances in the unobserved confounder u.

“if we violate Assumption (1), we would also need to somehow adjust for bias resulting from imbalances in the unobserved confounder u”

• AssumptionThe authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.

Nearly half of the experimental studies published in the discipline's most prestigious journals during 2012-2014 raise concerns about posttreatment bias.

“In short, nearly half of the experimental studies published in our discipline's most prestigious journals during this period raise concerns about posttreatment bias.”

For any finite ATE, examples can be constructed where the bias from imbalance in observed or unobserved covariates is -∞, ∞, or anything in between depending on parameter values.

“For any finite ATE, we can construct examples where the bias will be -∞, ∞, or anything in between depending on the value of parameters like Y”

Once we have conditioned on a posttreatment variable, we have eliminated the assurance of unconfoundedness provided by randomization.

“Once we have conditioned on a posttreatment variable, we have eliminated the assurance of unconfoundedness provided by randomization.”

Controlling for a posttreatment variable leads to a new bias that is a weighted combination of the biases from selecting only cases where x=1 and cases where x=0.

“controlling for a posttreatment variable leads to a new bias that is simply a combination of the biased estimates we would get from selecting only cases where x = 1 and the estimates from selecting only cases where x = 0.”

Even when the posttreatment covariate has no effect on the outcome, conditioning on it still leads to inconsistent estimates because the posttreatment covariate and the outcome share an unmeasured cause.

“Even in such a favorable context, conditioning on x still leads to inconsistent estimates because the posttreatment covariate (x) and the outcome (y) share an unmeasured cause (u).”

By selecting based on a criterion that is partially a function of unobserved covariates and the treatment, researchers inadvertently create imbalance in the treatment and control conditions with respect to u.

“By selecting based on a criterion that is partially a function of unobserved covariates and the treatment, we have inadvertently created imbalance in the treatment and control conditions with respect to u.”

Including posttreatment variables as covariates can bias estimates by creating imbalance with respect to the unmeasured confounder.

“including posttreatment variables as covariates for any of these reasons can bias estimates by creating imbalance with respect to the unmeasured confounder.”

Because perceptions of Eid's ideology were measured after the manipulation, any analysis that conditions on ideological distance will be biased.

“Unfortunately, because perceptions of Eid's ideology were measured after the manipulation, any analysis that conditions on ideological distance will be biased.”

Adjusting for imbalance in unobservable variables is more challenging, requiring exogenous instruments and/or stronger assumptions.

“Adjusting for imbalance in unobservable variables is more challenging, requiring either the availability of exogenous instruments and/or stronger (and more limiting) assumptions”

Standard estimates such as the difference in conditional means will be biased regardless of sample size, measurement precision, or estimation method when conditioning on a posttreatment variable.

“Standard estimates such as the difference in conditional means ( ˆ ) will therefore be biased regardless of sample size, measurement precision, or estimation method.”

When a posttreatment variable is included in the set of conditioning variables either directly or indirectly, Assumption (1) is violated.

“when we include a posttreatment variable in the set of conditioning variables either directly or indirectly, Assumption (1) is violated.”

• AssumptionThe authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.

Attrition rates in the judge perceptions experiment were 2.3% in control and 1.5% in treatment.

“Attrition rates were 2.3% in control and 1.5% in treatment.”

Published research in political science identified posttreatment bias as problematic over a decade ago.

“published research in political science identified posttreatment bias (in passing) as problematic over a decade ago”

• cites King and Zeng 2006Not checked yet

  Not checked in the free run.

Acharya, Blackwell, and Sen (2016) show that as many as four out of five observational studies in top journals may condition on posttreatment variables.

“Acharya, Blackwell, and Sen (2016), for instance, show that as many as four out of five observational studies in top journals may condition on posttreatment variables.”

• cites Acharya, Blackwell, and Sen 2016Not checked yet

  Not checked in the free run.

Dickson, Gordon, and Huber (2015) exclude cases of 'perverse' targeting of a contributor when at least one citizen did not contribute.

“First, DGH exclude cases of so-called “perverse” targeting of a contributor when at least one citizen did not contribute (2015,119).”

• cites Dickson, Gordon, and Huber (2015)Not checked yet

  Not checked in the free run.

There is still significant confusion in the wider discipline about the sources and consequences of posttreatment bias.

“there is still significant confusion in the wider discipline about its sources and consequences”

All subsample estimates in the ideological distance analysis are biased.

“However, all of these subsample estimates are also biased.”

Perverse targeting is a posttreatment behavior given the expected effect of the manipulations.

“However, perverse targeting is a posttreatment behavior given the expected effect of the manipulations.”

Estimating a two-stage least squares model using random assignment as an instrument for treatment status estimates a different estimand known as the complier average causal effect (CACE).

“Another approach to noncompliance is to estimate a two-stage least squares model using random assignment as an instrument for treatment status.Here again, however, we are estimating a different estimand known as the complier average causal effect (CACE).”

Dickson, Gordon, and Huber (2015) control for lagged average contributions, average resoluteness, and perverse or predatory targeting to try to ensure that the effects of the treatments at time t are not fully mediated by behavior and outcomes in previous periods.

“Second, DGH controls for lagged average contributions, average resoluteness, and perverse or predatory targeting to try to ensure that the effects of the treatments at time t are not fully mediated by behavior and outcomes in previous periods (2015,122).”

• cites Dickson, Gordon, and Huber (2015)Not checked yet

  Not checked in the free run.

Calculating the difference in outcomes between respondents assigned to receive treatment and those assigned to receive control is an unbiased estimate of the intention to treat (ITT) effect.

“The simplest is to calculate the difference in outcomes between respondents assigned to receive treatment and those assigned to receive the control, which is an unbiased estimate of the intention to treat (ITT) effect.”

Randomization eliminates omitted variable bias in expectation.

“Though it is not necessary to do so (randomization eliminates omitted variable bias in expectation), regression adjustment for covariates has been shown to induce only minor bias and to potentially increase efficiency under realistic conditions (e.g., Lin 2013).”

Many treatments could differentially affect the types of participants who pass attention checks via other mechanisms (e.g., changing respondent engagement or affecting the contents of working memory), thereby imbalancing the sample.

“Researchers may neglect this issue when the content of the attention check is not directly related to the experimental randomization, but many treatments could differentially affect the types of participants who pass these measures via other mechanisms (e.g., changing respondent engagement or affecting the contents of working memory), thereby imbalancing the sample.”

The analysis suggests that these results may be attributable to posttreatment bias.

“Dickson, Gordon, and Huber (2015) note that both models are sensitive to model specification; our analysis suggests that these results may be attributable to posttreatment bias.”

Simply dropping cases or controlling for compliance status in a regression model can lead to biased estimates of the ATE.

“For the reasons stated above, simply dropping cases or controlling for compliance status in a regression model can lead to biased estimates of the ATE.”

Collecting outcome data among some subjects with missing outcomes combines double sampling with extreme value bounds.

“An alternative approach is to collect outcome data among some subjects with missing outcomes (Coppock et al. 2017), which combines double sampling with extreme value bounds.”

• cites Coppock et al. 2017Not checked yet

  Not checked in the free run.

Many mediation methods like Imai, Keele, and Tingley (2010) or related alternatives such as marginal structural models (Robins, Hernan, and Brumback 2000) or structural nested mean models (Robins 1999) are founded on the exact same assumption.

“Many mediation methods like Imai, Keele, and Tingley (2010) or related alternatives such as marginal structural models (Robins, Hernan, and Brumback 2000) or structural nested mean models (Robins 1999) are founded on the exact same assumption.”

• cites Imai, Keele, and Tingley (2010)Not checked yet

  Not checked in the free run.

• cites Robins, Hernan, and Brumback 2000Not checked yet

  Not checked in the free run.

• cites Robins 1999Not checked yet

  Not checked in the free run.

The most common mediation models all rely in some way on the assumption that researchers have access to every relevant covariate.

“Thus, the most common mediation models all rely in some way on the assumption that researchers have access to every relevant covariate.”

Interpretation of CACE is especially thorny in the presence of two-sided noncompliance where compliance status cannot be directly observed and an additional monotonicity assumption (no defiers) must be invoked.

“Interpretation is especially thorny in the presence of two-sided noncompliance where compliance status cannot be directly observed and an additional monotonicity assumption (no defiers) must be invoked (Angrist, Imbens, and Rubin 1996;see Gerber and Green 2012, 131-209, for more on these points).”

• cites Angrist, Imbens, and Rubin 1996Not checked yet

  Not checked in the free run.

• cites Gerber and Green 2012Not checked yet

  Not checked in the free run.

Unless attrition and nonresponse are unrelated to potential outcomes and treatment, excluding observations from the final analysis is equivalent to conditioning on a posttreatment variable.

“However, unless attrition and nonresponse are unrelated to potential outcomes and treatment, this practice is equivalent to conditioning on a posttreatment variable.”

The approach frequently attributed to Baron and Kenny (1986) does not identify the direct or indirect effects of interest absent additional assumptions including sequential ignorability, which essentially assumes away the possibility of unmeasured confounders.

“However, this approach, which is frequently attributed to Baron and Kenny (1986), does not identify the direct or indirect effects of interest absent additional assumptions including sequential ignorability, which essentially assumes away the possibility of unmeasured confounders.”

• cites Baron and Kenny (1986)Not checked yet

  Not checked in the free run.

Without those constant-parameter assumptions, we cannot be sure a variable is not posttreatment unless we accept the sharp null of no effect for any unit.

“Without them, we cannot be sure a variable is not posttreatment unless we accept the sharp null of no effect for any unit.”

• AssumptionThe authors state this as an assumption: Critical parameters including the treatment effect, the effect of the treatment on the covariate, and the effects of the confounders on outcomes and covariates were constant for each individual.

Failing to reject the null hypothesis is not direct evidence for the null.

“Failing to reject the null hypothesis is not direct evidence for the null.”

Failing to reject the null hypothesis H0: E(x|T=0)=E(x|T=1) does not rule out posttreatment bias in analyses that condition on x.

“However, failing to reject the null hypothesis H 0 : E (x|T = 0) = E (x|T = 1) does not rule out posttreatment bias in analyses that condition on x.”

Any attempt to include posttreatment variables in a mediation analysis comes at an inferential cost.

“Scholars, reviewers, and editors should recognize that any attempt to include posttreatment variables in a mediation analysis comes at an inferential cost.”

Unpacking the 'black box' of experimental treatments must be paid for in the form of assumptions, biased estimates, or both.

“Unpacking the “black box” of experimental treatments must be paid for in the form of assumptions, biased estimates, or both.”

Bullock, Green, and Ha (2010) outline experimental designs that facilitate the study of causal mediation by directly manipulating posttreatment mediators as well as treatment assignments.

“For example, Bullock, Green, and Ha (2010) outline experimental designs that facilitate the study of causal mediation by directly manipulating posttreatment mediators as well as treatment assignments.”

• cites Bullock, Green, and Ha (2010)Not checked yet

  Not checked in the free run.

Aronow, Baron, and Pinson (2015) show that in a more general setting, it will often not be possible to provide bounds that exclude -∞ and ∞ for the potential bias from conditioning on a posttreatment variable.

“Indeed, Aronow, Baron, and Pinson (2015) show that in a more general setting, it will often not be possible to provide bounds that exclude -∞ and ∞ for the potential bias from conditioning on a posttreatment variable.”

• cites Aronow, Baron, and Pinson (2015)Not checked yet

  Not checked in the free run.

Standard approaches such as controlling for intervening variables or subsetting data are incorrect.

“As shown above, standard approaches such as controlling for intervening variables or subsetting data are incorrect.”

The bias that posttreatment conditioning introduces into so much experimental research can easily be avoided.

“Happily, then, the bias that posttreatment conditioning introduces into so much experimental research can easily be avoided.”

Asking questions about certain highly salient covariates like group identification before an outcome variable can affect subsequent responses.

“In particular, asking questions about certain highly salient covariates like group identification before an outcome variable can affect subsequent responses (e.g., Kosloff et al. 2010;Leach et al. 2010).”

• cites Kosloff et al. 2010Not checked yet

  Not checked in the free run.

• cites Leach et al. 2010Not checked yet

  Not checked in the free run.

Once an experiment has been conducted, it is not possible to rule out alternative mechanisms without the possibility of posttreatment bias.

“However, once an experiment has been conducted, it is not possible to rule out alternative mechanisms without the possibility of posttreatment bias.”