The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index; none of the 26 sources its claims cite is checked for support yet.
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Open accessAbstract
In principle, experiments offer a straightforward method for social scientists to accurately estimate causal effects. However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation. Typical examples include controlling for posttreatment variables in statistical models, eliminating observations based on posttreatment criteria, or subsetting the data based on posttreatment variables. Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects. Moreover, problems associated with conditioning on posttreatment variables remain largely unrecognized in the field, which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals. We demonstrate the severity of experimental posttreatment bias analytically and document the magnitude of the potential distortions it induces using visualizations and reanalyses of real‐world data. We conclude by providing applied researchers with recommendations for best practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
52 Own interpretation · 30 Own result · 18 Not checked yet · 12 No basis given · 1 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“almost 1 in 10 engaged in both (10.7%, eight studies)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“14.7% of studies subsetted the data based on potential posttreatment criteria (11 of 75 studies reviewed)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“problems associated with conditioning on posttreatment variables remain largely unrecognized in the field”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“21.3%(16 of 75) controlled for a posttreatment covariate in a statistical model”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the studies that controlled for a posttreatment variable, six used a mediation technique (8%).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most strikingly, 12% of studies conditioned on a variable shown to be affected by the experimental treatment in analyses contained within the article itself (9 of 75).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“few acknowledge potential concerns regarding the bias that posttreatment conditioning can introduce”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the magnitude of the negative coefficient approximately doubles relative to the unconditional estimate among respondents who perceive themselves as further from Eid (-.460; 95% CI: -0.567, -0.353).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Trump's endorsement reduced the likelihood of supporting Eid's retention by -0.214 (p < .01,95% CI: -0.301, -0.127) on the four-point scale.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 15
- It may state two separate claims as one. 22
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while some studies lost cases due to posttreatment attrition (8.0%)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“About 1 in 4 drop cases or subset the data based on posttreatment criteria, and nearly a third include posttreatment variables as covariates.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These effects are opposite in sign and in both cases highly significant (p < .01 in both directions).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The sign of the estimated treatment effect reverses in the subsample of respondents who perceive themselves as being close to Eid, becoming positive (.257, 95% CI: 0.132, 0.382)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most notably, the appropriations treatment effect estimate in the limited information condition more than doubles in magnitude and becomes nearly statistically significant in the fourth column (p < .11).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By contrast, the effect of appropriations among groups with limited information and the effect of limited information in either compensation group are not distinguishable from zero.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These practices are typically employed in an effort to address practical problems like noncompliance or to try to answer difficult inferential questions such as identifying causal mechanisms.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Reviewers often ask authors to try to rule out alternative explanations in this way.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Insufficient information to code | 1.3% |”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plainn = 75- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assumption (1) implies that individuals in the treatment and control conditions will be similar in expectation with respect to unobserved confounders such as SES.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As long as we do not condition on a posttreatment variable, randomization guarantees that Assumption (1) is satisfied and both quantities go to zero.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, Assumption (1) requires that x is not causally related to T”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although it is possible to construct examples where this bias is zero, it will not be zero in general.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Dropping or Selecting Observations Based on Criteria Influenced by the Treatment.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“dropping cases or subsetting based on posttreatment criteria will unbalance the treatment and control conditions with respect to unmeasured confounders and bias our treatment effect estimates.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Dropping or Selecting Observations Based on Criteria Influenced by the Treatment.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“the dangers of posttreatment bias have long been recognized in the fields of statistics, econometrics, and political methodology”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
What the sources say
- Not checked yetcites Acharya, Blackwell, and Sen 2016
- Not checked yetcites Elwert and Winship 2014
- Not checked yetcites King and Zeng 2006
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Instead, it is a biased estimate of the direct effect of the Trump endorsement, and the bias can be in any direction at all.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, control group participants who think Eid is centrist on average and perceive themselves to be relatively close to her are being compared to treatment group participants who think she is close to them after finding out she was endorsed by President Trump.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“More recent work has amplified these points in the context of observational research”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
What the sources say
- Not checked yetcites Acharya, Blackwell, and Sen 2016
- Not checked yetcites Blackwell 2013
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By conditioning on a posttreatment variable, we have unbalanced the treatment and control groups in terms of ideology and unmeasured confounders and thus biased the treatment effect estimate.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this scenario, dropping respondents based on posttreatment attention checks is the equivalent of selecting on a posttreatment covariate and would again risk bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Pretreatment Moderators, Control Variables, and Attention Checks
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, both approaches provide biased treatment effect estimates that do not correspond to meaningful causal estimands.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Though it is not necessary to do so (randomization eliminates omitted variable bias in expectation), regression adjustment for covariates has been shown to induce only minor bias and to potentially increase efficiency under realistic conditions (e.g., Lin 2013).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Pretreatment Moderators, Control Variables, and Attention Checks
What the sources say
- Not checked yetcites Lin 2013
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Dickson, Gordon, and Huber (2015) note that both models are sensitive to model specification; our analysis suggests that these results may be attributable to posttreatment bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
What the sources say
- Not checked yetcites Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We replicate the appropriations treatment effect for full information groups, but our analysis raises concerns about posttreatment bias for both the limited information effect in the salary condition and the appropriation effect in the limited information condition.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Under more realistic assumptions, however, the choices are more limited: Gerber and Green (2012) recommend extreme value bounds (Manski 1989), where analysts estimate the largest and smallest ATEs possible if missing information were filled in with extreme outcomes.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
What the sources say
- Not checked yetcites Gerber and Green (2012)
- Not checked yetcites Manski 1989
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“If we are willing to assume that missingness is not a function of unmeasured confounders, we can use familiar methods such as imputation or marginal structural models.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Missingness is not a function of unmeasured confounders.
In the paper
“Finally, Huber (2012) seeks to reduce bias from attrition using inverse probability weighting and instrumental variables for missingness.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
What the sources say
- Not checked yetcites Huber (2012)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is no reason to believe that these assumptions are correct in real-world data.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Critical parameters including the treatment effect, the effect of the treatment on the covariate, and the effects of the confounders on outcomes and covariates were constant for each individual.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach is not only very difficult to execute (it requires a treatment that affects the mediator but not the outcome) but also subject to criticism for implausible assumptions.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, even in the simplified examples presented above, posttreatment bias will not be eliminated unless the effect of the treatment on the covariate (␥ ) is precisely zero-something that cannot be established using traditional hypothesis testing.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though these intentions are laudable, we demonstrate that posttreatment conditioning undermines the value of randomization and biases treatment effect estimates using analytical results as well as a reanalysis of real-world data from two studies.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, it is important to note that posttreatment bias cannot be easily diagnosed or remedied empirically.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Imai, Tingley, and Yamamoto (2013) outline several designs that allow researchers to estimate mediation effects, but these too come with additional assumptions (e.g., a consistency assumption) or require use of less intuitive estimands (e.g., average complier indirect effects).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
What the sources say
- Not checked yetcites Imai, Tingley, and Yamamoto (2013)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we also show that scholars can address the concerns that motivate the use of these practices using existing analytical approaches.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many of the practices described above appear to be driven by authors' efforts to show that their proposed mechanism is responsible for the treatment effect.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, mediation analyses require strong assumptions that may be inconsistent with the goals of experimental research.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The authors state this as an assumption.
Treatment assignment is independent of potential outcomes conditional on covariates.
“Assumption (1) : (y [T =1] , y [T =0] ) ⊥ ⊥ T |X, which states that treatment assignment is independent of potential outcomes conditional on covariates.”
The authors state this as an assumption.
All variables are observed without error.
“For the sake of expositional clarity, and without loss of generality, we assume that all variables are observed without error.”
The authors state this as an assumption.
The main parameters in Model 4 do not vary as a function of T or x.
“Note that Model 4 also assumes that the main parameters in the model ( , ␣ Y , , , and Y ) do not vary as a function of T or x, which is why we can move these parameters outside of the expectations.”
The authors state this as an assumption.
The influence of unmeasured confounders along the various causal paths will not somehow cancel out.
“For instance, we assume that the influence of unmeasured confounders along the various causal paths will not somehow cancel out.”
The authors state this as an assumption.
x has no direct effect on y in the true model.
“If we instead allow x to have a direct effect on y in the true model, the biases we describe below still hold, but the calculations involved are more complex.We make this simplifying assumption so that we can focus our exposition on the posttreatment bias that arises from unblocking the path from u to y.”
The authors state this as an assumption.
β = 0.
“To simplify exposition, we focus here only on the bias resulting from the imbalance in u induced by controlling for the posttreatment variable x by assuming that  = 0.”
The authors state this as an assumption.
Critical parameters including the treatment effect, the effect of the treatment on the covariate, and the effects of the confounders on outcomes and covariates were constant for each individual.
“Second, we made the simplifying assumption in our examples above that critical parameters including the treatment effect ( ), the effect of the treatment on the covariate (␥ ), and the effects of the confounders on outcomes and covariates ( Y , X ) were constant for each individual.”
The authors state this as an assumption.
Missingness is not a function of unmeasured confounders.
“If we are willing to assume that missingness is not a function of unmeasured confounders, we can use familiar methods such as imputation or marginal structural models.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“An important caveat for these analyses is that attention was measured post-treatment, which has the potential to undermine inferences drawn from these data43”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Additional information
“• Reanalysis of Broockman and Butler (2015)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
“an additional demonstration of posttreatment bias using data fromBroockman and Butler (2017).”
Citations
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Marginal Structural Models and Causal Inference in Epidemiology
Robins, J. M., Hernán, M. Á., & Brumback, B. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00011 · Cited by 5,581
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
A general approach to causal mediation analysis.
Imai, K., Keele, L., & Tingley, D. (2010) · Psychological Methods · doi:10.1037/a0020761 · Cited by 4,143
Identification and estimation of local average treatment effects
Imbens, G. W., & Angrist, J. D. (1994) · Econometrica · doi:10.2307/2951620 · Cited by 4,101
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Identification and Estimation of Local Average Treatment Effects
Angrist, J. D., & Imbens, G. W. (1995) · National Bureau of Economic Research · doi:10.3386/t0118 · Cited by 2,915
Field Experiments: Design, Analysis, and Interpretation
GREEN, D. P. (2024) · doi:10.61700/sgeirzx0v0mjh1252 · Cited by 1,322
Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable
Elwert, F., & Winship, C. (2014) · Annual Review of Sociology · doi:10.1146/annurev-soc-071913-043455 · Cited by 1,168 · cited 2×
“…T and u, which unbalances the experiment with respect to u and makes accurately estimating the causal e↵ect impossible without further assumptions (Elwert and Winship 2014).11 In our example, conditioning on political interest unbalances the treatment and control groups on SES, which in turn…”
BackgroundYes, but what’s the mechanism? (don’t expect an easy answer).
Bullock, J. G., Green, D. P., & Ha, S. E. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0018933 · Cited by 1,063
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951
The Consquences of Adjustment for a Concomitant Variable That Has Been Affected by the Treatment
Rosenbaum, P. R. (1984) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981697 · Cited by 849
Cues that Matter: How Political Ads Prime Racial Attitudes During Campaigns
Valentino, N. A., Hutchings, V. L., & White, I. K. (2002) · American Political Science Review · doi:10.1017/s0003055402004240 · Cited by 770
Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique
Lin, W. T. (2013) · The Annals of Applied Statistics · doi:10.1214/12-aoas583 · Cited by 686
The Econometrics of Randomized Experiments
Athey, S. C., & Imbens, G. W. (2017) · Handbook of economic field experiments · doi:10.1016/bs.hefe.2016.10.003 · Cited by 561
The Dangers of Extreme Counterfactuals
King, G., & Zeng, L. (2005) · Political Analysis · doi:10.1093/pan/mpj004 · Cited by 549
Explaining Causal Findings Without Bias: Detecting and Assessing Direct Effects
Acharya, A. R., Blackwell, M., & Sen, M. (2016) · American Political Science Review · doi:10.1017/s0003055416000216 · Cited by 542
Anatomy of the Selection Problem
Manski, C. F. (1989) · The Journal of Human Resources · doi:10.2307/145818 · Cited by 539
Zhou, H., & Ayelet Fishbach, B. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000056 · Cited by 462
Experimental Designs for Identifying Causal Mechanisms
Imai, K., Tingley, D., & Yamamoto, T. (2013) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2012.01032.x · Cited by 408
Mastering 'metrics: the path from cause to effect
(2015) · Choice Reviews Online · doi:10.5860/choice.189854 · Cited by 389
Substituting the End for the Whole: Why Voters Respond Primarily to the Election‐Year Economy
Healy, A., & Lenz, G. (2013) · American Journal of Political Science · doi:10.1111/ajps.12053 · Cited by 357
Broockman, D., & Butler, D. M. (2015) · American Journal of Political Science · doi:10.1111/ajps.12243 · Cited by 298
A Note on Dropping Experimental Subjects who Fail a Manipulation Check
Aronow, P. M., Baron, J., & Pinson, L. E. (2019) · Political Analysis · doi:10.1017/pan.2019.5 · Cited by 280
Cognitive Biases and the Strength of Political Arguments
Arceneaux, K. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2011.00573.x · Cited by 256
A Source of Bias in Public Opinion Stability
Druckman, J. N., Fein, J., & Leeper, T. J. (2012) · American Political Science Review · doi:10.1017/s0003055412000123 · Cited by 249
The “Race Card” Revisited: Assessing Racial Priming in Policy Contests
Huber, G. A., & Lapinski, J. S. (2006) · American Journal of Political Science · doi:10.1111/j.1540-5907.2006.00192.x · Cited by 184
Group Devaluation and Group Identification
Leach, C. W., Mosquera, P. M. R., Vliek, M., & Hirt, E. (2010) · Journal of Social Issues · doi:10.1111/j.1540-4560.2010.01661.x · Cited by 129
Malesky, E. J., Schuler, P., & Tran, A. (2011) · American Political Science Review · doi:10.1017/s0003055412000408 · Cited by 113
A Framework for Dynamic Causal Inference in Political Science
Blackwell, M. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2012.00626.x · Cited by 112
Computation, causation, and discovery
(1999) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(99)91187-2 · Cited by 109
Source Cues and Public Support for the Supreme Court
Clark, T. S., & Kastellec, J. (2015) · American Politics Research · doi:10.1177/1532673x14560809 · Cited by 108
Designing and Analyzing Randomized Experiments: Application to a Japanese Election Survey Experiment
Horiuchi, Y., Imai, K., & Taniguchi, N. (2007) · American Journal of Political Science · doi:10.1111/j.1540-5907.2007.00274.x · Cited by 101
VIOLATING IGNORABILITY OF TREATMENT BY CONTROLLING FOR TOO MANY FACTORS
Wooldridge, J. M. (2005) · Econometric Theory · doi:10.1017/s0266466605050516 · Cited by 101
Elections and Deceptions: An Experimental Study on the Behavioral Effects of Democracy
Corazzini, L., Kube, S., Maréchal, M. A., & Nicolò, A. (2013) · American Journal of Political Science · doi:10.1111/ajps.12078 · Cited by 90
A Note on Dropping Experimental Subjects Who Fail a Manipulation CheckPreprint
Aronow, P. M., Baron, J., & Pinson, L. E. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2683588 · Cited by 60
Treatment Spillover Effects across Survey Experiments
Transue, J. E., Lee, D. J., & Aldrich, J. H. (2009) · Political Analysis · doi:10.1093/pan/mpn012 · Cited by 58
Incentives to Identify: Racial Identity in the Age of Affirmative Action
Antman, F., & Duncan, B. (2015) · The Review of Economics and Statistics · doi:10.1162/rest_a_00527 · Cited by 48
Kosloff, S., Greenberg, J., Schmader, T., Dechesne, M., & Weise, D. (2010) · Journal of Experimental Psychology General · doi:10.1037/a0018809 · Cited by 47
Institutional Sources of Legitimate Authority: An Experimental Investigation
Dickson, E. S., Gordon, S. C., & Huber, G. A. (2014) · American Journal of Political Science · doi:10.1111/ajps.12139 · Cited by 40
Huber, M. (2011) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998611411917 · Cited by 34
Design, Identification, and Sensitivity Analysis for Patient Preference Trials
Knox, D., Yamamoto, T., Baum, M., & Berinsky, A. J. (2019) · Journal of the American Statistical Association · doi:10.1080/01621459.2019.1585248 · Cited by 34
Political Quid Pro Quo Agreements: An Experimental Study
Großer, J., Reuben, E., & Tymula, A. A. (2013) · American Journal of Political Science · doi:10.1111/ajps.12015 · Cited by 24
Coppock, A., Gerber, A. S., GREEN, D. P., et al. (2017) · Political Analysis · doi:10.1017/pan.2016.6 · Cited by 11
Party Labels and Vote Choice in Judicial Elections
Burnett, C. M., & Tiede, L. B. (2014) · American Politics Research · doi:10.1177/1532673x14548477 · Cited by 10
A Natural Experiment: Inadvertent Priming of Party Identification in a Split-Sample Survey
Weiner, M. D. (2015) · Survey Practice · doi:10.29115/sp-2015-0029 · Cited by 7
Cited by
Sorted by citations
Shifting attention to accuracy can reduce misinformation online
Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021) · Nature · doi:10.1038/s41586-021-03344-2 · Cited by 1,155
Vaccari, C., & Chadwick, A. (2020) · Social Media + Society · doi:10.1177/2056305120903408 · Cited by 978
“As possible solutions, they cite designs that include “a treatment that affects the mediator but not the outcome” (Montgomery et al., 2018, p. 772).”
BackgroundInfluentialThe shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
Nyhan, B. J., Porter, E. V., Reifler, J., & Wood, T. (2017) · Political Behavior · doi:10.1007/s11109-019-09528-x · Cited by 557
Self-citationThe Ties That Double Bind: Social Roles and Women's Underrepresentation in Politics
Teele, D. L., Kalla, J., & Rosenbluth, F. M. (2018) · American Political Science Review · doi:10.1017/s0003055418000217 · Cited by 408
Metadata
- Authors
- Montgomery, J. M., Nyhan, B., & Torres, M.
- Affiliations
- Washington University in St. Louis (US)
- Dartmouth College (US)
- Venue
- American Journal of Political Science
- Publisher
- Wiley
- Published
- 2018-03-30
- Volume
- 62
- Issue
- 3
- Pages
- 760-775
- ISSN
- 0092-5853 (ISSN-L)
- 1540-5907
- OpenAlex
- W2794592087
- Cited by
- 800
- References
- 49
- Citations by year
- 2026: 68 · 2025: 128 · 2024: 120 · 2023: 114 · 2022: 105 · 2021: 89 · 2020: 95 · 2019: 48 · 2018: 27
- FWCI
- 53.73
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Political Science (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- causal effect estimation, experimental design, confounding, bias in treatment effect estimates
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
