The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index; none of the 26 sources its claims cite is checked for support yet.
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Open accessAbstract
In principle, experiments offer a straightforward method for social scientists to accurately estimate causal effects. However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation. Typical examples include controlling for posttreatment variables in statistical models, eliminating observations based on posttreatment criteria, or subsetting the data based on posttreatment variables. Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects. Moreover, problems associated with conditioning on posttreatment variables remain largely unrecognized in the field, which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals. We demonstrate the severity of experimental posttreatment bias analytically and document the magnitude of the potential distortions it induces using visualizations and reanalyses of real‐world data. We conclude by providing applied researchers with recommendations for best practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
52 Own interpretation · 30 Own result · 18 Not checked yet · 12 No basis given · 1 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“almost 1 in 10 engaged in both (10.7%, eight studies)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“14.7% of studies subsetted the data based on potential posttreatment criteria (11 of 75 studies reviewed)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“problems associated with conditioning on posttreatment variables remain largely unrecognized in the field”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“21.3%(16 of 75) controlled for a posttreatment covariate in a statistical model”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the studies that controlled for a posttreatment variable, six used a mediation technique (8%).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most strikingly, 12% of studies conditioned on a variable shown to be affected by the experimental treatment in analyses contained within the article itself (9 of 75).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“few acknowledge potential concerns regarding the bias that posttreatment conditioning can introduce”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the magnitude of the negative coefficient approximately doubles relative to the unconditional estimate among respondents who perceive themselves as further from Eid (-.460; 95% CI: -0.567, -0.353).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Trump's endorsement reduced the likelihood of supporting Eid's retention by -0.214 (p < .01,95% CI: -0.301, -0.127) on the four-point scale.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 15
- It may state two separate claims as one. 22
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while some studies lost cases due to posttreatment attrition (8.0%)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“About 1 in 4 drop cases or subset the data based on posttreatment criteria, and nearly a third include posttreatment variables as covariates.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These effects are opposite in sign and in both cases highly significant (p < .01 in both directions).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The sign of the estimated treatment effect reverses in the subsample of respondents who perceive themselves as being close to Eid, becoming positive (.257, 95% CI: 0.132, 0.382)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most notably, the appropriations treatment effect estimate in the limited information condition more than doubles in magnitude and becomes nearly statistically significant in the fourth column (p < .11).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By contrast, the effect of appropriations among groups with limited information and the effect of limited information in either compensation group are not distinguishable from zero.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These practices are typically employed in an effort to address practical problems like noncompliance or to try to answer difficult inferential questions such as identifying causal mechanisms.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Reviewers often ask authors to try to rule out alternative explanations in this way.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Insufficient information to code | 1.3% |”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plainn = 75- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assumption (1) implies that individuals in the treatment and control conditions will be similar in expectation with respect to unobserved confounders such as SES.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As long as we do not condition on a posttreatment variable, randomization guarantees that Assumption (1) is satisfied and both quantities go to zero.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, Assumption (1) requires that x is not causally related to T”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although it is possible to construct examples where this bias is zero, it will not be zero in general.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Dropping or Selecting Observations Based on Criteria Influenced by the Treatment.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“dropping cases or subsetting based on posttreatment criteria will unbalance the treatment and control conditions with respect to unmeasured confounders and bias our treatment effect estimates.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Dropping or Selecting Observations Based on Criteria Influenced by the Treatment.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“the dangers of posttreatment bias have long been recognized in the fields of statistics, econometrics, and political methodology”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
What the sources say
- Not checked yetcites Acharya, Blackwell, and Sen 2016
- Not checked yetcites Elwert and Winship 2014
- Not checked yetcites King and Zeng 2006
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Instead, it is a biased estimate of the direct effect of the Trump endorsement, and the bias can be in any direction at all.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, control group participants who think Eid is centrist on average and perceive themselves to be relatively close to her are being compared to treatment group participants who think she is close to them after finding out she was endorsed by President Trump.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“More recent work has amplified these points in the context of observational research”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
What the sources say
- Not checked yetcites Acharya, Blackwell, and Sen 2016
- Not checked yetcites Blackwell 2013
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By conditioning on a posttreatment variable, we have unbalanced the treatment and control groups in terms of ideology and unmeasured confounders and thus biased the treatment effect estimate.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this scenario, dropping respondents based on posttreatment attention checks is the equivalent of selecting on a posttreatment covariate and would again risk bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Pretreatment Moderators, Control Variables, and Attention Checks
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, both approaches provide biased treatment effect estimates that do not correspond to meaningful causal estimands.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Though it is not necessary to do so (randomization eliminates omitted variable bias in expectation), regression adjustment for covariates has been shown to induce only minor bias and to potentially increase efficiency under realistic conditions (e.g., Lin 2013).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Pretreatment Moderators, Control Variables, and Attention Checks
What the sources say
- Not checked yetcites Lin 2013
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Dickson, Gordon, and Huber (2015) note that both models are sensitive to model specification; our analysis suggests that these results may be attributable to posttreatment bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
What the sources say
- Not checked yetcites Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We replicate the appropriations treatment effect for full information groups, but our analysis raises concerns about posttreatment bias for both the limited information effect in the salary condition and the appropriation effect in the limited information condition.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Under more realistic assumptions, however, the choices are more limited: Gerber and Green (2012) recommend extreme value bounds (Manski 1989), where analysts estimate the largest and smallest ATEs possible if missing information were filled in with extreme outcomes.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
What the sources say
- Not checked yetcites Gerber and Green (2012)
- Not checked yetcites Manski 1989
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“If we are willing to assume that missingness is not a function of unmeasured confounders, we can use familiar methods such as imputation or marginal structural models.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Missingness is not a function of unmeasured confounders.
In the paper
“Finally, Huber (2012) seeks to reduce bias from attrition using inverse probability weighting and instrumental variables for missingness.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
What the sources say
- Not checked yetcites Huber (2012)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is no reason to believe that these assumptions are correct in real-world data.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Critical parameters including the treatment effect, the effect of the treatment on the covariate, and the effects of the confounders on outcomes and covariates were constant for each individual.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach is not only very difficult to execute (it requires a treatment that affects the mediator but not the outcome) but also subject to criticism for implausible assumptions.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, even in the simplified examples presented above, posttreatment bias will not be eliminated unless the effect of the treatment on the covariate (␥ ) is precisely zero-something that cannot be established using traditional hypothesis testing.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though these intentions are laudable, we demonstrate that posttreatment conditioning undermines the value of randomization and biases treatment effect estimates using analytical results as well as a reanalysis of real-world data from two studies.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, it is important to note that posttreatment bias cannot be easily diagnosed or remedied empirically.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Imai, Tingley, and Yamamoto (2013) outline several designs that allow researchers to estimate mediation effects, but these too come with additional assumptions (e.g., a consistency assumption) or require use of less intuitive estimands (e.g., average complier indirect effects).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
What the sources say
- Not checked yetcites Imai, Tingley, and Yamamoto (2013)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we also show that scholars can address the concerns that motivate the use of these practices using existing analytical approaches.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many of the practices described above appear to be driven by authors' efforts to show that their proposed mechanism is responsible for the treatment effect.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, mediation analyses require strong assumptions that may be inconsistent with the goals of experimental research.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The authors state this as an assumption.
Treatment assignment is independent of potential outcomes conditional on covariates.
“Assumption (1) : (y [T =1] , y [T =0] ) ⊥ ⊥ T |X, which states that treatment assignment is independent of potential outcomes conditional on covariates.”
The authors state this as an assumption.
All variables are observed without error.
“For the sake of expositional clarity, and without loss of generality, we assume that all variables are observed without error.”
The authors state this as an assumption.
The main parameters in Model 4 do not vary as a function of T or x.
“Note that Model 4 also assumes that the main parameters in the model ( , ␣ Y , , , and Y ) do not vary as a function of T or x, which is why we can move these parameters outside of the expectations.”
The authors state this as an assumption.
The influence of unmeasured confounders along the various causal paths will not somehow cancel out.
“For instance, we assume that the influence of unmeasured confounders along the various causal paths will not somehow cancel out.”
The authors state this as an assumption.
x has no direct effect on y in the true model.
“If we instead allow x to have a direct effect on y in the true model, the biases we describe below still hold, but the calculations involved are more complex.We make this simplifying assumption so that we can focus our exposition on the posttreatment bias that arises from unblocking the path from u to y.”
The authors state this as an assumption.
β = 0.
“To simplify exposition, we focus here only on the bias resulting from the imbalance in u induced by controlling for the posttreatment variable x by assuming that  = 0.”
The authors state this as an assumption.
Critical parameters including the treatment effect, the effect of the treatment on the covariate, and the effects of the confounders on outcomes and covariates were constant for each individual.
“Second, we made the simplifying assumption in our examples above that critical parameters including the treatment effect ( ), the effect of the treatment on the covariate (␥ ), and the effects of the confounders on outcomes and covariates ( Y , X ) were constant for each individual.”
The authors state this as an assumption.
Missingness is not a function of unmeasured confounders.
“If we are willing to assume that missingness is not a function of unmeasured confounders, we can use familiar methods such as imputation or marginal structural models.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“An important caveat for these analyses is that attention was measured post-treatment, which has the potential to undermine inferences drawn from these data43”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Additional information
“• Reanalysis of Broockman and Butler (2015)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
“an additional demonstration of posttreatment bias using data fromBroockman and Butler (2017).”
Citations
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Marginal Structural Models and Causal Inference in Epidemiology
Robins, J. M., Hernán, M. Á., & Brumback, B. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00011 · Cited by 5,581
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
A general approach to causal mediation analysis.
Imai, K., Keele, L., & Tingley, D. (2010) · Psychological Methods · doi:10.1037/a0020761 · Cited by 4,143
Identification and estimation of local average treatment effects
Imbens, G. W., & Angrist, J. D. (1994) · Econometrica · doi:10.2307/2951620 · Cited by 4,101
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Identification and Estimation of Local Average Treatment Effects
Angrist, J. D., & Imbens, G. W. (1995) · National Bureau of Economic Research · doi:10.3386/t0118 · Cited by 2,915
Field Experiments: Design, Analysis, and Interpretation
GREEN, D. P. (2024) · doi:10.61700/sgeirzx0v0mjh1252 · Cited by 1,322
Cited by
Sorted by citations
Shifting attention to accuracy can reduce misinformation online
Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021) · Nature · doi:10.1038/s41586-021-03344-2 · Cited by 1,155
Vaccari, C., & Chadwick, A. (2020) · Social Media + Society · doi:10.1177/2056305120903408 · Cited by 978
“As possible solutions, they cite designs that include “a treatment that affects the mediator but not the outcome” (Montgomery et al., 2018, p. 772).”
BackgroundInfluentialThe shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
Nyhan, B. J., Porter, E. V., Reifler, J., & Wood, T. (2017) · Political Behavior · doi:10.1007/s11109-019-09528-x · Cited by 557
Self-citationThe Ties That Double Bind: Social Roles and Women's Underrepresentation in Politics
Teele, D. L., Kalla, J., & Rosenbluth, F. M. (2018) · American Political Science Review · doi:10.1017/s0003055418000217 · Cited by 408
On doing relevant and rigorous experiments: Review and recommendations
Lonati, S., Quiroga, B. F., Zehnder, C., & Antonakis, J. (2018) · Journal of Operations Management · doi:10.1016/j.jom.2018.10.003 · Cited by 386
Unexpected Event during Survey Design: Promise and Pitfalls for Causal Inference
Muñoz, J., Falcó-Gimeno, A., & Hernández, E. (2019) · Political Analysis · doi:10.1017/pan.2019.27 · Cited by 334
“In the UESD, all covariates aremeasured posttreatment, so variables that can be potentially a ected by the event should not be included to avoid posttreatment bias (Montgomery, Nyhan, and Torres 2018).”
BackgroundData Collection via Online Platforms: Challenges and Recommendations for Future Research
Newman, A., Bavik, Y. L., Mount, M. P., & Shao, B. (2020) · Applied Psychology · doi:10.1111/apps.12302 · Cited by 360
No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments
Kane, J., & Barabas, J. (2018) · American Journal of Political Science · doi:10.1111/ajps.12396 · Cited by 276
“…use FMCs to conduct analyses on “passers” only, this practice raises concerns regarding posttreatment bias (e.g., Acharya, Blackwell, and Sen 2016; Montgomery, Nyhan, and Torres 2018) and/or unrepresentative samples (Berinsky, Margolis, and Sances 2014, 747–48). a treatment-irrelevant FMC…”
BackgroundAnti-Intellectualism, Populism, and Motivated Resistance to Expert Consensus
Merkley, E. (2019) · Public Opinion Quarterly · doi:10.1093/poq/nfz053 · Cited by 303
Pennycook, G., & Rand, D. G. (2022) · Nature Communications · doi:10.1038/s41467-022-30073-5 · Cited by 270
“An important caveat for these analyses is that attention was measured post-treatment, which has the potential to undermine inferences drawn from these data 43 .”
BackgroundClifford, S., Sheagley, G., & Piston, S. (2021) · American Political Science Review · doi:10.1017/s0003055421000241 · Cited by 262
“…article recommends measuring covariates in a prior survey wave, cautioning researchers to “carefully separate pretreatment questions from their experiment and outcome measures to avoid inadvertently affecting the treatment effects they seek to estimate” (Montgomery, Nyhan, and Torres 2018, 773).”
BackgroundPeterson, E., & Iyengar, S. (2019) · doi:10.31235/osf.io/pmn7h · Cited by 241
“7If incentives a↵ect how certain respondents are of their responses, this specification risks inducing post-treatment bias in our estimates (see e.g., Montgomery et al., 2018).”
ResultElite rhetoric can undermine democratic norms
Clayton, K., Davis, N. T., Nyhan, B., Porter, E., Ryan, T. J., & Wood, T. J. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2024125118 · Cited by 226
“As a result, we greatly reduce the risk of posttreatment bias due to differential attrition between conditions, which otherwise plagues survey experiments of this type (38).”
BackgroundSelf-citationThe Generalizability of Online Experiments Conducted During the COVID-19 Pandemic
Peyton, K., Huber, G. A., & Coppock, A. (2021) · Journal of Experimental Political Science · doi:10.1017/xps.2021.17 · Cited by 174
‘If a fight starts, watch the crowd’: The effect of violence on popular support for social movements
Muñoz, J., & Anduiza, E. (2019) · Journal of Peace Research · doi:10.1177/0022343318820575 · Cited by 154
“However, we do not have a pure pre-treatment measure of support for those in the treatment group, and conditioning on posttreatment variables can induce bias on the estimates (Montgomery, Nyhan & Torres, 2016).”
BackgroundGlobalization and Electoral Outcomes: Evidence from Italy
Caselli, M., Fracasso, A., & Traverso, S. (2018) · Economics and Politics · doi:10.1111/ecpo.12147 · Cited by 150
“However, the inclusion of potentially post-treatment controls may not only block some causal paths, but it may also introduce bias if the variables are themselves endogenous (Montgomery et al., 2018).”
Krupnikov, Y., & Ryan, J. B. (2022) · doi:10.1017/9781108923323 · Cited by 175
Miller, L. B., Rice, R. E., Gustafson, A., & Goldberg, M. H. (2022) · Environment and Behavior · doi:10.1177/00139165221131002 · Cited by 158
“However, the generally weak support for this role, the cross-sectional nature of our data, and limitations of mediation analyses (Fiedler et al., 2011; Fiedler et al., 2018; Green et al., 2010; Montgomery et al., 2018) make it difficult to clearly test that relationship.”
BackgroundPartisanship, Political Knowledge, and the Dunning‐Kruger Effect
Anson, I. G. (2018) · Political Psychology · doi:10.1111/pops.12490 · Cited by 155
Legacies of the Third Reich: Concentration Camps and Out-group Intolerance
Homola, J., Pereira, M. M., & Tavits, M. (2020) · American Political Science Review · doi:10.1017/s0003055419000832 · Cited by 123
“45 Werecognize that the currentuseof the camps canbe interpretedas a posttreatment variable, which may lead to biased estimates (Montgomery, Nyhan, and Torres 2018).”
BackgroundPolitics of Nostalgia and Populism: Evidence from Turkey
Elçi, E. (2021) · British Journal of Political Science · doi:10.1017/s0007123420000666 · Cited by 140
Common methodological mistakes
Wulff, J. N., Sajons, G. B., Pogrebna, G., Lonati, S., Bastardoz, N., Banks, G. C., & Antonakis, J. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2023.101677 · Cited by 136
Online searches to evaluate misinformation can increase its perceived veracity
Aslett, K., Sanderson, Z., Godel, W., Persily, N., Nagler, J., & Tucker, J. A. (2023) · Nature · doi:10.1038/s41586-023-06883-y · Cited by 151
“To be clear, the information returned by the Google search results is post-treatment, so this analysis does not infer a causal relationship 32 , but it provides evidence consistent with the theory that low-quality information returned by search engines could explain the search effect that we…”
ResultWetts, R., & Willer, R. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119866268 · Cited by 111
Onyeador, I. N., Daumeyer, N. M., Rucker, J. M., Duker, A., Kraus, M. W., & Richeson, J. A. (2020) · Personality and Social Psychology Bulletin · doi:10.1177/0146167220942625 · Cited by 102
“Consistent with best practices (Montgomery et al., 2016), we report results using the full sample of White participants in the main text; however, the pattern of results holds if the analyses are conducted only using data from participants who passed the manipulation check.”
ResultApplicants’ Fairness Perceptions of Algorithm-Driven Hiring Procedures
Lavanchy, M., Reichert, P., Narayanan, J., & Savani, K. (2023) · Journal of Business Ethics · doi:10.1007/s10551-022-05320-w · Cited by 145
“These questions were asked before condition randomization to avoid any potential post-treatment bias (Montgomery et al., 2018).”
BackgroundQuillian, L., Lee, J. J., & Oliver, M. (2020) · Social Forces · doi:10.1093/sf/soaa026 · Cited by 105
“Conditioning on callback in calculating job offers conditions on a posttreatment variable, which can create bias in analyses of experiments (Coppock 2019; Montgomery et al. 2018).”
BackgroundText and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates
Keith, K. A., Jensen, D. D., & O’Connor, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.474 · Cited by 127
“By contrast, conditioning on a collider can create dependence between treatment-collider-outcome5 (Pearl, 2009a) potentially introducing more bias (Montgomery et al., 2018; Elwert and Winship, 2014).”
BackgroundThe Shape of and Solutions to the MTurk Quality CrisisPreprint
Kennedy, R., Clifford, S., Burleigh, T. J., Jewell, R., & Waggoner, P. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3272468 · Cited by 99
Marshall, J. (2022) · American Journal of Political Science · doi:10.1111/ajps.12741 · Cited by 103
“These assumptions are analogous to the conditions under which the bias associated with conditioning the sample on a posttreatment variable disappears (see Hernán and Robins 2011, Montgomery, Nyhan, and Torres 2018).”
Grice, J. W., Medellin, E., Jones, I. T., Horvath, S., McDaniel, H., O'Lansen, C., & Baker, M. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920922982 · Cited by 97
Pfaff, S. J., Crabtree, C. D., Kern, H., & Holbein, J. B. (2020) · Public Administration Review · doi:10.1111/puar.13235 · Cited by 107
Avoiding Post-Treatment Bias in Audit Experiments
Coppock, A. (2018) · Journal of Experimental Political Science · doi:10.1017/xps.2018.9 · Cited by 92
“Response is a consequence of treatment; answering these important questions well is complicated by post-treatment bias (Montgomery et al., 2018).”
BackgroundMedical Marijuana Users are More Likely to Use Prescription Drugs Medically and Nonmedically
Caputi, T. L., & Humphreys, K. N. (2018) · Journal of Addiction Medicine · doi:10.1097/adm.0000000000000405 · Cited by 89
“These covariates are not included in our main analysis because they introduce possible post-treatment bias to the model, an increasingly well-recognized issue in the literature (eg, Montgomery et al., 2016).”
MethodPrecise Answers to Vague Questions: Issues With Interactions
Rohrer, J. M., & Arslan, R. C. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211007368 · Cited by 90
“Interactions may be causal or not, but the same applies to all effect estimates from observational studies and sometimes even to effect estimates from experiments (e.g., Montgomery et al., 2018).”
BackgroundThere is no liberal media bias in which news stories political journalists choose to cover
Hassell, H. J. G., Holbein, J. B., & Miles, M. R. (2020) · Science Advances · doi:10.1126/sciadv.aay9344 · Cited by 94
“We avoid measures of response quality, as these are only observed among those who respond, and hence, they are especially susceptible to posttreatment bias (29).”
Chen, C., Haduong, P., Brennan, K. A., Sonnert, G., & Sadler, P. M. (2018) · Computer Science Education · doi:10.1080/08993408.2018.1547564 · Cited by 87
“…K – 12 introductory programming experiences; accounting for school and any post-treatment variables would bias our inferences about the relationship between introductory experiences and student achievement (see thorough explanation in Montgomery, Nyhan, & Torres, 2018, and in; Rosenbaum, 1984).”
BackgroundMethodKlar, S., Leeper, T. J., & Robison, J. (2019) · Journal of Experimental Political Science · doi:10.1017/xps.2019.26 · Cited by 81
Anderson, S. E., Butler, D. M., & Harbridge‐Yong, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108768375 · Cited by 78
Smith, B. B., & Waldner, D. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108776837 · Cited by 75
Individualizing gamified systems: The role of trait competitiveness and leaderboard design
Höllig, C. E., Tumasjan, A., & Welpe, I. M. (2018) · Journal of Business Research · doi:10.1016/j.jbusres.2018.10.046 · Cited by 81
Dimitriadis, S., & Koning, R. M. (2022) · Management Science · doi:10.1287/mnsc.2022.4334 · Cited by 73
Repetition Increases Perceived Truth Even for Known Falsehoods
Fazio, L. K. (2020) · Collabra Psychology · doi:10.1525/collabra.347 · Cited by 87
Migchelbrink, K., & de Walle, S. V. (2019) · Public Administration Review · doi:10.1111/puar.13138 · Cited by 78
Kvarven, A., Strømland, E., Wollbrant, C., Andersson, D., Johannesson, M., Tinghög, G., Västfjäll, D., & Myrseth, K. O. R. (2020) · Journal of the Economic Science Association · doi:10.1007/s40881-020-00084-3 · Cited by 68
The Correlates of State Policy and the Structure of State Panel Data
Grossmann, M., Jordan, M. P., & McCrain, J. (2021) · State Politics & Policy Quarterly · doi:10.1017/spq.2021.17 · Cited by 71
“We lag all controls to account for concerns surrounding post-treatment bias (Montgomery, Nyhan, and Torres 2018), but that does not imply that the version with the most controls is necessarily preferable.”
BackgroundThe Generalizability of Online Experiments Conducted During The COVID-19 PandemicPreprint
Peyton, K., Huber, G. A., & Coppock, A. (2020) · doi:10.31235/osf.io/s45yg · Cited by 69
Bernauer, T., Mohrenberg, S., & Koubi, V. (2019) · The Review of International Organizations · doi:10.1007/s11558-019-09354-0 · Cited by 72
Trujillo, K. L., Motta, M., Callaghan, T., & Sylvester, S. M. (2020) · Political Research Quarterly · doi:10.1177/1065912920907695 · Cited by 66
Cirone, A. E., & Pepinsky, T. B. (2021) · Annual Review of Political Science · doi:10.1146/annurev-polisci-051120-104325 · Cited by 67
The (Null) Effects of Clickbait Headlines on Polarization, Trust, and Learning
Munger, K., Luca, M., Nagler, J., & Tucker, J. A. (2020) · Public Opinion Quarterly · doi:10.1093/poq/nfaa008 · Cited by 66
The Long-Run Effects of School Racial Diversity on Political Identity
Billings, S. B., Chyn, E. T., & Haggag, K. (2021) · American Economic Review Insights · doi:10.1257/aeri.20200336 · Cited by 65
The Influence of Identity Salience on Framing Effectiveness: An Experiment
Diamond, E. P. (2020) · Political Psychology · doi:10.1111/pops.12669 · Cited by 69
“The survey firm provided these control variables based on demographic questionnaires completed when the subjects joined the service, mitigating concerns of post-treatment bias (Montgomery et al., 2018).”
BackgroundScientific-Consensus Communication About Contested Science: A Preregistered Meta-Analysis
van Stekelenburg, A., Schaap, G., Veling, H., Riet, J. V. ‘., & Buijzen, M. (2022) · Psychological Science · doi:10.1177/09567976221083219 · Cited by 69
“This procedure may have affected the results of these experiments (Montgomery et al., 2018), potentially inflating effects.”
Kalla, J., & Porter, E. V. (2020) · British Journal of Political Science · doi:10.1017/s0007123419000711 · Cited by 63
How Do Civilians Attribute Blame for State Indiscriminate Violence?
Pechenkina, A., Bausch, A. W., & Skinner, K. K. (2016) · Journal of Peace Research · doi:10.1177/0022343319829798 · Cited by 60
Use of the auditory brainstem response for assessment of cochlear synaptopathy in humans
Bramhall, N. F. (2021) · The Journal of the Acoustical Society of America · doi:10.1121/10.0007484 · Cited by 61
Nyhan, B. J., Porter, E. V., Reifler, J., & Wood, T. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2995128 · Cited by 61
Self-citationDoes Class-Based Campaigning Work? How Working Class Appeals Attract and Polarize Voters
Robison, J., Stubager, R., Thau, M., & Tilley, J. R. (2020) · Comparative Political Studies · doi:10.1177/0010414020957684 · Cited by 67
“Thus, we asked respondents about their class identification at the beginning of the Danish survey, and therefore before the experiment, and on a previous wave of the US survey to avoid issues of post-treatment bias (Montgomery et al., 2018).”
BackgroundThe Feedback Effects of Controversial Police Use of Force
Mullinix, K. J., Bolsen, T., & Norris, R. J. (2020) · Political Behavior · doi:10.1007/s11109-020-09646-x · Cited by 57
Analyze the attentive and bypass bias: mock vignette checks in survey experiments
Kane, J., Velez, Y. R., & Barabas, J. (2023) · Political Science Research and Methods · doi:10.1017/psrm.2023.3 · Cited by 57
Johnson, M. S., Adams, V. M., & Byrne, J. (2023) · People and Nature · doi:10.1002/pan3.10557 · Cited by 57
Kane, J. (2024) · Journal of Experimental Political Science · doi:10.1017/xps.2024.1 · Cited by 60
Obeid, R., Bisson, J. B., Cosenza, A., Harrison, A. J., James, F., Saade, S., & Gillespie‐Lynch, K. (2020) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-020-04507-2 · Cited by 60
“Pleasantness of contact was also unexpectedly influenced by the race of the child randomly assigned to each vignette, so including it in identification models would constitute “conditioning on a post-treatment variable”, which is not sound (Montgomery et al. 2018).”
BackgroundHauwaert, S. M. V., & Huber, R. A. (2020) · European Journal of Political Research · doi:10.1111/1475-6765.12380 · Cited by 53
Voters’ Partisan Responses to Politicians’ Immoral Behavior
Walter, A. S., & Redlawsk, D. P. (2019) · Political Psychology · doi:10.1111/pops.12582 · Cited by 58
“If the work by Montgomery et al. (2018) is sustained through additional research, experimentalists in general will need to consider whether the potential costs of such strategies outweigh the risks of biasing results by measuring moderators after an experimental treatment.”
BackgroundInfluentialMartin, N. S., & Blinder, S. B. (2020) · Political Behavior · doi:10.1007/s11109-020-09596-4 · Cited by 57
Demonizing the enemy: the influence of Russian state-sponsored media on American audiences
Fisher, A. (2020) · Post-Soviet Affairs · doi:10.1080/1060586x.2020.1730121 · Cited by 51
“…Appendix D (https://www.aleksandrfisher.com/uploads/1/2/1/9/121940797/demonizingth eenemy_onlineappendix_june2019.pdf) and find identical results (Mongomery, Nyhan, and Torres 2018).2 I randomly assigned participants to four groups: (1) Control group – Individuals only complete the…”
BackgroundDisappearing dissent? Repression and state consolidation in Mexico
Osorio, J., Schubiger, L. I., & WEINTRAUB, M. I. (2018) · Journal of Peace Research · doi:10.1177/0022343318751035 · Cited by 54
Public perception and acceptance of negative emission technologies – framing effects in Switzerland
Wenger, A., Stauffacher, M., & Dallo, I. (2021) · Climatic Change · doi:10.1007/s10584-021-03150-9 · Cited by 50
Xiao, X. (2022) · New Media & Society · doi:10.1177/14614448221104948 · Cited by 54
“Following suggestions from prior research (Montgomery et al., 2018), before taking part in the experiment, participants completed a pretest questionnaire that contained demographics and media-use variables.”
MethodDisplaying News Source Trustworthiness Ratings Reduces Sharing Intentions for False News Posts
Celadin, T., Capraro, V., Pennycook, G., & Rand, D. G. (2023) · Journal of Online Trust and Safety · doi:10.54501/jots.v1i5.100 · Cited by 54
Anger and support for retribution in Mexico’s drug war
García‐Ponce, O., Young, L., & Zeitzoff, T. (2022) · Journal of Peace Research · doi:10.1177/00223433221075191 · Cited by 50
Stetter, C., Mennig, P., & Sauer, J. (2021) · European Review of Agricultural Economics · doi:10.1093/erae/jbab057 · Cited by 48
(Un)Natural Disasters: Electoral Cycles in Disaster Relief
Cooperman, A. D. (2022) · Comparative Political Studies · doi:10.1177/00104140211047410 · Cited by 47
“I do not include below average rainfall and drought declaration in the samemodel due to the potential for post-treatment bias (Montgomery et al., 2018) since below average rainfall predicts drought declaration.”
BackgroundScaling Up? Unpacking the Effect of Deliberative Mini-Publics on Legitimacy Perceptions
Germann, M., Mariën, S., & Muradova, L. (2022) · Political Studies · doi:10.1177/00323217221137444 · Cited by 65
“These results should be taken with care because conditioning on post-treatment manipulation and memory check questions could bias causal inferences (Montgomery et al., 2018).”
BackgroundAoki, N., Tatsumi, T., Naruse, G., & Maeda, K. (2024) · Government Information Quarterly · doi:10.1016/j.giq.2024.101965 · Cited by 75
Naoi, M. (2020) · Annual Review of Political Science · doi:10.1146/annurev-polisci-050317-063806 · Cited by 45
How Citizens Want Their Legislator to Vote
Dassonneville, R. C., Blais, A., Sevi, S., et al. (2020) · Legislative Studies Quarterly · doi:10.1111/lsq.12275 · Cited by 45
Taking Stock of Solidarity between People of Color: A Mini Meta-Analysis of Five Experiments
Pérez, E. O., Vicuña, B. V., & Ramos, A. (2023) · American Political Science Review · doi:10.1017/s0003055423001120 · Cited by 44
“To guard against possible post-treatment bias (Montgomery, Nyhan, and Torres 2018), we reanalyze our data by including all respondents who failed this check.”
High-Status Exemplars and the Misperception of the Asian-White Wealth Gap
Kuo, E. E., Kraus, M. W., & Richeson, J. A. (2019) · Social Psychological and Personality Science · doi:10.1177/1948550619867940 · Cited by 41
Verhulsdonk, I., Nai, A., & Karp, J. A. (2021) · Political Research Quarterly · doi:10.1177/10659129211023590 · Cited by 48
“To be sure, the fact that perceived humor of the message (the mediator) is measured, logically, after exposure to the experimental treatment could raise issues of post-treatment biases (Montgomery et al., 2018), most notably in terms of sequential ignorability - that is, the fact that unmeasured confounders could poss…”
BackgroundInfluentialThe effects of IMF loan conditions on poverty in the developing world
Biglaiser, G., & McGauvran, R. J. (2022) · Journal of International Relations and Development · doi:10.1057/s41268-022-00263-1 · Cited by 42
Network Centralization and Collective Adaptability to a Shifting Environment
Bernstein, E. S., Shore, J. C., & Jang, A. J. (2022) · Organization Science · doi:10.1287/orsc.2022.1584 · Cited by 46
Transparent communication of evidence does not undermine public trust in evidence
Kerr, J. R., Schneider, C. R., Freeman, A. L. J., Marteau, T. M., & van der Linden, S. (2022) · PNAS Nexus · doi:10.1093/pnasnexus/pgac280 · Cited by 42
“Although condition assignment had no significant effect on failure rates in our studies, there is still a risk of bias whensubsettingonpost-treatmentvariables(51).”
BackgroundThe role of media in influencing students’ STEM career interest
Chen, C., Hardjo, S., Sonnert, G., Hui, J., & Sadler, P. M. (2023) · International Journal of STEM Education · doi:10.1186/s40594-023-00448-1 · Cited by 50
“In other words, the nesting is a post-treatment event, which should not be adjusted for according to established literature in statistics (Montgomery et al., 2018; Rosenbaum, 1984), cautioning that adjusting for post-treatment information may increase bias in estimation, instead of decreasing the…”
BackgroundPragmatic bias impedes women’s access to political leadership
Corbett, C., Voelkel, J. G., Cooper, M., & Willer, R. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2112616119 · Cited by 50
How Dropping Subjects Who Failed Manipulation Checks Can Bias Your Results: An Illustrative Case
Varaine, S. (2022) · Journal of Experimental Political Science · doi:10.1017/xps.2022.28 · Cited by 41
Causal inference for psychologists who think that causal inference is not for them
Rohrer, J. M. (2024) · Social and Personality Psychology Compass · doi:10.1111/spc3.12948 · Cited by 51
Song, H., Markowitz, D. M., & Taylor, S. H. (2022) · Journal of Communication · doi:10.1093/joc/jqac017 · Cited by 41
Communicating the Scientific Consensus on Climate Change: Diverse Audiences and Effects Over Time
Goldberg, M. H., Gustafson, A., van der Linden, S., Rosenthal, S. A., & Leiserowitz, A. (2022) · Environment and Behavior · doi:10.1177/00139165221129539 · Cited by 46
“In hindsight, we realize the placement of this question after the treatment creates the risk of post-treatment bias (Montgomery et al., 2018).”
BackgroundCan Politicians Exploit Ethnic Grievances? An Experimental Study of Land Appeals in Kenya
Horowitz, J. D., & Klaus, K. (2018) · Political Behavior · doi:10.1007/s11109-018-9485-1 · Cited by 39
Field experiments in operations management
Gao, Y. A., Li, M., & Sun, S. (2023) · Journal of Operations Management · doi:10.1002/joom.1240 · Cited by 39
Bramhall, N. F., McMillan, G. P., & Kampel, S. D. (2020) · Hearing Research · doi:10.1016/j.heares.2021.108310 · Cited by 40
Pickett, J. T. (2018) · Crime & Delinquency · doi:10.1177/0011128718763136 · Cited by 43
Solodoch, O. (2020) · Political Studies · doi:10.1177/0032321720946163 · Cited by 37
Truth be told: How “true” and “false” labels influence user engagement with fact-checks
Aruguete, N., Bachmann, I., Calvo, E. F., Valenzuela, S., & Ventura, T. (2023) · New Media & Society · doi:10.1177/14614448231193709 · Cited by 39
“As defined by Montgomery et al. (2018), post-treatment bias includes “common practices including dropping participants who fail manipulation checks; controlling for variables measured after the treatment, such as potential mediators; or subsetting samples based on post-treatment variables.””
The Effect of Protesters’ Gender on Public Reactions to Protests and Protest Repression
Naunov, M. (2024) · American Political Science Review · doi:10.1017/s0003055424000133 · Cited by 46
“8 To avoid post-treatment bias, I do not condition my analysis on the manipulation checks (Montgomery, Nyhan, and Torres 2018).”
Partisan‐Motivated Evaluations of Sexual Misconduct and the Mitigating Role of the #MeToo Movement
Klar, S., & McCoy, A. (2021) · American Journal of Political Science · doi:10.1111/ajps.12619 · Cited by 38
Metadata
- Authors
- Montgomery, J. M., Nyhan, B., & Torres, M.
- Affiliations
- Washington University in St. Louis (US)
- Dartmouth College (US)
- Venue
- American Journal of Political Science
- Publisher
- Wiley
- Published
- 2018-03-30
- Volume
- 62
- Issue
- 3
- Pages
- 760-775
- ISSN
- 0092-5853 (ISSN-L)
- 1540-5907
- OpenAlex
- W2794592087
- Cited by
- 800
- References
- 49
- Citations by year
- 2026: 68 · 2025: 128 · 2024: 120 · 2023: 114 · 2022: 105 · 2021: 89 · 2020: 95 · 2019: 48 · 2018: 27
- FWCI
- 53.73
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Political Science (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- causal effect estimation, experimental design, confounding, bias in treatment effect estimates
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
