The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index; none of the 26 sources its claims cite is checked for support yet.
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Open accessAbstract
In principle, experiments offer a straightforward method for social scientists to accurately estimate causal effects. However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation. Typical examples include controlling for posttreatment variables in statistical models, eliminating observations based on posttreatment criteria, or subsetting the data based on posttreatment variables. Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects. Moreover, problems associated with conditioning on posttreatment variables remain largely unrecognized in the field, which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals. We demonstrate the severity of experimental posttreatment bias analytically and document the magnitude of the potential distortions it induces using visualizations and reanalyses of real‐world data. We conclude by providing applied researchers with recommendations for best practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
52 Own interpretation · 30 Own result · 18 Not checked yet · 12 No basis given · 1 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“almost 1 in 10 engaged in both (10.7%, eight studies)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“14.7% of studies subsetted the data based on potential posttreatment criteria (11 of 75 studies reviewed)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“problems associated with conditioning on posttreatment variables remain largely unrecognized in the field”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“21.3%(16 of 75) controlled for a posttreatment covariate in a statistical model”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the studies that controlled for a posttreatment variable, six used a mediation technique (8%).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most strikingly, 12% of studies conditioned on a variable shown to be affected by the experimental treatment in analyses contained within the article itself (9 of 75).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“few acknowledge potential concerns regarding the bias that posttreatment conditioning can introduce”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the magnitude of the negative coefficient approximately doubles relative to the unconditional estimate among respondents who perceive themselves as further from Eid (-.460; 95% CI: -0.567, -0.353).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Trump's endorsement reduced the likelihood of supporting Eid's retention by -0.214 (p < .01,95% CI: -0.301, -0.127) on the four-point scale.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As described in the subsection “Practices That Lead to Posttreatment Bias,” conditioning on ideological distance actually unbalances the sample by respondents' self-reported ideology even though self-reported ideology is measured pretreatment.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, among respondents who perceive themselves as ideologically close to Eid, treatment group respondents are significantly more conservative on our 7-point ideology scale than are control group respondents (4.633 versus 3.652, p < .01 in a t-test).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reason is that the treatment increases perceptions of Eid's conservatism (from 3.821 in the control group to 4.781 in the treatment group, p < .01 in a t-test).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This effect becomes significant at the p < .05level in the fourth column when we drop cases and include lagged controls.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results are largely unchanged when we include lagged behavioral controls in the second column.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first column, which omits any posttreatment controls or conditioning, shows that the appropriations treatment is significant only in the full information condition.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when we instead drop cases based on contributor targeting in the third column, the limited information treatment becomes significant at the p < .10 level in the salary condition.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we find that the magnitude of the effect estimates varies substantially when we condition on posttreatment variables.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 3 demonstrates that posttreatment conditioning induces substantial differences in the estimated effects of DGH's treatments.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We replicate the appropriations treatment effect for full information groups, but our analysis raises concerns about posttreatment bias for both the limited information effect in the salary condition and the appropriation effect in the limited information condition.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Unfortunately, the lagged measures are themselves affected by the manipulations.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We find that a significant fraction of the experimental studies published in the discipline's most prestigious journals drop observations based on posttreatment variables or control for posttreatment variables in their statistical analysis.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| No conditioning on posttreatment variables | 52.0% |”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plainn = 75- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Controls for/interacts with a | 21.3% |”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plainn = 75- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Drops cases based on posttreatment | 14.7% |”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plainn = 75- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Engages in posttreatment conditioning | 46.7% |”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plainn = 75- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In total, the evidence we provide demonstrates that posttreatment conditioning is a frequent and significant problem in political science.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Both types of posttreatment conditioning | 10.7% |”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plainn = 75- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Any method that generates an unbiased estimate of the DCEV ( ) will then also generate an unbiased estimate of the ATE ( ).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conditioning on posttreatment variables eliminates the advantages of randomization because we are now comparing dissimilar groups.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Intuition of Posttreatment Bias
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirically, then, the answer to the question of whether the discipline already understands posttreatment bias is clear: It does not.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The problem is that by conditioning on a posttreatment variable, we have unbalanced the treatment and control groups with respect to every other possible confounder.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Intuition of Posttreatment Bias
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“if we violate Assumption (1), we would also need to somehow adjust for bias resulting from imbalances in the unobserved confounder u”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In short, nearly half of the experimental studies published in our discipline's most prestigious journals during this period raise concerns about posttreatment bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For any finite ATE, we can construct examples where the bias will be -∞, ∞, or anything in between depending on the value of parameters like Y”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Once we have conditioned on a posttreatment variable, we have eliminated the assurance of unconfoundedness provided by randomization.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Problem with Conditioning on Posttreatment Variables
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“controlling for a posttreatment variable leads to a new bias that is simply a combination of the biased estimates we would get from selecting only cases where x = 1 and the estimates from selecting only cases where x = 0.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Including Posttreatment Variables as Covariates.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even in such a favorable context, conditioning on x still leads to inconsistent estimates because the posttreatment covariate (x) and the outcome (y) share an unmeasured cause (u).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Problem with Conditioning on Posttreatment Variables
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By selecting based on a criterion that is partially a function of unobserved covariates and the treatment, we have inadvertently created imbalance in the treatment and control conditions with respect to u.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Dropping or Selecting Observations Based on Criteria Influenced by the Treatment.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“including posttreatment variables as covariates for any of these reasons can bias estimates by creating imbalance with respect to the unmeasured confounder.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Including Posttreatment Variables as Covariates.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, because perceptions of Eid's ideology were measured after the manipulation, any analysis that conditions on ideological distance will be biased.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Adjusting for imbalance in unobservable variables is more challenging, requiring either the availability of exogenous instruments and/or stronger (and more limiting) assumptions”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Standard estimates such as the difference in conditional means ( ˆ ) will therefore be biased regardless of sample size, measurement precision, or estimation method.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Problem with Conditioning on Posttreatment Variables
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“when we include a posttreatment variable in the set of conditioning variables either directly or indirectly, Assumption (1) is violated.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Problem with Conditioning on Posttreatment Variables
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Attrition rates were 2.3% in control and 1.5% in treatment.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plainn = 1,182- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“published research in political science identified posttreatment bias (in passing) as problematic over a decade ago”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
What the sources say
- Not checked yetcites King and Zeng 2006
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Acharya, Blackwell, and Sen (2016), for instance, show that as many as four out of five observational studies in top journals may condition on posttreatment variables.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
What the sources say
- Not checked yetcites Acharya, Blackwell, and Sen 2016
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“First, DGH exclude cases of so-called “perverse” targeting of a contributor when at least one citizen did not contribute (2015,119).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
What the sources say
- Not checked yetcites Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“there is still significant confusion in the wider discipline about its sources and consequences”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, all of these subsample estimates are also biased.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, perverse targeting is a posttreatment behavior given the expected effect of the manipulations.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Another approach to noncompliance is to estimate a two-stage least squares model using random assignment as an instrument for treatment status.Here again, however, we are estimating a different estimand known as the complier average causal effect (CACE).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Instrumental Variables to Address Noncompliance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Second, DGH controls for lagged average contributions, average resoluteness, and perverse or predatory targeting to try to ensure that the effects of the treatments at time t are not fully mediated by behavior and outcomes in previous periods (2015,122).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
What the sources say
- Not checked yetcites Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The simplest is to calculate the difference in outcomes between respondents assigned to receive treatment and those assigned to receive the control, which is an unbiased estimate of the intention to treat (ITT) effect.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Instrumental Variables to Address Noncompliance
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Though it is not necessary to do so (randomization eliminates omitted variable bias in expectation), regression adjustment for covariates has been shown to induce only minor bias and to potentially increase efficiency under realistic conditions (e.g., Lin 2013).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Pretreatment Moderators, Control Variables, and Attention Checks
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Researchers may neglect this issue when the content of the attention check is not directly related to the experimental randomization, but many treatments could differentially affect the types of participants who pass these measures via other mechanisms (e.g., changing respondent engagement or affecting the contents of working memory), thereby imbalancing the sample.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Pretreatment Moderators, Control Variables, and Attention Checks
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Dickson, Gordon, and Huber (2015) note that both models are sensitive to model specification; our analysis suggests that these results may be attributable to posttreatment bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the reasons stated above, simply dropping cases or controlling for compliance status in a regression model can lead to biased estimates of the ATE.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Instrumental Variables to Address Noncompliance
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
In the paper
“An alternative approach is to collect outcome data among some subjects with missing outcomes (Coppock et al. 2017), which combines double sampling with extreme value bounds.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
What the sources say
- Not checked yetcites Coppock et al. 2017
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Many mediation methods like Imai, Keele, and Tingley (2010) or related alternatives such as marginal structural models (Robins, Hernan, and Brumback 2000) or structural nested mean models (Robins 1999) are founded on the exact same assumption.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
What the sources say
- Not checked yetcites Imai, Keele, and Tingley (2010)
- Not checked yetcites Robins, Hernan, and Brumback 2000
- Not checked yetcites Robins 1999
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, the most common mediation models all rely in some way on the assumption that researchers have access to every relevant covariate.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Interpretation is especially thorny in the presence of two-sided noncompliance where compliance status cannot be directly observed and an additional monotonicity assumption (no defiers) must be invoked (Angrist, Imbens, and Rubin 1996;see Gerber and Green 2012, 131-209, for more on these points).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Instrumental Variables to Address Noncompliance
What the sources say
- Not checked yetcites Angrist, Imbens, and Rubin 1996
- Not checked yetcites Gerber and Green 2012
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, unless attrition and nonresponse are unrelated to potential outcomes and treatment, this practice is equivalent to conditioning on a posttreatment variable.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, this approach, which is frequently attributed to Baron and Kenny (1986), does not identify the direct or indirect effects of interest absent additional assumptions including sequential ignorability, which essentially assumes away the possibility of unmeasured confounders.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
What the sources say
- Not checked yetcites Baron and Kenny (1986)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without them, we cannot be sure a variable is not posttreatment unless we accept the sharp null of no effect for any unit.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Critical parameters including the treatment effect, the effect of the treatment on the covariate, and the effects of the confounders on outcomes and covariates were constant for each individual.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Failing to reject the null hypothesis is not direct evidence for the null.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, failing to reject the null hypothesis H 0 : E (x|T = 0) = E (x|T = 1) does not rule out posttreatment bias in analyses that condition on x.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Scholars, reviewers, and editors should recognize that any attempt to include posttreatment variables in a mediation analysis comes at an inferential cost.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unpacking the “black box” of experimental treatments must be paid for in the form of assumptions, biased estimates, or both.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“For example, Bullock, Green, and Ha (2010) outline experimental designs that facilitate the study of causal mediation by directly manipulating posttreatment mediators as well as treatment assignments.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
What the sources say
- Not checked yetcites Bullock, Green, and Ha (2010)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Indeed, Aronow, Baron, and Pinson (2015) show that in a more general setting, it will often not be possible to provide bounds that exclude -∞ and ∞ for the potential bias from conditioning on a posttreatment variable.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
What the sources say
- Not checked yetcites Aronow, Baron, and Pinson (2015)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown above, standard approaches such as controlling for intervening variables or subsetting data are incorrect.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Happily, then, the bias that posttreatment conditioning introduces into so much experimental research can easily be avoided.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In particular, asking questions about certain highly salient covariates like group identification before an outcome variable can affect subsequent responses (e.g., Kosloff et al. 2010;Leach et al. 2010).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
What the sources say
- Not checked yetcites Kosloff et al. 2010
- Not checked yetcites Leach et al. 2010
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, once an experiment has been conducted, it is not possible to rule out alternative mechanisms without the possibility of posttreatment bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 15
- It may state two separate claims as one. 22
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while some studies lost cases due to posttreatment attrition (8.0%)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“About 1 in 4 drop cases or subset the data based on posttreatment criteria, and nearly a third include posttreatment variables as covariates.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These effects are opposite in sign and in both cases highly significant (p < .01 in both directions).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The sign of the estimated treatment effect reverses in the subsample of respondents who perceive themselves as being close to Eid, becoming positive (.257, 95% CI: 0.132, 0.382)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most notably, the appropriations treatment effect estimate in the limited information condition more than doubles in magnitude and becomes nearly statistically significant in the fourth column (p < .11).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By contrast, the effect of appropriations among groups with limited information and the effect of limited information in either compensation group are not distinguishable from zero.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These practices are typically employed in an effort to address practical problems like noncompliance or to try to answer difficult inferential questions such as identifying causal mechanisms.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Reviewers often ask authors to try to rule out alternative explanations in this way.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Insufficient information to code | 1.3% |”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plainn = 75- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assumption (1) implies that individuals in the treatment and control conditions will be similar in expectation with respect to unobserved confounders such as SES.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As long as we do not condition on a posttreatment variable, randomization guarantees that Assumption (1) is satisfied and both quantities go to zero.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, Assumption (1) requires that x is not causally related to T”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although it is possible to construct examples where this bias is zero, it will not be zero in general.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Dropping or Selecting Observations Based on Criteria Influenced by the Treatment.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“dropping cases or subsetting based on posttreatment criteria will unbalance the treatment and control conditions with respect to unmeasured confounders and bias our treatment effect estimates.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Dropping or Selecting Observations Based on Criteria Influenced by the Treatment.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“the dangers of posttreatment bias have long been recognized in the fields of statistics, econometrics, and political methodology”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
What the sources say
- Not checked yetcites Acharya, Blackwell, and Sen 2016
- Not checked yetcites Elwert and Winship 2014
- Not checked yetcites King and Zeng 2006
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Instead, it is a biased estimate of the direct effect of the Trump endorsement, and the bias can be in any direction at all.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, control group participants who think Eid is centrist on average and perceive themselves to be relatively close to her are being compared to treatment group participants who think she is close to them after finding out she was endorsed by President Trump.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“More recent work has amplified these points in the context of observational research”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
What the sources say
- Not checked yetcites Acharya, Blackwell, and Sen 2016
- Not checked yetcites Blackwell 2013
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By conditioning on a posttreatment variable, we have unbalanced the treatment and control groups in terms of ideology and unmeasured confounders and thus biased the treatment effect estimate.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this scenario, dropping respondents based on posttreatment attention checks is the equivalent of selecting on a posttreatment covariate and would again risk bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Pretreatment Moderators, Control Variables, and Attention Checks
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, both approaches provide biased treatment effect estimates that do not correspond to meaningful causal estimands.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Though it is not necessary to do so (randomization eliminates omitted variable bias in expectation), regression adjustment for covariates has been shown to induce only minor bias and to potentially increase efficiency under realistic conditions (e.g., Lin 2013).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Pretreatment Moderators, Control Variables, and Attention Checks
What the sources say
- Not checked yetcites Lin 2013
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Dickson, Gordon, and Huber (2015) note that both models are sensitive to model specification; our analysis suggests that these results may be attributable to posttreatment bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
What the sources say
- Not checked yetcites Dickson, Gordon, and Huber (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We replicate the appropriations treatment effect for full information groups, but our analysis raises concerns about posttreatment bias for both the limited information effect in the salary condition and the appropriation effect in the limited information condition.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Under more realistic assumptions, however, the choices are more limited: Gerber and Green (2012) recommend extreme value bounds (Manski 1989), where analysts estimate the largest and smallest ATEs possible if missing information were filled in with extreme outcomes.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
What the sources say
- Not checked yetcites Gerber and Green (2012)
- Not checked yetcites Manski 1989
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“If we are willing to assume that missingness is not a function of unmeasured confounders, we can use familiar methods such as imputation or marginal structural models.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Missingness is not a function of unmeasured confounders.
In the paper
“Finally, Huber (2012) seeks to reduce bias from attrition using inverse probability weighting and instrumental variables for missingness.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
What the sources say
- Not checked yetcites Huber (2012)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is no reason to believe that these assumptions are correct in real-world data.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Critical parameters including the treatment effect, the effect of the treatment on the covariate, and the effects of the confounders on outcomes and covariates were constant for each individual.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach is not only very difficult to execute (it requires a treatment that affects the mediator but not the outcome) but also subject to criticism for implausible assumptions.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, even in the simplified examples presented above, posttreatment bias will not be eliminated unless the effect of the treatment on the covariate (␥ ) is precisely zero-something that cannot be established using traditional hypothesis testing.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though these intentions are laudable, we demonstrate that posttreatment conditioning undermines the value of randomization and biases treatment effect estimates using analytical results as well as a reanalysis of real-world data from two studies.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, it is important to note that posttreatment bias cannot be easily diagnosed or remedied empirically.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Imai, Tingley, and Yamamoto (2013) outline several designs that allow researchers to estimate mediation effects, but these too come with additional assumptions (e.g., a consistency assumption) or require use of less intuitive estimands (e.g., average complier indirect effects).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
What the sources say
- Not checked yetcites Imai, Tingley, and Yamamoto (2013)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we also show that scholars can address the concerns that motivate the use of these practices using existing analytical approaches.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many of the practices described above appear to be driven by authors' efforts to show that their proposed mechanism is responsible for the treatment effect.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, mediation analyses require strong assumptions that may be inconsistent with the goals of experimental research.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The authors state this as an assumption.
Treatment assignment is independent of potential outcomes conditional on covariates.
“Assumption (1) : (y [T =1] , y [T =0] ) ⊥ ⊥ T |X, which states that treatment assignment is independent of potential outcomes conditional on covariates.”
The authors state this as an assumption.
All variables are observed without error.
“For the sake of expositional clarity, and without loss of generality, we assume that all variables are observed without error.”
The authors state this as an assumption.
The main parameters in Model 4 do not vary as a function of T or x.
“Note that Model 4 also assumes that the main parameters in the model ( , ␣ Y , , , and Y ) do not vary as a function of T or x, which is why we can move these parameters outside of the expectations.”
The authors state this as an assumption.
The influence of unmeasured confounders along the various causal paths will not somehow cancel out.
“For instance, we assume that the influence of unmeasured confounders along the various causal paths will not somehow cancel out.”
The authors state this as an assumption.
x has no direct effect on y in the true model.
“If we instead allow x to have a direct effect on y in the true model, the biases we describe below still hold, but the calculations involved are more complex.We make this simplifying assumption so that we can focus our exposition on the posttreatment bias that arises from unblocking the path from u to y.”
The authors state this as an assumption.
β = 0.
“To simplify exposition, we focus here only on the bias resulting from the imbalance in u induced by controlling for the posttreatment variable x by assuming that  = 0.”
The authors state this as an assumption.
Critical parameters including the treatment effect, the effect of the treatment on the covariate, and the effects of the confounders on outcomes and covariates were constant for each individual.
“Second, we made the simplifying assumption in our examples above that critical parameters including the treatment effect ( ), the effect of the treatment on the covariate (␥ ), and the effects of the confounders on outcomes and covariates ( Y , X ) were constant for each individual.”
The authors state this as an assumption.
Missingness is not a function of unmeasured confounders.
“If we are willing to assume that missingness is not a function of unmeasured confounders, we can use familiar methods such as imputation or marginal structural models.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“An important caveat for these analyses is that attention was measured post-treatment, which has the potential to undermine inferences drawn from these data43”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Additional information
“• Reanalysis of Broockman and Butler (2015)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
“an additional demonstration of posttreatment bias using data fromBroockman and Butler (2017).”
Citations
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Marginal Structural Models and Causal Inference in Epidemiology
Robins, J. M., Hernán, M. Á., & Brumback, B. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00011 · Cited by 5,581
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
A general approach to causal mediation analysis.
Imai, K., Keele, L., & Tingley, D. (2010) · Psychological Methods · doi:10.1037/a0020761 · Cited by 4,143
Identification and estimation of local average treatment effects
Imbens, G. W., & Angrist, J. D. (1994) · Econometrica · doi:10.2307/2951620 · Cited by 4,101
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Identification and Estimation of Local Average Treatment Effects
Angrist, J. D., & Imbens, G. W. (1995) · National Bureau of Economic Research · doi:10.3386/t0118 · Cited by 2,915
Field Experiments: Design, Analysis, and Interpretation
GREEN, D. P. (2024) · doi:10.61700/sgeirzx0v0mjh1252 · Cited by 1,322
Cited by
Sorted by citations
Shifting attention to accuracy can reduce misinformation online
Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021) · Nature · doi:10.1038/s41586-021-03344-2 · Cited by 1,155
Vaccari, C., & Chadwick, A. (2020) · Social Media + Society · doi:10.1177/2056305120903408 · Cited by 978
“As possible solutions, they cite designs that include “a treatment that affects the mediator but not the outcome” (Montgomery et al., 2018, p. 772).”
BackgroundInfluentialThe shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
Nyhan, B. J., Porter, E. V., Reifler, J., & Wood, T. (2017) · Political Behavior · doi:10.1007/s11109-019-09528-x · Cited by 557
Self-citationThe Ties That Double Bind: Social Roles and Women's Underrepresentation in Politics
Teele, D. L., Kalla, J., & Rosenbluth, F. M. (2018) · American Political Science Review · doi:10.1017/s0003055418000217 · Cited by 408
Metadata
- Authors
- Montgomery, J. M., Nyhan, B., & Torres, M.
- Affiliations
- Washington University in St. Louis (US)
- Dartmouth College (US)
- Venue
- American Journal of Political Science
- Publisher
- Wiley
- Published
- 2018-03-30
- Volume
- 62
- Issue
- 3
- Pages
- 760-775
- ISSN
- 0092-5853 (ISSN-L)
- 1540-5907
- OpenAlex
- W2794592087
- Cited by
- 800
- References
- 49
- Citations by year
- 2026: 68 · 2025: 128 · 2024: 120 · 2023: 114 · 2022: 105 · 2021: 89 · 2020: 95 · 2019: 48 · 2018: 27
- FWCI
- 53.73
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Political Science (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- causal effect estimation, experimental design, confounding, bias in treatment effect estimates
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
