The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index; none of the 26 sources its claims cite is checked for support yet.
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Open accessAbstract
In principle, experiments offer a straightforward method for social scientists to accurately estimate causal effects. However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation. Typical examples include controlling for posttreatment variables in statistical models, eliminating observations based on posttreatment criteria, or subsetting the data based on posttreatment variables. Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects. Moreover, problems associated with conditioning on posttreatment variables remain largely unrecognized in the field, which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals. We demonstrate the severity of experimental posttreatment bias analytically and document the magnitude of the potential distortions it induces using visualizations and reanalyses of real‐world data. We conclude by providing applied researchers with recommendations for best practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 26 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“almost 1 in 10 engaged in both (10.7%, eight studies)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear? Based: unclearA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“14.7% of studies subsetted the data based on potential posttreatment criteria (11 of 75 studies reviewed)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“problems associated with conditioning on posttreatment variables remain largely unrecognized in the field”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“21.3%(16 of 75) controlled for a posttreatment covariate in a statistical model”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the studies that controlled for a posttreatment variable, six used a mediation technique (8%).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most strikingly, 12% of studies conditioned on a variable shown to be affected by the experimental treatment in analyses contained within the article itself (9 of 75).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“few acknowledge potential concerns regarding the bias that posttreatment conditioning can introduce”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the magnitude of the negative coefficient approximately doubles relative to the unconditional estimate among respondents who perceive themselves as further from Eid (-.460; 95% CI: -0.567, -0.353).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Trump's endorsement reduced the likelihood of supporting Eid's retention by -0.214 (p < .01,95% CI: -0.301, -0.127) on the four-point scale.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 15
- It may state two separate claims as one. 22
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while some studies lost cases due to posttreatment attrition (8.0%)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“About 1 in 4 drop cases or subset the data based on posttreatment criteria, and nearly a third include posttreatment variables as covariates.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These effects are opposite in sign and in both cases highly significant (p < .01 in both directions).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The sign of the estimated treatment effect reverses in the subsample of respondents who perceive themselves as being close to Eid, becoming positive (.257, 95% CI: 0.132, 0.382)”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most notably, the appropriations treatment effect estimate in the limited information condition more than doubles in magnitude and becomes nearly statistically significant in the fourth column (p < .11).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By contrast, the effect of appropriations among groups with limited information and the effect of limited information in either compensation group are not distinguishable from zero.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These practices are typically employed in an effort to address practical problems like noncompliance or to try to answer difficult inferential questions such as identifying causal mechanisms.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Reviewers often ask authors to try to rule out alternative explanations in this way.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Insufficient information to code | 1.3% |”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
Strength:plain· not yet calibratedn = 75Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assumption (1) implies that individuals in the treatment and control conditions will be similar in expectation with respect to unobserved confounders such as SES.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As long as we do not condition on a posttreatment variable, randomization guarantees that Assumption (1) is satisfied and both quantities go to zero.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, Assumption (1) requires that x is not causally related to T”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Why Experiments Generate Unbiased Estimates of Treatment Effects
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
- The authors state this as an assumption: Treatment assignment is independent of potential outcomes conditional on covariates.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although it is possible to construct examples where this bias is zero, it will not be zero in general.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Dropping or Selecting Observations Based on Criteria Influenced by the Treatment.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“dropping cases or subsetting based on posttreatment criteria will unbalance the treatment and control conditions with respect to unmeasured confounders and bias our treatment effect estimates.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Dropping or Selecting Observations Based on Criteria Influenced by the Treatment.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“the dangers of posttreatment bias have long been recognized in the fields of statistics, econometrics, and political methodology”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Abstract
What the sources say
- Not checked yetcites Acharya, Blackwell, and Sen 2016
- Not checked yetcites Elwert and Winship 2014
- Not checked yetcites King and Zeng 2006
Strength:boosted· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Instead, it is a biased estimate of the direct effect of the Trump endorsement, and the bias can be in any direction at all.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, control group participants who think Eid is centrist on average and perceive themselves to be relatively close to her are being compared to treatment group participants who think she is close to them after finding out she was endorsed by President Trump.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“More recent work has amplified these points in the context of observational research”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Don't We Already Know This?
What the sources say
- Not checked yetcites Acharya, Blackwell, and Sen 2016
- Not checked yetcites Blackwell 2013
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By conditioning on a posttreatment variable, we have unbalanced the treatment and control groups in terms of ideology and unmeasured confounders and thus biased the treatment effect estimate.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Analysis: An Original Study of Judge Perceptions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this scenario, dropping respondents based on posttreatment attention checks is the equivalent of selecting on a posttreatment covariate and would again risk bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Pretreatment Moderators, Control Variables, and Attention Checks
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, both approaches provide biased treatment effect estimates that do not correspond to meaningful causal estimands.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Though it is not necessary to do so (randomization eliminates omitted variable bias in expectation), regression adjustment for covariates has been shown to induce only minor bias and to potentially increase efficiency under realistic conditions (e.g., Lin 2013).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Pretreatment Moderators, Control Variables, and Attention Checks
What the sources say
- Not checked yetcites Lin 2013
Strength:plain· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“Dickson, Gordon, and Huber (2015) note that both models are sensitive to model specification; our analysis suggests that these results may be attributable to posttreatment bias.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
What the sources say
- Not checked yetcites Dickson, Gordon, and Huber (2015)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We replicate the appropriations treatment effect for full information groups, but our analysis raises concerns about posttreatment bias for both the limited information effect in the salary condition and the appropriation effect in the limited information condition.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Reanalysis: Dickson, Gordon, and Huber (2015)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Under more realistic assumptions, however, the choices are more limited: Gerber and Green (2012) recommend extreme value bounds (Manski 1989), where analysts estimate the largest and smallest ATEs possible if missing information were filled in with extreme outcomes.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
What the sources say
- Not checked yetcites Gerber and Green (2012)
- Not checked yetcites Manski 1989
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“If we are willing to assume that missingness is not a function of unmeasured confounders, we can use familiar methods such as imputation or marginal structural models.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
- The authors state this as an assumption: Missingness is not a function of unmeasured confounders.
In the paper
“Finally, Huber (2012) seeks to reduce bias from attrition using inverse probability weighting and instrumental variables for missingness.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Use Double Sampling, Extreme Value Bounds, or Instruments to Account for Attrition
What the sources say
- Not checked yetcites Huber (2012)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is no reason to believe that these assumptions are correct in real-world data.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes? Checkable: unclear✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- The authors state this as an assumption: Critical parameters including the treatment effect, the effect of the treatment on the covariate, and the effects of the confounders on outcomes and covariates were constant for each individual.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach is not only very difficult to execute (it requires a treatment that affects the mediator but not the outcome) but also subject to criticism for implausible assumptions.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, even in the simplified examples presented above, posttreatment bias will not be eliminated unless the effect of the treatment on the covariate (␥ ) is precisely zero-something that cannot be established using traditional hypothesis testing.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though these intentions are laudable, we demonstrate that posttreatment conditioning undermines the value of randomization and biases treatment effect estimates using analytical results as well as a reanalysis of real-world data from two studies.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, it is important to note that posttreatment bias cannot be easily diagnosed or remedied empirically.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > The Inadequacy of Empirical Tests for Posttreatment Bias
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Imai, Tingley, and Yamamoto (2013) outline several designs that allow researchers to estimate mediation effects, but these too come with additional assumptions (e.g., a consistency assumption) or require use of less intuitive estimands (e.g., average complier indirect effects).”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Understand the Costs of Mediation Analysis
What the sources say
- Not checked yetcites Imai, Tingley, and Yamamoto (2013)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we also show that scholars can address the concerns that motivate the use of these practices using existing analytical approaches.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many of the practices described above appear to be driven by authors' efforts to show that their proposed mechanism is responsible for the treatment effect.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
Strength:hedged· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, mediation analyses require strong assumptions that may be inconsistent with the goals of experimental research.”
§ How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It > Conclusion
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
The authors state this as an assumption.
Treatment assignment is independent of potential outcomes conditional on covariates.
“Assumption (1) : (y [T =1] , y [T =0] ) ⊥ ⊥ T |X, which states that treatment assignment is independent of potential outcomes conditional on covariates.”
The authors state this as an assumption.
All variables are observed without error.
“For the sake of expositional clarity, and without loss of generality, we assume that all variables are observed without error.”
The authors state this as an assumption.
The main parameters in Model 4 do not vary as a function of T or x.
“Note that Model 4 also assumes that the main parameters in the model ( , ␣ Y , , , and Y ) do not vary as a function of T or x, which is why we can move these parameters outside of the expectations.”
The authors state this as an assumption.
The influence of unmeasured confounders along the various causal paths will not somehow cancel out.
“For instance, we assume that the influence of unmeasured confounders along the various causal paths will not somehow cancel out.”
The authors state this as an assumption.
x has no direct effect on y in the true model.
“If we instead allow x to have a direct effect on y in the true model, the biases we describe below still hold, but the calculations involved are more complex.We make this simplifying assumption so that we can focus our exposition on the posttreatment bias that arises from unblocking the path from u to y.”
The authors state this as an assumption.
β = 0.
“To simplify exposition, we focus here only on the bias resulting from the imbalance in u induced by controlling for the posttreatment variable x by assuming that  = 0.”
The authors state this as an assumption.
Critical parameters including the treatment effect, the effect of the treatment on the covariate, and the effects of the confounders on outcomes and covariates were constant for each individual.
“Second, we made the simplifying assumption in our examples above that critical parameters including the treatment effect ( ), the effect of the treatment on the covariate (␥ ), and the effects of the confounders on outcomes and covariates ( Y , X ) were constant for each individual.”
The authors state this as an assumption.
Missingness is not a function of unmeasured confounders.
“If we are willing to assume that missingness is not a function of unmeasured confounders, we can use familiar methods such as imputation or marginal structural models.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“An important caveat for these analyses is that attention was measured post-treatment, which has the potential to undermine inferences drawn from these data43”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Year of the Broockman and Butler reanalysis differs
MinorThe discussion identifies the reanalyzed study as Broockman and Butler (2017), while the supporting-information list calls it Broockman and Butler (2015).
“• Reanalysis of Broockman and Butler (2015)”
“an additional demonstration of posttreatment bias using data fromBroockman and Butler (2017).”
Citations
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Marginal Structural Models and Causal Inference in Epidemiology
Robins, J. M., Hernán, M. Á., & Brumback, B. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00011 · Cited by 5,581
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
A general approach to causal mediation analysis.
Imai, K., Keele, L., & Tingley, D. (2010) · Psychological Methods · doi:10.1037/a0020761 · Cited by 4,143
Identification and estimation of local average treatment effects
Imbens, G. W., & Angrist, J. D. (1994) · Econometrica · doi:10.2307/2951620 · Cited by 4,101
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Identification and Estimation of Local Average Treatment Effects
Angrist, J. D., & Imbens, G. W. (1995) · National Bureau of Economic Research · doi:10.3386/t0118 · Cited by 2,915
Field Experiments: Design, Analysis, and Interpretation
GREEN, D. P. (2024) · doi:10.61700/sgeirzx0v0mjh1252 · Cited by 1,322
Flagged first, then by citations
1 of the 49 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 601–700 of 793
How Social Cleavages Facilitate State Revenue Extraction: Evidence from Qing ChinaPreprint
Mazumder, S., & Wang, Y. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3622309
Framed National Images Influence Policy Attitudes Among Targeted Foreign CitizensPreprint
Rhee, K., Crabtree, C. D., & Horiuchi, Y. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3776777
Trauma Propagation in Social NetworksPreprint
Sadish, D. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3799738
Estimating firms’ bank-switching costsPreprint
Liaudinskas, K., & Grigaitçe, K. (2021) · RePEc: Research Papers in Economics
All Options Are on the Table? Time Horizons and the Decision-Making Process in Conflict
Dvir, R. (2021) · Foreign Policy Analysis · doi:10.1093/fpa/orab028
The Public Is Less Likely to Support Women (But not Men) Politicians when They Wear MasksPreprint
Muroga, K., & Crabtree, C. D. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3917681
(Un)Natural Disasters: Distributive Politics in Northeast BrazilPreprint
Cooperman, A. (2021) · Social Science Research Network · doi:10.2139/ssrn.3598331 · Cited by 1
Masumura, Y., & Tago, A. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3906409
Compromise, Voter Punishment in Primaries, and Legislative Gridlock
Anderson, S. E., Butler, D. M., & Harbridge‐Yong, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108768375.007
Voter Punishment Is Rare but Real
Anderson, S. E., Butler, D. M., & Harbridge‐Yong, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108768375.005
Primary Voters as the Source of Punishment
Anderson, S. E., Butler, D. M., & Harbridge‐Yong, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108768375.004
Rejecting Compromise, Getting Gridlock
Anderson, S. E., Butler, D. M., & Harbridge‐Yong, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108768375.001
Trauma Spillover in Social NetworksPreprint
Sadish, D. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/t6b2y
The Deeply Involved Are Different
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.005
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.011
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.008
A House Divided against Itself?
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.002
Subtleties of Partisan Division
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.003
(2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.012
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.006
On the robustness of social-circle surveys: Respondent selection issues, egocentrism, and homophily
Ahlstrom‐Vij, K. (2022) · Electoral Studies · doi:10.1016/j.electstud.2021.102433
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.004
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.010
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.009
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.013
Perceptions of the Most Sacred Duty
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.007
Krupnikov, Y., & Ryan, J. B. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108923323.001
Structuring Negotiations in the Shadow of Primary Voter Punishment
Anderson, S. E., Butler, D. M., & Harbridge‐Yong, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108768375.006
Anderson, S. E., Butler, D. M., & Harbridge‐Yong, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108768375.008
Legislators Reject Half-Loaf Compromises Because They Fear Voter Retribution
Anderson, S. E., Butler, D. M., & Harbridge‐Yong, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108768375.003
Legislators Reject Half-Loaf Compromises
Anderson, S. E., Butler, D. M., & Harbridge‐Yong, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108768375.002
Party Identification and Social Relationships: Exploring the Causal MechanismsPreprint
Ogura, I. (2022) · doi:10.33774/apsa-2022-vgw52
Won’t you be my neighbor? Geography, peer learning, and entrepreneur performance in TogoPreprint
Dimitriadis, S., & Koning, R. M. (2022) · doi:10.31235/osf.io/n867s
Observability Obstructs Motivated Reasoning in AutocracyPreprint
Сироткина, Е. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4050130
Sect, Nation, and Identity after the Fall of Mosul: Evidence from a Natural ExperimentPreprint
Barrie, C. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3219735
Anderson, S. E., Butler, D. M., & Harbridge‐Yong, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108768375.009
Snap elections increase political trust: a quasi-experiment from the UKPreprint
Turnbull-Dugarte, S. J. (2021) · doi:10.31219/osf.io/n3sjh
Bateson, R. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4129550
Post-Instrument Bias in Linear ModelsPreprint
Glynn, A., Rueda, M., & Schuessler, J. (2023) · doi:10.31235/osf.io/axn4t
Sight-Reading Race in Early Modern Drama: Dog Whistles, Signifiers, and the Grammars of Blackness
Chapman, M., & Kelly, J. (2023) · doi:10.54027/fkzb1585
Does One More Year of Rural Preschool Education Have More Dosage Effects?
Chen, Z., Zhang, X., Chen, S., et al. (2023) · Atlantis Highlights in Computer Sciences/Atlantis highlights in computer sciences · doi:10.2991/978-94-6463-040-4_166
Measuring How Much Judges Matter for Case OutcomesPreprint
Copus, R. W., & Hübert, R. (2023) · doi:10.31219/osf.io/8bv6x
Miller, S. M., & Song, M. (2023) · Review of Public Personnel Administration · doi:10.1177/0734371x231162028
Chambers, C. (2023) · Peer Community In Registered Reports · doi:10.24072/pci.rr.100185
Xu, J., Kaneko, S., & Goto, D. (2023) · Research Square · doi:10.21203/rs.3.rs-3262330/v1
Ko, I. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4621637
van den Hoogen, E., de Koster, W., & van der Waal, J. (2023) · European Societies · doi:10.1080/14616696.2023.2275593
Health consequences of exposure to environmental living conditions
Aretz, B. (2023) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.829747624 · Cited by 1
Civil Conflict and Investments: Evidence from a State Bifurcation in IndiaPreprint
Meka, S., Krishnan, K., & Kar, T. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4638968
From Better Monitoring to Better Decisions: Improving Conservation using Community Science
Binley, A. D. (2023) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2023-15796
Actual and Perceived Partisan Bias in Judgments of Political Misinformation as LiesPreprint
Reins, L. M., & Wiegmann, A. (2023) · doi:10.31234/osf.io/357fh
Estrella, G. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4673933
Schuessler, J., Glynn, A., & Rueda, M. (2026) · American Journal of Political Science · doi:10.1111/ajps.70064
Gender, Advice, and the Candidacy Gap in American Politics
Cassidy, J., & Clarke, A. J. (2024) · Journal of Women Politics & Policy · doi:10.1080/1554477x.2024.2294511
Wang, L., & Feldman, L. (2024) · The International Journal of Press/Politics · doi:10.1177/19401612241233543
Reassessing Gendered Reactions to Terrorist Attacks: Slumps or Bumps?
Horiuchi, Y., & Johnson, M. C. (2026) · American Political Science Review · doi:10.1017/s000305542610149x
Cremaschi, S., & Juan Masullo, J. (2024) · doi:10.31219/osf.io/ve8w4
Partisan Politics and Constrained Institutions
Cormier, B. (2024) · doi:10.1093/oso/9780198882732.003.0002
Cormier, B. (2024) · doi:10.1093/oso/9780198882732.003.0005
Cormier, B. (2024) · doi:10.1093/oso/9780198882732.003.0007
Cormier, B. (2024) · doi:10.1093/oso/9780198882732.002.0003
Cormier, B. (2024) · doi:10.1093/oso/9780198882732.003.0004
Cormier, B. (2024) · doi:10.1093/oso/9780198882732.003.0006
Cormier, B. (2024) · doi:10.1093/oso/9780198882732.003.0003
Cormier, B. (2024) · doi:10.1093/oso/9780198882732.003.0001
Status Threat and American Identity in a Multiracial DemocracyPreprint
Versteegen, P. L., & Syropoulos, S. (2024) · doi:10.31234/osf.io/9w8q5
An Invitation to Causal Inference in Environmental PsychologyPreprint
Bulbulia, J. A., & Hine, D. W. (2024) · doi:10.31234/osf.io/tbjx8
Fitzgerald, J., Stroet, P., Weißmüller, K. S., & van Witteloostuijn, A. C. (2024) · Journal of Business Ethics · doi:10.1007/s10551-024-05731-x
Devroe, R., & Wauters, B. (2024) · Electoral Studies · doi:10.1016/j.electstud.2024.102820
Failing to Mobilize: Prosecutors’ Framing Efforts and Anticorruption Campaigns
Vilaça, L. (2024) · Socius Sociological Research for a Dynamic World · doi:10.1177/23780231241255578
Judicial Bias and Ethnic Disparities at the ICTY: Evidence from 30 Years of ProsecutionsPreprint
Hashimoto, B., & Gray, K. W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4860722
Bishop, D., & Hulme, C. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241267904
Does vote buying undermine confidence in ballot secrecy? Theory and experimental evidence
Ascencio, S. J., & Chang, H. I. (2024) · Political Science Research and Methods · doi:10.1017/psrm.2024.54
Traditional Authorities and Strategies in Demands for Self-Determination
Neupert-Wentz, C., & Kelle, F. L. (2024) · International Studies Quarterly · doi:10.1093/isq/sqae134
Wasteful Bluffing in Charitable Giving - An Experiment on Indirect SignallingPreprint
Kaiser, J. P., & Koch, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5019334
French Immersion and Labour Market Earnings
Manuel, N. (2024) · Canadian Public Policy · doi:10.3138/cpp.2023-057
Agnolin, P., Colantone, I., & Stanig, P. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5061054
أثر برامج التربية الإعلامية الرقمية في تعزيز وعي المراهقين بالتزييف المرئي العميق
عبدالحليم, م. م. م. (2025) · Al-Maǧallah Al-Miṣriyyaẗ Li-Buḥūṯ Al-Rāʼī Al-ʿām · doi:10.21608/joa.2025.403543
How affinity with places affects the indirect experience of climate extreme weather events
Keller, E., Marsh, J. E., Richardson, B. H., & Ball, L. J. (2025) · Global Environmental Psychology · doi:10.5964/gep.10749
Kraitzman, A. P., DeMora, S. L., & Albarracín, D. (2025) · Political Communication · doi:10.1080/10584609.2025.2472768
Altruism, Ethnic Identity, and the Limits of Shared Hardship
Bracic, A. (2025) · The Journal of Race Ethnicity and Politics · doi:10.1017/rep.2025.4
Counterfactual Identification Under Monotonicity Constraints
Maiti, A., Plečko, D., & Bareinboim, E. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i25.34888
Chaudhry, S., Dotson, M. R., & Heiss, A. (2025) · doi:10.31235/osf.io/hu6a5_v2
Consequences of Corruption for Political System Support: Evidence From a Brazilian Scandal
Micheli, D. D., & Taylor, W. K. (2025) · Comparative Political Studies · doi:10.1177/00104140251342918
Bird, C. C., Houston, R., & Shieh, M. (2025) · Interest Groups & Advocacy · doi:10.1057/s41309-025-00238-w
“Reasoning” with Rhetoric: On the Style-Evidence Tradeoff in LLM-Generated Counter-Arguments
Verma, P., Jaidka, K., & Churina, S. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35913 · Cited by 7
Observed Correction Increases Accuracy
Bode, L., & Vraga, E. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780197565896.003.0003
Bode, L., & Vraga, E. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780197565896.003.0005
Bode, L., & Vraga, E. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780197565896.002.0006
Bode, L., & Vraga, E. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780197565896.003.0009
Bode, L., & Vraga, E. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780197565896.003.0008
Bode, L., & Vraga, E. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780197565896.003.0007
Bode, L., & Vraga, E. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780197565896.003.0001
Bode, L., & Vraga, E. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780197565896.002.0009
Bode, L., & Vraga, E. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780197565896.003.0004
The Theory of Observed Correction
Bode, L., & Vraga, E. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780197565896.003.0002
Metadata
- Authors
- Montgomery, J. M., Nyhan, B., & Torres, M.
- Affiliations
- Washington University in St. Louis (US)
- Dartmouth College (US)
- Venue
- American Journal of Political Science
- Publisher
- Wiley
- Published
- 2018-03-30
- Volume
- 62
- Issue
- 3
- Pages
- 760-775
- ISSN
- 0092-5853 (ISSN-L)
- 1540-5907
- OpenAlex
- W2794592087
- Cited by
- 800
- References
- 49
- Citations by year
- 2026: 68 · 2025: 128 · 2024: 120 · 2023: 114 · 2022: 105 · 2021: 89 · 2020: 95 · 2019: 48 · 2018: 27
- FWCI
- 53.73
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Political Science (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- causal effect estimation, experimental design, confounding, bias in treatment effect estimates
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
