Comparing Models of Change to Estimate the Mediated Effect in the Pretest–Posttest Control Group Design
Open accessAbstract
Models to assess mediation in the pretest-posttest control group design are understudied in the behavioral sciences even though it is the design of choice for evaluating experimental manipulations. The paper provides analytical comparisons of the four most commonly used models used to estimate the mediated effect in this design: Analysis of Covariance (ANCOVA), difference score, residualized change score, and cross-sectional model. Each of these models are fitted using a Latent Change Score specification and a simulation study assessed bias, Type I error, power, and confidence interval coverage of the four models. All but the ANCOVA model make stringent assumptions about the stability and cross-lagged relations of the mediator and outcome that may not be plausible in real-world applications. When these assumptions do not hold, Type I error and statistical power results suggest that only the ANCOVA model has good performance. The four models are applied to an empirical example.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 49 cited sources checked · 0 supported · 0 flagged
- Findingown data
When the assumptions of the non-ANCOVA models fail, Type I error and power results indicate that only ANCOVA performs well.
“When these assumptions do not hold, Type I error and statistical power results suggest that only the ANCOVA model has good performance.”
Quote in the paperSubstantial claim - Findingown data
In the simulation, the ANCOVA model had higher statistical power than the cross-sectional model except in a few conditions.
“The statistical power for the ANCOVA model was unaffected by the predictors in the simulation study and was higher than the cross-sectional model except for a few conditions.”
Quote in the paperSubstantial claim - Findingown data
In the empirical example, the mediated effect estimate under the residualized change score model was 0.222 (SE=0.032).
“The mediated effect estimate under the ANCOVA model was 0.237 (SE= 0.033) and the mediated effect estimate under the residualized change score model was 0.222 (SE = 0.032).”
Quote in the paperNot a substantial claim - Findingown data
In the empirical example, there was insufficient evidence that the difference score, residualized change score, or cross-sectional model fit the data better than the ANCOVA model.
“There was not sufficient evidence to suggest that the difference score, residualized change score, or the cross-sectional model fit the data better than did the ANCOVA model.”
Quote in the paperSubstantial claim - Findingown data
In the simulation, every model except the cross-sectional model yielded an unbiased estimate of the longitudinal mediated effect.
“In summary, all models produced an unbiased estimate of the longitudinal mediated effect except for the cross-sectional model, as was expected.”
Quote in the paperSubstantial claim
- Findingown data
Power for the residualized change score model was lowest when pretest correlation, M2 cross-lag, and Y2 cross-lag were all 0.50.
“Power to detect the mediated effect with the residualized change score model was lowest when the pretest correlation, M2 cross-lag, and Y2 cross-lag were equal to 0.50 (see Figure 7).”
Quote in the paperNot a substantial claim - Findingown data
For the residualized change score model, there was a two-way interaction of pretest correlation by M2 cross-lag on power.
“There was also a two-way interaction of pretest correlation by M2 cross-lag (ηp2=0.01).”
Quote in the paperNot a substantial claim - Findingown data
For the residualized change score model, Type I error rates were above the top of the robustness interval except when both M2 and Y2 cross-lags were zero.
“The Type 1 error rates for the residualized change score model were above the top-end of the robustness interval except when both the M2 cross-lag and the Y2 cross-lag were equal to 0.00.”
Quote in the paperSubstantial claim - Findingown data
For the residualized change score model, Type I error rates were above the top of the robustness interval when pretest correlation was 0.50 compared with zero.
“The Type 1 error rates were above the top-end of the robustness interval when the pretest correlation was equal to 0.50 compared to when the pretest correlation was equal to zero.”
Quote in the paperSubstantial claim - Findingown data
Power for the residualized change score model was lower when M2 cross-lag or Y2 cross-lag was 0.50 versus zero.
“Power to detect the mediated effect for the residualized change score model was lower when either the M2 cross-lag or the Y2 cross-lag were equal to 0.50 as compared to when they were equal to zero.”
Quote in the paperNot a substantial claim - Findingown data
The cross-sectional model's relative bias was substantial (> .4) for all conditions except when pretest correlation, M2 cross-lag, and Y2 cross-lag were all zero.
“The relative bias of the cross-sectional model was quite substantial for all conditions (relative bias > .4) except when pretest correlation, M2 cross-lag, and Y2 cross-lag were all equal to zero.”
Quote in the paperSubstantial claim - Findingown data
In the empirical example, the cross-sectional mediated effect estimate was larger than the ANCOVA estimate.
“The mediated effect estimate for the cross-sectional model was larger in magnitude than the mediated effect estimate for the ANCOVA model, again supporting the simulation results.”
Quote in the paperSubstantial claim - Findingreasoning
The power to detect the mediated effect for the difference score, residualized change score, and cross-sectional models was affected by the factors outlined in the model comparisons section.
“The power to detect the mediated effect for the difference score, residualized change score, and cross-sectional models were all affected by the factors outlined above in the model comparisons section.”
Quote in the paperSubstantial claim - Findingown data
There were no additional predictors of power for the ANCOVA or cross-sectional models.
“There were no additional predictors of power for the ANCOVA model or for the cross-sectional model (see Figure 5).”
Quote in the paperNot a substantial claim - Findingown data
In the simulation, the ANCOVA model's statistical power was unaffected by the predictors.
“The statistical power for the ANCOVA model was unaffected by the predictors in the simulation study and was higher than the cross-sectional model except for a few conditions.”
Quote in the paperNot a substantial claim - Findingown data
In the empirical example, the mediated effect estimate ranged from 0.181 (SE=0.032) under the difference score model to 0.254 (SE=0.037) under the cross-sectional model.
“The mediated effect estimate ranged from 0.181 (SE= 0.032) under the difference score model to 0.254 (SE = 0.037) under the cross-sectional model.”
Quote in the paperSubstantial claim - Findingown data
There were no significant predictors of relative bias for the ANCOVA, difference score, or residualized change score models.
“There were no significant predictors of relative bias for either the ANCOVA, difference score, or residualized change score model.”
Quote in the paperNot a substantial claim - Findingown data
For the cross-sectional model, there was a three-way interaction of sample size by M2 cross-lag by Y2 cross-lag on Type I error rates.
“there was however a three-way interaction of sample size by M2 cross-lag by Y2 cross-lag (ηp2=0.01) on Type 1 error rates for the cross-sectional score model.”
Quote in the paperNot a substantial claim - Findingown data
For the difference score, residualized change score, and cross-sectional models, Type I error rates increased and exceeded the top of the robustness interval as true a, sample size, M2 cross-lag, and Y2 cross-lag increased.
“There were main effects of true value of a (ηp2≥0.02), sample size (ηp2≥0.03), M2 cross-lag (ηp2=0.02), and Y2 cross-lag (ηp2≥0.01) on Type 1 error rates for the difference score, residualized change score, and the cross-sectional models.”
- Also stated in: § Type 1 Error Rates – b Path Equal To Zero · plain
“As these factors increased, the Type 1 error rates for these models increased and exceeded the top-end of the robustness interval.”
Quote in the paperSubstantial claim - Also stated in: § Type 1 Error Rates – b Path Equal To Zero · plain
- Findingown data
Power for the difference score model was lowest when stability was 0.30, pretest correlation was 0.50, and Y2 cross-lag was 0.50.
“Power was lowest when stability was equal to 0.30, pretest correlation was equal to 0.50, and when the Y2 cross-lag was equal to 0.50 (see Figure 6).”
Quote in the paperNot a substantial claim - Findingown data
In the empirical example, the ANCOVA and residualized change score models produced the most similar mediated effect estimates.
“The mediated effect estimate under the ANCOVA and residualized change score models were the most similar.”
Quote in the paperNot a substantial claim - Findingown data
Power to detect the mediated effect was very similar for the ANCOVA and cross-sectional models.
“The power to detect the mediated effect for both the ANCOVA model and the cross-sectional model were very similar.”
Quote in the paperSubstantial claim - Findingown data
In the simulation, the ANCOVA model was the only model without elevated Type 1 error rates when by2m2 was zero but am2x was greater than zero.
“The ANCOVA model was the only model that did not have elevated Type 1 error rates for the case when by2m2 was zero but am2x was greater than zero.”
Quote in the paperSubstantial claim - Findingown data
In the empirical example, the mediated effect estimates for the difference score and residualized change score models were both smaller than the ANCOVA estimate.
“Additionally, the mediated effect estimates for the difference score and residualized change score model were both smaller in magnitude than the mediated effect estimate for the ANCOVA model.”
Quote in the paperSubstantial claim - Findingown data
In the empirical example, the mediated effect estimate under the ANCOVA model was 0.237 (SE=0.033).
“The mediated effect estimate under the ANCOVA model was 0.237 (SE= 0.033) and the mediated effect estimate under the residualized change score model was 0.222 (SE = 0.032).”
Quote in the paperNot a substantial claim - Findingnone given
Overall, the ANCOVA model performed best for estimating the mediated effect in terms of Type 1 error rates, bias, confidence interval coverage, and power.
“Overall, the ANCOVA model performed the best for estimating the mediated effect in terms of Type 1 error rates, bias, confidence interval coverage, and power.”
Quote in the paperSubstantial claim - Findingown data
In the simulation, the cross-sectional model yielded an unbiased estimate of the longitudinal mediated effect only when the pretest correlation and cross-lagged paths were zero.
“The only condition for which the cross-sectional model produced an unbiased estimate of the longitudinal mediated effect was when the pretest correlation and cross-lagged paths were zero.”
Quote in the paperSubstantial claim - Findingown data
Type I error rates exceeded the robustness interval only when the a path was non-zero and the b path was zero.
“The Type I error rates only exceeded the robustness interval when the a path was non-zero and the b path was zero therefore only these results were presented.”
Quote in the paperSubstantial claim - Findingown data
The ANCOVA model's Type I error rates never exceeded the robustness interval.
“The Type 1 error rates for the ANCOVA model never exceeded the robustness interval so these results were not reported.”
Quote in the paperNot a substantial claim - Findingown data
In the empirical example, all chi-square difference tests comparing the respective models to the ANCOVA model were statistically significant.
“All the chi-square difference tests were statistically significant when comparing the respective models to the ANCOVA model.”
Quote in the paperNot a substantial claim - Findingown data
For the cross-sectional model, there were main effects of true mediated effect, pretest correlation, M2 cross-lag, and Y2 cross-lag on relative bias.
“There were main effects of true value of the Mediated Effect (ηp2≥0.06), pretest correlation (ηp2≥0.02), M2 cross-lag (ηp2≥0.01), and Y2 cross-lag (ηp2≥0.10) on the relative bias for the cross-sectional model.”
Quote in the paperSubstantial claim - Findingown data
For the difference score model, Type I error rates were within the robustness interval when both cross-lags were zero, except when pretest correlation was 0.50.
“The Type 1 error rates for the difference score model were within the robustness interval when both the M2 cross-lag and Y2 cross-lag were equal to zero except for when the pretest correlation was equal to 0.50 (see Table 2).”
Quote in the paperNot a substantial claim - Findingown data
For the difference score model, there were main effects of stability and Y2 cross-lag on power.
“In addition to the main effects of sample size and true value of the mediated effect, there were main effects of stability (ηp2=0.03) and Y2 cross-lag (ηp2=0.01) on the power to detect the mediated effect with the difference score model.”
Quote in the paperNot a substantial claim - Findingreasoning
When there was a pretest correlation and both cross-lagged paths were non-zero, the residualized change score model had much lower power than the ANCOVA model.
“When there was a pretest correlation and both cross-lagged paths were non-zero, the power of the residualized change score model was much lower than that of the ANCOVA model.”
Quote in the paperSubstantial claim - Premisecitation
Using pretest scores from two waves adjusts for time-invariant confounders in a randomized experiment.
“using the pretest scores from two-waves of data adjusts for time invariant confounders in a randomized experiment”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites James, 1980
Not checked in the free run.
- cites Judd & Kenny, 1981
Not checked in the free run.
- Interpretationnone given
All models except ANCOVA impose stringent assumptions about stability and cross-lagged relations of the mediator and outcome that may not be plausible in real-world applications.
“All but the ANCOVA model make stringent assumptions about the stability and cross-lagged relations of the mediator and outcome that may not be plausible in real-world applications.”
Quote in the paperSubstantial claim - Interpretationcitation
This aggregate adjustment in the residualized change score model may lead to erroneous inference about group differences in some cases.
“which may lead to erroneous inference regarding group differences in some cases”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Maxwell, Delaney, & Manheimer, 1985
Not checked in the free run.
- Premisecitation
In the pretest-posttest design with a mediating variable, pretest measures of M and Y can be used to estimate the mediated effect of X on Y through M at posttest.
“When a mediating variable (M) is involved, pretest measures of M and of Y can be used in the estimation of the mediated effect of X on Y through M at posttest.”
Quote in the paperSubstantial claim - Premisenone given
Prior research had not demonstrated Type 1 error rate, confidence interval coverage, and power for cross-sectional estimates when the true longitudinal model corresponds to a pretest-posttest control group design.
“However, what has not been demonstrated in this prior research, was the Type 1 error rate, confidence interval coverage, and power for cross-sectional estimates when the true longitudinal model corresponds to a pretest-posttest control group design.”
Quote in the paperSubstantial claim - Premisecitation
The cross-sectional estimate of the mediated effect is biased when the true underlying model is longitudinal.
“The cross-sectional estimate of the mediated effect when there is a true underlying longitudinal model is biased and this bias has been shown in Maxwell and Cole (2007) and Maxwell et al (2011).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Maxwell and Cole (2007)
Not checked in the free run.
- cites Maxwell et al (2011)
Not checked in the free run.
- Premisecitation
Four statistical models can be used to investigate the mediated effect: cross-sectional, difference score, residualized change score, and ANCOVA.
“Four statistical models can be used to investigate the mediated effect: the cross – sectional model, the difference score model (Jansen, et al., 2013; MacKinnon et al., 1991), the residualized change score model (Miller, Trost, & Brown, 2002; Reid & Aiken, 2013), the Analysis of Covariance (ANCOVA) model (Jang et al., 2012; Schmiege, Broaddus, Levin, & Bryan, 2009).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Jansen, et al., 2013
Not checked in the free run.
- cites MacKinnon et al., 1991
Not checked in the free run.
- cites Miller, Trost, & Brown, 2002
Not checked in the free run.
- Interpretationreasoning
In the residualized change score model, when the covariance between residualized change in M and the residual from predicting residualized change in Y is zero, its b path equals the ANCOVA b path.
“When the Cov(ResΔM,ResΔY^−ResΔY) term is equal to zero, the b path in the residualized change score model (bR)will equal the b path in the ANCOVA model (bY2M2).”
Quote in the paperSubstantial claim- The authors state this as an assumption: The data-generating model is the ANCOVA model.
- Premisereasoning
The by2m2 path in the ANCOVA model is a function of the pretest correlation, cross-lagged paths, and the effects of the experimental manipulation on the mediator (am2x) and outcome (cy2x).
“The by2m2 path in the ANCOVA model is a function of the pretest correlation, cross-lagged paths, and the effects of the experimental manipulation on both the mediator (am2x) and the outcome (cy2x).”
Quote in the paperSubstantial claim - Premisecitation
In mediation for design, mediating variables are typically chosen before the experiment because they are related to the outcome.
“This framework for assessing mediation is referred to as mediation for design and within this framework mediating variables are typically chosen prior to the experiment because these mediating variables are related to the outcome (i.e., they are correlated; MacKinnon, 2008).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites MacKinnon, 2008
Not checked in the free run.
- Interpretationnone given
The four models provide different estimates of the mediated effect.
“In summary, each of these four models provide a different estimate of the mediated effect.”
Quote in the paperSubstantial claim - Interpretationreasoning
The residualized change score and difference score models cannot be compared with a chi-square difference test because they have the same degrees of freedom.
“The residualized change score and the difference score models cannot be compared based on a chi-square difference test because they have the same degrees of freedom.”
Quote in the paperNot a substantial claim - Interpretationown data
The larger cross-sectional empirical estimate supports the simulation results.
“The mediated effect estimate for the cross-sectional model was larger in magnitude than the mediated effect estimate for the ANCOVA model, again supporting the simulation results.”
Quote in the paperSubstantial claim - Interpretationown data
The smaller empirical estimates for the difference score and residualized change score models support the simulation result that these models may be underpowered relative to ANCOVA when pretest covariance and non-zero cross-lagged paths combine.
“This supports the simulation results that suggests these models may be underpowered compared to the ANCOVA model when there is some combination of pretest covariance and non-zero cross-lagged paths.”
Quote in the paperSubstantial claim - Interpretationreasoning
When the assumptions are violated, non-causal associations between X and Y2 adjusting for M2 make the mediated effect a function of both a causal effect of X on Y2 through M2 and non-causal associations.
“That is, there will be non-causal associations between X and Y2 adjusting for M2 rendering the mediated effect a function of some causal effect of X on Y2 through M2 and some non-causal associations.”
Quote in the paperSubstantial claim - Interpretationreasoning
It is very unlikely that the by2m2 path estimated with the difference score, residualized change score, or cross-sectional models will be zero when any combination of pretest correlation, cross-lagged paths, and experimental effects are non-zero.
“Therefore, it is very unlikely that the by2m2 path estimated with the difference score, residualized change score, or cross-sectional models will be zero when any of combination of these paths are non-zero.”
Quote in the paperSubstantial claim - Premisereasoning
With two waves of mediator and outcome data, multiple models can be fitted to assess the mediated effect.
“With two waves of data measured for both the mediating and outcome variable a variety of models can be fitted to the data to assess the mediated effect.”
Quote in the paperSubstantial claim - Interpretationcitation
If the true generating model is longitudinal with stable individual differences, the cross-sectional model yields biased estimation of the mediated effect.
“This also implies that if the true generating model is longitudinal in nature (e.g., there are stable individual differences across time) the cross-sectional model will result in biased estimation of the mediated effect (Cole & Maxwell, 2003; Maxwell & Cole, 2007; Maxwell et al., 2011).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Cole & Maxwell, 2003
Not checked in the free run.
- cites Maxwell & Cole, 2007
Not checked in the free run.
- cites Maxwell et al., 2011
Not checked in the free run.
- Interpretationreasoning
The unstandardized M2-to-Y2 relation adjusted for X, called the b path, differs across the four change models depending on several factors.
“The unstandardized relation of M2 to Y2 adjusted for X varies across the four different models depending on a number of factors and will be referred to as the b path throughout this paper.”
Quote in the paperSubstantial claim - Interpretationreasoning
No analytical comparisons are needed for the a path component of the mediated effect.
“Therefore no analytical comparisons are needed for this component of the mediated effect.”
Quote in the paperSubstantial claim - Interpretationreasoning
If the ANCOVA model generated the data, the difference score model's predicted change in Y does not account for all variance in change in Y, leaving a residual.
“assuming the data-generating model is the ANCOVA model, the predicted ΔY using the difference score model will not explain all the variance in ΔY and therefore there will be a residual of this prediction.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The data-generating model is the ANCOVA model.
- Interpretationreasoning
In the difference score model, when the covariance between change in M and the residual from predicting change in Y is zero, its b path equals the ANCOVA b path.
“When the Cov(ΔM,ΔY^−ΔY) term is equal to zero, the b path in the difference score model (bΔ)will equal the b path in the ANCOVA model (bY2M2).”
Quote in the paperSubstantial claim- The authors state this as an assumption: The data-generating model is the ANCOVA model.
- Interpretationcitation
The difference in the b path across models can be interpreted as an unmeasured confounder problem by using prior results.
“The difference in the b path across models can be thought of as an unmeasured confounder problem by applying results from Clark (2005) and Hanushek and Jackson (1977).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Clark (2005)
Not checked in the free run.
- cites Hanushek and Jackson (1977)
Not checked in the free run.
- Interpretationreasoning
If the causal mediation assumptions are violated or not plausible, the mediated effect estimate cannot be interpreted as a causal effect.
“If any of these assumptions are violated or not plausible, however, the mediated effect estimate will not be interpretable as a causal effect.”
Quote in the paperSubstantial claim - Interpretationreasoning
In the cross-sectional model, when the covariance between M2 and the residual from predicting Y2 is zero, its b path equals the ANCOVA b path.
“Therefore, when the Cov(M2,Y^2−Y2) term is equal to zero, the b path in the cross-sectional model (b)will equal the b path in the ANCOVA model (bY2M2).”
Quote in the paperSubstantial claim- The authors state this as an assumption: The data-generating model is the ANCOVA model.
- Interpretationreasoning
The unstandardized relation of X to M2 is equivalent in expectation across all four models because the covariances of X with M2, Δm, and Rm are equivalent when the covariances of X with M1 and Y1 are zero, assuming successful randomization.
“The unstandardized relation of X to M2 in the pretest-posttest control group design is equivalent in expectation across all models because the covariances of X and M2, X and Δm, and X and Rm are equivalent when the covariance between X and M1 and X and Y1 are zero (i.e., assuming successful randomization of units to levels of X).”
Quote in the paperSubstantial claim- The authors state this as an assumption: Randomization successfully assigns units to control and treatment groups so that groups do not differ systematically at pretest.
- Interpretationreasoning
The b path component-the relation of the mediator to the outcome adjusted for X-is the component of the mediated effect that differs across the four models.
“The following section provides analytical derivations of the differences in the b path component (i.e., relation of the mediator to the outcome adjusted for X) of the mediated effect estimate as this is the component of the mediated effect that differs across the four models discussed earlier.”
Quote in the paperSubstantial claim - Interpretationnone given
The four models differ in their assumptions about how pretest measures relate to posttest measures.
“The four models for two waves of data for estimating the mediated effect adjusted for pretest measures differ from one another in the assumptions they make regarding the relation of pretest measures to posttest measures.”
Quote in the paperSubstantial claim - Interpretationreasoning
Differences between non-ANCOVA and ANCOVA mediated effect estimates depend on pretest correlation, stabilities, and cross-lagged paths.
“Generally speaking, the mediated effect estimates for the non-ANCOVA (i.e., difference score, residualized change score, and cross-sectional) models will differ from the mediated effect estimate of the ANCOVA model as a function of the pretest correlation, stabilities, and cross-lagged paths.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The data-generating model is the ANCOVA model.
- Interpretationreasoning
The association between change in M and that residual captures the difference in the b path between the difference score and ANCOVA models.
“Any relation of the predictor ΔM with this residual will capture the difference in the b path across the two models”
Quote in the paperSubstantial claim- The authors state this as an assumption: The data-generating model is the ANCOVA model.
- Interpretationreasoning
Adjusting for pretest scores adjusts for time-invariant confounders of the pretest scores.
“One of the benefits of adjusting for pretest scores is that when an adjustment on pretest scores is done, any time invariant confounders of the pretest scores will be adjusted for as a consequence of adjusting for the pretest scores themselves.”
Quote in the paperSubstantial claim - Interpretationreasoning
The mediated effect estimates from difference score, residualized change score, and cross-sectional models generally do not equal the ANCOVA mediated effect estimate because their b path components differ.
“In summary, the mediated effect estimate for difference score, residualized change score, and cross-sectional models will not generally equal the mediated effect estimate for the ANCOVA model because of differences in the b path component of the mediated effect.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The data-generating model is the ANCOVA model.
- Premisenone given
Assuming random assignment to two groups and pretest/posttest measures, all four two-wave models can be fitted with the LCS specification.
“Assuming random assignment to two groups and the two waves represent pretest measure (M1, Y1) and posttest measures (M2, Y2), all four two wave models previously mentioned can be fitted with the LCS specification.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Random assignment to two groups is assumed.
- Interpretationnone given
If the pretest covariance or cross-lagged paths were large and negative, the opposite pattern of empirical mediated effect estimates could occur.
“It is important to note, however, if the pretest covariance or cross-lagged paths were large in magnitude and negative, the opposite pattern of results could occur when comparing the difference score, residualized change score, and cross-sectional mediated effect estimates to the ANCOVA mediated effect estimate.”
Quote in the paperNot a substantial claim - Premisecitation
Cross-sectional estimates of mediated effects are often biased because they do not allow mediating variables to exert influence on dependent variables over time.
“Cross-sectional estimates of mediated effects will often be biased because they do not allow for mediating variables (M) to exert their influence on dependent variables (Y), which presumably occurs over a specific period of time.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Gollob & Reichardt, 1991
Not checked in the free run.
- cites Maxwell & Cole, 2007
Not checked in the free run.
- cites Maxwell, et al., 2011
Not checked in the free run.
- Interpretationown data
This study confirms previous literature's general finding that the cross-sectional mediated effect is biased as an estimate of a longitudinal mediated effect.
“This study confirms the general findings of previous literature regarding the bias of the cross-sectional mediated effect as an estimate of a longitudinal mediated effect.”
Quote in the paperSubstantial claim - Interpretationreasoning
These difference score assumptions are strong because it is unlikely that stabilities are exactly one and cross-lagged paths are exactly zero.
“These are strong assumptions to make because it is unlikely that the stabilities are exactly equal to one and the cross-lagged paths are exactly equal to zero.”
Quote in the paperSubstantial claim - Interpretationreasoning
With pretest measures of mediator and outcome that may not be perfectly reliable, the impact of measurement error on the mediated effect may be more complicated than in the single mediator model.
“With the addition of pretest measures of the mediator and outcome that may not be measured perfectly reliably, the impact of measurement error on the estimate of the mediated effect may be more complicated than in the single mediator model.”
Quote in the paperNot a substantial claim - Interpretationnone given
The pretest-posttest control design improves on cross-sectional mediation analysis by generating information on the relation of a randomized experiment on the change in M and the change in Y.
“The pretest-posttest control design improves on cross-sectional mediation analysis by generating information on the relation of a randomized experiment on the change in M and the change in Y.”
Quote in the paperSubstantial claim - Interpretationreasoning
The increased power of the cross-sectional model comes at the cost of a biased estimate of the mediated effect.
“The increased power of the cross-sectional model comes at the cost of a biased estimate of the mediated effect.”
Quote in the paperSubstantial claim - Premisereasoning
As the time interval varies, estimates of mediated effects vary because they depend on the time interval during which they are assessed.
“As time interval varies, so do estimates of mediated effects because estimates of mediated effects depend on the time interval during which they are assessed.”
Quote in the paperSubstantial claim - Interpretationreasoning
When researchers use the cross-sectional, difference score, or residualized change score model, they make assumptions about the stability of the mediator and outcome and about cross-lagged paths.
“Given the findings of this study, when researchers use the cross-sectional model, the difference score model (Jansen et al., 2012; MacKinnon et al., 1991), or the residualized change score model (Miller, et al., 2002; Reid & Aiken, 2013) they are making assumptions about the conditions that need to be met regarding the stability of the mediator and outcome and any cross-lagged paths.”
Quote in the paperSubstantial claim - Interpretationreasoning
When the constraints these models place on stability and cross-lagged paths are unrealistic, statistical properties such as Type 1 error or model fit are affected.
“When these constraints are unrealistic, for example, there are cross-lagged paths in the population but these are not modeled in the difference score model, statistical properties of the model will be affected (e.g., Type 1 error or model fit).”
Quote in the paperSubstantial claim - Interpretationreasoning
The mediator's stability will not equal the simple regression of posttest on pretest unless the cross-lagged path from pretest outcome to posttest mediator is zero and mediator and outcome are not related at pretest.
“The stability of the mediator will not equal the simple regression of the posttest score on the pretest score unless the cross-lagged path from pretest outcome to posttest mediator is zero and the mediator and outcome are not related at pretest.”
Quote in the paperSubstantial claim - Interpretationreasoning
The outcome's stability will not equal the simple regression of posttest on pretest unless both cross-lagged paths are zero and mediator and outcome are not related at pretest, which is unlikely.
“The stability of the outcome will not equal the simple regression of the posttest score on the pretest score unless both the cross-lagged paths are zero and the mediator and outcome are not related at pretest which we already established as being unlikely.”
Quote in the paperSubstantial claim - Interpretationreasoning
This combination of conditions leads to a smaller mediated effect estimate than when either cross-lagged path is present separately.
“This particular combination of conditions leads to a smaller estimate of the mediated effect than when either one of the cross-lagged paths are present separately (See Equations 21 – 23).”
Quote in the paperNot a substantial claim
- May combine two claims. 54
- Classified as probably not a substantial claim. 23
May combine two claims.
Findingown dataPower was greater than 0.80 for each sample size for medium to large true mediated effects and greater than 0.80 for all true effects when N=500.
“Power to detect the mediated effect was greater than 0.80 for each sample size for medium to large true values of the mediated effect and was greater than 0.80 for all true values of the mediated effect when the sample size was equal to 500.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
FindingcitationThe distribution-of-product method results were reported because they perform better than normal theory and similarly to percentile bootstrap for detecting the mediated effect in this study and prior research.
“The distribution of a product method results were reported because they perform better than normal theory results and they perform similarly to the percentile bootstrap results when detecting the mediated effect in this study and in prior research (MacKinnon, Lockwood, & Williams, 2004).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites MacKinnon, Lockwood, & Williams, 2004
Not checked in the free run.
May combine two claims.
FindingreasoningWhen the assumptions of the difference score, residualized change score, and cross-sectional models did not hold, their power to detect the mediated effect decreased, whereas ANCOVA and cross-sectional power did not decrease as a function of these factors.
“When the assumptions of these models (i.e., assumptions about stabilities and cross-lagged paths) did not hold, the power to detect the mediated effect decreased whereas the power to detect the mediated effect with the ANCOVA and cross-sectional models did not decrease as a function of these factors.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone givenThe LCS ANCOVA model has fit identical to the traditional ANCOVA model, being saturated with zero degrees of freedom.
“The fit of this model will be identical to the fit of the traditional ANCOVA model1 (i.e., saturated, 0 dfs).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataFor the residualized change score model, there were main effects of M2 cross-lag and Y2 cross-lag on power.
“For the residualized change score model, in addition to the main effects of sample size and true value of the mediated effect there were main effects of M2 cross-lag (ηp2=0.02) and Y2 cross-lag (ηp2=0.02).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe Type I error rate pattern for the cross-sectional model was similar to the difference score model but generally higher.
“The pattern of results for the Type 1 error rates for the cross-sectional model were similar to the pattern of results for the difference score model but with generally higher values of Type 1 errors.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor the cross-sectional model, there was a main effect of stability and two-way interactions of true mediated effect with stability and with Y2 cross-lag on relative bias.
“There was an additional main effect of stability (ηp2=0.04) on the relative bias of the cross-sectional model, a two-way interaction of true value of the mediated effect by stability (ηp2=0.01) and a two-way interaction of true value of the mediated effect by Y2 cross-lag (ηp2=0.02) (see Figures 3 – 4).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn the empirical example, the residualized change score model had the best relative fit indices (CFI=0.956, RMSEA=0.087), though fit could improve.
“The residualized change score model performed the best in terms of the relative fit indices (CFI = 0.956, RMSEA = 0.087) although there is room for improvement.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenOther bias measures, absolute bias and standardized bias, led to similar conclusions as the reported relative bias results.
“Other measures of bias, absolute bias and standardized bias, led to similar conclusions as reported below.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor every model, there was a significant true mediated effect by sample size interaction and main effects of sample size and true mediated effect on power.
“For every model there was a significant true value of the mediated effect by sample size two-way interaction (ηp2≥0.01for all models) and main effects of sample size (ηp2≥0.01) and true value of the mediated effect (ηp2≥0.01) on the power to detect the mediated effect.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor the difference score model, Type I error rates exceeded the top of the robustness interval when either M2 cross-lag or Y2 cross-lag was 0.50, and were highest when both were 0.50.
“There was a significant three-way interaction of pretest correlation by M2 cross-lag by Y2 cross-lag (ηp2=0.01) on Type 1 error rate for the difference score model such that Type 1 error rates exceeded the top-end of the robustness interval when either the M2 cross-lag was equal to 0.50 or the Y2 cross-lag was equal to 0.50 but was the highest when both the M2 cross-lag and Y2 cross-lag were equal”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataPower for the difference score model was lower when stability was 0.30 versus 0.70 or when Y2 cross-lag was 0.50 versus zero.
“Power for the difference score model was lower when either stability was equal to 0.30 compared to 0.70 or Y2 cross-lag was equal to 0.50 compared to zero.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe confidence interval coverage results for the cross-sectional model were nearly identical to its relative bias results.
“Because the confidence interval coverage results for the cross-sectional model were nearly identical to the relative bias results for the crosssectional model, only the relative bias results are presented but the coverage results are available upon request.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor the cross-sectional model, there were main effects of pretest correlation and stability on Type I error rates.
“There were additional main effects of pretest correlation (ηp2=0.01) and stability (ηp2=0.01) on the Type 1 error rates for the cross-sectional model.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataWhen sample size was 50, 100, or 200, the cross-sectional model had more power than the ANCOVA model for small to medium true mediated effects; this difference disappeared at N=500.
“When sample size was 50, 100, or 200, the cross-sectional model had more power to detect the mediated effect than the ANCOVA model for small to medium true values of the mediated effect. This difference went away when the sample size was equal to 500.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataFor the difference score model, there were three-way interactions of pretest correlation by stability by Y2 cross-lag and true mediated effect by stability by Y2 cross-lag on power.
“There was also a three-way interaction of pretest correlation by stability by Y2 cross-lag (ηp2=0.01) and a three-way interaction of true value of the mediated effect by stability by Y2 cross-lag (ηp2=0.01) for the power of the difference score model.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor the cross-sectional model, relative bias was outside the boundaries (≤−0.10 or ≥0.10) as the true mediated effect decreased and when pretest correlation, M2 cross-lag, or Y2 cross-lag was 0.50.
“Relative bias was outside the boundaries (≤ −0.10 or ≥ 0.10) for the cross-sectional models as the true value of the mediated effect decreased and when either the pretest correlation, M2 cross-lag, or the Y2 cross-lag was equal to 0.50.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe residualized change score model had elevated Type 1 error rates, highest when the pretest correlation and both cross-lagged paths were present, though these conditions may not happen often in practice.
“The residualized change score model had elevated Type 1 error rates which were highest when the pretest correlation and both cross-lagged paths were present (See Equations 21 – 23) although these conditions may not happen often in practice.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe cross-sectional model performs worst in terms of Type 1 error and bias.
“First, the cross-sectional model performs the worst in terms to Type 1 error and bias.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe cross-sectional estimate of the mediated effect is unbiased under some conditions, specifically when there are no cross-lagged relations and when the stabilities are zero.
“The implications of the findings of this study is that there are some conditions for which the cross-sectional estimate of the mediated effect is unbiased (i.e., when there are no cross-lagged relations and when the stabilities are zero, see Equations 24 – 26).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe residualized change score model was unbiased and had similar power to the ANCOVA model with one exception.
“Third, the residualized change score model was unbiased and had similar power as the ANCOVA model with one exception.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe difference score model was unbiased and had lower statistical power than the ANCOVA model when the stabilities were low.
“Second, the difference score model was unbiased and had lower statistical power than the ANCOVA model when the stabilities were low.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Premisenone givenResearchers use each of the four models to address longitudinal mediation questions even though the models address different questions about change.
“Although each of these models address different questions about change from pretest to posttest, researchers use each of them to address questions regarding longitudinal mediation.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationThe pretest-posttest control group design allows nuisance variation in the outcome to be accounted for, yielding more powerful tests, narrower confidence intervals, and greater internal validity.
“This design allows researchers to take into account nuisance variation in the outcome variable subsequently leading to more powerful inferential test statistics, narrower and more precise confidence intervals, and increased internal validity”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Maxwell & Delaney, 2004
Not checked in the free run.
- cites Shadish et al., 2002
Not checked in the free run.
Classified as probably not a substantial claim.
InterpretationreasoningRandomization of X helps ensure equivalent balance across all measured and unmeasured variables, in expectation, that may affect X and M2.
“the randomization of X will help to ensure an equivalent balance across all measured and unmeasured variables, in expectation, that may theoretically affect X and M2.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationThe pretest-posttest design with a mediating variable can estimate a longitudinal mediated effect with only two waves and is sometimes called a half-longitudinal mediation model.
“The pretest-posttest design with a mediating variable can be used to estimate a longitudinal mediated effect when only two waves (e.g., pretest and posttest) of data are collected and is sometimes referred to as a half-longitudinal mediation model”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Cole & Maxwell, 2003
Not checked in the free run.
May combine two claims.
InterpretationreasoningBecause X is an experimental manipulation, the no-unmeasured-confounders assumptions for X-Y and X-M hold in the pretest-posttest control group design.
“Because X is an experimental manipulation, assumptions one and three outlined by VanderWeele and Vansteelandt (2009) will hold true in the pretest- posttest control group design.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites VanderWeele and Vansteelandt (2009)
Not checked in the free run.
- The authors state this as an assumption: Randomization successfully assigns units to control and treatment groups so that groups do not differ systematically at pretest.
Classified as probably not a substantial claim.
PremisecitationMaxwell and Cole (2007) and Maxwell et al. (2011) noted that research on the cross-sectional model's performance for estimating mediated effects in a longitudinal model with an experimental X is lacking.
“Maxwell and Cole (2007) and Maxwell et al (2011) noted the lack of research on the performance of the cross-sectional model for estimating mediated effects in a longitudinal model where X represents an experimental manipulation.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Maxwell and Cole (2007)
Not checked in the free run.
- cites Maxwell et al (2011)
Not checked in the free run.
May combine two claims.
PremisecitationThe difference score model is unconditional on pretest scores and assumes that the unstandardized pretest-posttest regression coefficient (stability) equals 1.0.
“The difference score model is unconditional on pretest scores (Cronbach & Furby, 1970; Dwyer, 1983; McArdle, 2009) and assumes that the unstandardized regression coefficient between the prestest and posttest measure (stability) is 1.0”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Cronbach & Furby, 1970
Not checked in the free run.
- cites Dwyer, 1983
Not checked in the free run.
- cites McArdle, 2009
Not checked in the free run.
Classified as probably not a substantial claim.
PremisecitationPrior literature has described limitations of cross-sectional mediation analysis for assessing longitudinal mediation.
“The limitations of cross-sectional mediation analysis for assessing longitudinal mediation have been described in several articles”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Cheong, MacKinnon, & Khoo, 2003
Not checked in the free run.
- cites Cole & Maxwell, 2003
Not checked in the free run.
- cites Fritz, 2014
Not checked in the free run.
Classified as probably not a substantial claim.
InterpretationreasoningThe same balance from randomizing X also holds for the relation between X and Y2 when M2 is not adjusted for.
“This also holds for the relation between X and Y2 not adjusted for M2.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningThe longitudinal mediated effect in this design is the effect of X on posttest outcome through its effect on M2, and the M2-Y2 relation can be considered causal if there is no time-varying confounding.
“we argue that the longitudinal mediated effect in this design is the mediated effect of X on posttest outcome through its effect on w for the M2 – Y2 relation to be free of any time-invariant third variable effects on the mediator outcome relation and can be considered a causal effect assuming no time-varying confounding”
Quote in the paperSubstantial claim- The authors state this as an assumption: X as an experimental manipulation marks the beginning of the mediational chain for all units.
- The authors state this as an assumption: There is no time-varying confounding of the M2–Y2 relation.
Probably not a substantial claim: may not pass S4 (based).
InterpretationreasoningThe benefits of adjusting for pretest scores in longitudinal experimental designs can be extended to designs that include a mediating variable.
“The benefits of adjusting for pretest scores in longitudinal experimental designs can be extended to designs that include a mediating variable.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisereasoningThe cross-sectional model is the simplest because it ignores pretest measures of the mediator and outcome and does not address change over time.
“The cross-sectional model is the simplest model because it does not take into account the pretest measures of the mediator and dependent variable and therefore does not address a question of change across time.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationVanderWeele and Vansteelandt (2009) describe four assumptions necessary for identifying a mediated effect: no unmeasured confounders of X-Y, M-Y, X-M, and no measured or unmeasured confounders of M-Y affected by treatment.
“VanderWeele and Vansteelandt (2009) describe the four assumptions necessary for identification of a mediated effect: No unmeasured confounders of the relation between X and Y.No unmeasured confounders of the relation between M and Y.No unmeasured confounders of the relation between X and M.No measured or unmeasured confounders of M and Y that have been affected by treatment.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites VanderWeele and Vansteelandt (2009)
Not checked in the free run.
May combine two claims.
PremisecitationLongitudinal data are needed to accurately assess mediated effects because mediation is a causal sequence over time.
“Given the temporal nature of the causal sequence, longitudinal data (i.e., data collected over time) is needed to accurately assess mediated effects”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Cole & Maxwell, 2003
Not checked in the free run.
- cites Gollob & Reichardt, 1991
Not checked in the free run.
- cites MacKinnon, 1994, 2008
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe parameters of the ANCOVA model can be estimated simultaneously via path analysis.
“Alternatively, the parameters of the ANCOVA model can be estimated simultaneously via path analysis”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites MacKinnon, 1994, 2008
Not checked in the free run.
- cites MacKinnon, et al., 2001
Not checked in the free run.
May combine two claims.
PremisecitationMediation models are inherently longitudinal, and time should be considered when assessing mediated effects.
“mediation models are inherently longitudinal and researchers should take time into account when assessing mediated effects”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Cheong, et al., 2003
Not checked in the free run.
- cites Cole & Maxwell, 2003
Not checked in the free run.
- cites Fritz, 2014
Not checked in the free run.
May combine two claims.
PremisecitationIn the residualized change score model, no treatment group variable is included in the regression of posttest on pretest, so posttest scores for both groups are adjusted using aggregate pretest scores.
“No treatment group variable is included in the regression of posttest scores on pretest scores which means posttest scores for units in both treatment groups are adjusted for pretest scores based on an aggregate of pretest scores across both treatment groups”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Cronbach & Furby, 1970
Not checked in the free run.
- cites Kisbu-Sakarya et al., 2013
Not checked in the free run.
May combine two claims.
InterpretationreasoningIn the empirical example, the residualized change score model's mediated effect estimate was most similar to the ANCOVA estimate, consistent with similar estimated stabilities and small cross-lagged relations.
“The residualized change score model had the most similar estimate of the mediated effect as the ANCOVA model which is supported by the fact the estimated stabilities across these models were similar and the cross-lagged relations were not large in magnitude.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningThe cross-sectional model's power was as high as or higher than ANCOVA in some conditions because the cross-sectional estimate was positively biased across many conditions.
“Additionally, the reason the power to detect the mediated effect with the cross-sectional model was as high as or higher than the power to detect the mediated effect with the ANCOVA model in some conditions was that the cross-sectional model estimate was positively biased across many conditions.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningUnder mediation for design, it is unlikely that the pretest correlation between mediator and dependent variable is zero or that the posttest b path is zero.
“Using this framework, it is unlikely either the relation between the mediating variable and the dependent variable at pretest (i.e., the pretest correlation) is equal to zero, or the relation between the mediating variable and the dependent variable at posttest (i.e., the b path) is equal to zero.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationcitationSensitivity analyses can assess the potential effects of unmeasured confounders of the M2-Y2 relation and the potential effects of a pretest correlation, cross-lagged paths, or varying stability when only cross-sectional data are available.
“Sensitivity analyses are an option for assessing both the potential effects of unmeasured confounders of the M2 – Y2 relation and the potential effects of a pretest correlation, cross-lagged paths, or varying stability if only cross-sectional data are available.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationJang, Kim, and Reeve (2012) stated that procedures for testing mediation in multi-wave longitudinal designs had not yet been developed.
“Jang, Kim, and Reeve (2012) specifically mentioned this lack of methodological information for estimating multi-wave mediation effects “…described well-known procedures to test for mediation with cross-sectional research designs, similar procedures to test for mediation with multi-wave longitudinal research designs have not yet been developed” (p. 1181).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Jang, Kim, and Reeve (2012)
Not checked in the free run.
May combine two claims.
InterpretationreasoningThe residualized change score b path differs from the ANCOVA b path because the simple regressions of M2 on M1 and Y2 on Y1 are not equivalent to ANCOVA stabilities and cross-lagged paths are non-zero.
“The b path from the residualized change score model differs from the b path from the ANCOVA model because the simple regression relating M2 to M1 and Y2 to Y1 are not equivalent to the stabilities estimated in the ANCOVA model and the cross-lagged paths are non-zero.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The data-generating model is the ANCOVA model.
May combine two claims.
Interpretationown dataThe difference score, residualized change score, and cross-sectional models made strict assumptions about stabilities and cross-lagged relations that the empirical data did not support.
“That is, each of these models made strict assumptions about stabilities and cross-lagged relations of the pretest-posttest model that were not supported by the data.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisereasoningThe mediated effect can equal zero in three cases: when a is zero, when b is zero, or when both a and b are zero.
“First, the mediated effect can equal zero if the a path is equal to zero. Second, the mediated effect can equal zero if the b path is equal to zero. Third, the mediated effect can equal zero if both the a and the b path are equal to zero.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationThe ANCOVA model assumes homogeneous within-group regression coefficients, no interaction between the covariate and treatment group, and that the covariate is measured without error.
“The ANCOVA model assumes that within group regression coefficients are homogenous, there is no interaction of the covariate (e.g., pretest scores) and the treatment group, and that the covariate is measured without error”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Maxwell & Delaney, 2004
Not checked in the free run.
Classified as probably not a substantial claim.
InterpretationreasoningThe residualized change score and difference score models represent change in different ways.
“These two models represent change in different ways.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningThe cross-sectional b path differs from the ANCOVA b path because stabilities are non-zero and cross-lagged paths are non-zero.
“The b path from the cross-sectional model differs from the b path from the ANCOVA model because the stabilities are non-zero and the cross-lagged paths are non-zero.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The data-generating model is the ANCOVA model.
Classified as probably not a substantial claim.
Premisenone givenThe mediated effect is tested by taking the product of two regression coefficients.
“The mediated effect in this article is tested by taking the product of two regression coefficients.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
InterpretationreasoningIf there are time-invariant covariates such as genetic background, treatment cannot affect them, making Assumption 4 plausible.
“Subsequently, if there are any time invariant covariates (i.e., stable individual differences) such as genetic background, the treatment will not be able to affect these time invariant covariates thus making Assumption 4 plausible.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
InterpretationreasoningThe cross-sectional model implies no stability of scores across time.
“The cross-sectional model implies that there is no stability of scores across time.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningThe pretest-posttest control group design is particularly well suited for estimating causal mediated effects because randomization satisfies Assumptions 1 and 3 and it can potentially satisfy Assumptions 2 and 4 if there are no time-varying covariates.
“The pretest-posttest control group design is particularly well suited for estimating causal mediated effects because it will satisfy Assumptions 1 and 3 because X is randomized and potentially satisfy Assumption 2 and 4 assuming there are no time varying covariates.”
Quote in the paperSubstantial claim- The authors state this as an assumption: There are no time-varying covariates.
Classified as probably not a substantial claim.
PremisecitationThe latent change score specification is an SEM approach that can represent simple and dynamic longitudinal change with manifest or latent measures of a time-dependent outcome.
“The latent change score (LCS) specification is a Structural Equation Modeling (SEM) approach to modeling longitudinal data that can represent simple and dynamic change over time with either manifest or latent measures of a time-dependent outcome (McArdle, 2001, 2009).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites McArdle, 2001, 2009
Not checked in the free run.
May combine two claims.
InterpretationreasoningThe mediated effect in the LCS ANCOVA model is estimated as the effect of X on change in Y through change in M and is equivalent to the traditional ANCOVA mediated effect, though some parameters are re-parameterizations.
“The mediated effect is estimated as the effect of experimental manipulation, X, on ΔY through its effect on ΔM (ab) and is equivalent to the estimate of the mediated effect in the traditional ANCOVA model although some of the parameters in the LCS specification are re-parameterizations of the traditional ANCOVA parameters.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningBecause all four models can be fitted with LCS, the difference score, residualized change score, and cross-sectional models can be viewed as nested within the ANCOVA model and compared using traditional SEM fit indices.
“Because these models can all be fitted with the LCS specification, we can view the difference score, residualized change score, and cross-sectional models nested within the ANCOVA model to which they can be compared using traditional (SEM) fit indices (See Table 1).”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningAssuming a pretest correlation is present, the difference score b path differs from the ANCOVA b path because mediator and outcome stabilities are not one and cross-lagged paths are not exactly zero.
“Assuming the pretest-correlation is present, the b path from the difference score model differs from the b path from the ANCOVA model because the stabilities of the mediator and outcome are not equal to one and the cross-lagged paths are not exactly zero.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The data-generating model is the ANCOVA model.
- The authors state this as an assumption: A pretest correlation is present.
May combine two claims.
Interpretationnone givenAlthough the difference score model produced elevated Type 1 error rates in some conditions, those conditions are not very likely to occur in practice.
“Although there were some conditions for which the difference score model produced elevated Type 1 error rates, these conditions are not very likely to occur in practice.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationAdding more waves of data may complicate estimation but provides longitudinal estimation of the X to M relation and the M to Y relation.
“The addition of more waves of data may complicate the estimation of mediation effects but provides longitudinal estimation of the X to M relation and longitudinal estimation of the M to Y relation (Cole & Maxwell, 2003; MacKinnon, 1994; 2008).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Cole & Maxwell, 2003
Not checked in the free run.
- cites MacKinnon, 1994
Not checked in the free run.
- cites MacKinnon, 2008
Not checked in the free run.
May combine two claims.
PremisereasoningAssuming the mediator and outcome are related at pretest, the difference score model assumes the stabilities of both mediator and outcome are exactly one and the cross-lagged paths are exactly zero.
“Assuming the mediator and outcome are related at pretest, the difference score model assumes the stabilities of both the mediator and the outcome are exactly equal to one and it assumes the cross-lagged paths are exactly equal to zero.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The mediator and outcome are related at pretest.
May combine two claims.
Interpretationnone givenThe LCS specification can compare non-ANCOVA models to the ANCOVA model and test the assumptions the non-ANCOVA models make about pretest correlation, stabilities, and cross-lagged paths.
“The LCS specification can be used to compare the non-ANCOVA models to the ANCOVA model and test the assumptions the non-ANCOVA models make regarding the pretest correlation, stabilities, and cross-lagged paths.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenAll assumptions and effects regarding stability, timing of effects, and cross-lagged relations across two waves apply across three or more waves.
“That is, all the assumptions and effects regarding stability, timing of effects, and cross-lagged relations across two waves of data will now apply across three or more waves of data.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationnone givenAll four models can be estimated using the LCS specification, and the non-ANCOVA models can be compared to the ANCOVA model using standard SEM fit indices.
“These models can all be estimated using the LCS specification as described in this paper and the non-ANCOVA models can be compared to the ANCOVA model using standard SEM fit indices.”
Quote in the paperSubstantial claimMay combine two claims.
PremisereasoningAssuming mediator and outcome are related at pretest, the residualized change score model assumes cross-lagged paths are zero and stabilities equal the simple regression of posttest on pretest for mediator and outcome.
“Assuming the mediator and outcome are related at pretest, the residualized change score model assumes the cross-lagged paths are equal to zero and the stabilities of the mediator and outcome are equal to the simple regression of posttest scores on pretest scores for the mediator and outcome, respectively.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The mediator and outcome are related at pretest.
Classified as probably not a substantial claim.
Premisenone givenThe difference score and residualized change score models reduce pretest and posttest mediator and dependent variable scores to a single score for M and a single score for Y, allowing methods for a single mediator and single dependent variable to be applied.
“The benefit, however, of some of these models of change, in particular the difference score and residualized change score models is that they reduce the pretest and posttest mediator and dependent variable scores to a single score for M and a single score for Y, respectively, which allows for methods developed for a single mediator and single dependent variable to be easily applied.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationUnreliable measures of mediator and dependent variables can substantially bias mediated effect estimates in most cases, but the pattern can be complicated and counter-intuitive.
“Unreliable measures of the mediator and dependent variables can substantially bias estimates of the mediated effect in most cases but the pattern of results can be complicated and even counter-intuitive in some cases (Fritz, Kenny, & MacKinnon, 2016; Hoyle & Kenny, 1999).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Fritz, Kenny, & MacKinnon, 2016
Not checked in the free run.
- cites Hoyle & Kenny, 1999
Not checked in the free run.
May combine two claims.
Interpretationnone givenIf researchers use the cross-sectional, difference score, or residualized change score model, they may miss the true mediated effect and report biased estimates of it.
“If researchers use the cross-sectional, difference score, or residualized change score model, they may be inadvertently missing the true mediated effect that is present and they may be reporting biased estimates of the true mediated effect.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningIf stabilities are close to one and cross-lagged paths are close to zero, the difference score model would fit approximately as well as the ANCOVA model and their mediated effect estimates would be very similar.
“If the stabilities are close to one and the cross-lagged paths are close to zero, then the difference score model would fit approximately as well as the ANCOVA model and the estimate of the mediated effect across both models would be very similar.”
Quote in the paperSubstantial claimMay combine two claims.
Premisenone givenIn the single mediator model with linear relations, measurement error in the mediator reduces the mediated effect and inflates the direct effect.
“In general, measurement error in the mediator leads to a reduced mediated effect and consequently an inflated direct effect in the single mediator model with linear relations.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
There is a lack of methodological information on estimating mediated effects in the pretest-posttest control group design.
“Models to assess mediation in the pretest-posttest control group design are understudied in the behavioral sciences”
“None of this work has addressed statistical and methodological aspects of the simplest longitudinal model and most common experimental design, the pretest-posttest control group design”
“Therefore there is little information regarding the best statistical model for estimating the mediated effect in this design.”
The authors state this as an assumption.
X as an experimental manipulation marks the beginning of the mediational chain for all units.
“Because X is an experimental manipulation which we assume marks the beginning of the mediational chain for all units”
The authors state this as an assumption.
There are no unmeasured confounders of the M1–Y1 or M2–Y2 relations.
“we assume that there are no unmeasured confounders of the relation between M1 and Y1 or M2 and Y2”
The authors state this as an assumption.
There are no measured or unmeasured confounders of the M2–Y2 relation that are affected by X.
“We also assume there are no measured or unmeasured confounders of the M2 to Y2 relation that are affected by X”
The authors state this as an assumption.
All variables are measured without error.
“and all variables are measured without error”
The authors state this as an assumption.
Randomization successfully assigns units to control and treatment groups so that groups do not differ systematically at pretest.
“Assuming there is successful randomization of units to the control group and treatment group so that these groups do not differ systematically at pretest”
The authors state this as an assumption.
There is no time-varying confounding of the M2–Y2 relation.
“assuming no time-varying confounding”
The paper relies on this without support.
The four models are the most commonly used models for estimating the mediated effect in this design.
“the four most commonly used models used to estimate the mediated effect in this design”
The authors state this as an assumption.
The data-generating model is the ANCOVA model.
“assuming the data-generating model is the ANCOVA model”
“correspond to the ANCOVA model in the Monte Carlo simulation”
The authors state this as an assumption.
Random assignment to two groups is assumed.
“Assuming random assignment to two groups”
The authors state this as an assumption.
A pretest correlation is present.
“Assuming the pretest-correlation is present”
The authors state this as an assumption.
The mediator and outcome were measured without error at both pretest and posttest.
“Further, it was assumed the mediator and outcome were measured perfectly at pretest and posttest.”
The authors state this as an assumption.
There are no time-varying covariates.
“potentially satisfy Assumption 2 and 4 assuming there are no time varying covariates.”
The authors state this as an assumption.
The mediator and outcome are related at pretest.
“Assuming the mediator and outcome are related at pretest, the difference score model assumes the stabilities of both the mediator and the outcome are exactly equal to one and it assumes the cross-lagged paths are exactly equal to zero.”
“Assuming the mediator and outcome are related at pretest, the residualized change score model assumes the cross-lagged paths are equal to zero and the stabilities of the mediator and outcome are equal to the simple regression of posttest scores on pretest scores for the mediator and outcome, respectively.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
We controlled all mediators and outcomes for their baseline value.
“We controlled all mediators and outcomes for their baseline value, following findings from a simulation study that found this approach superior to other methods like difference scores in terms of error rates, bias, power, and confidence interval coverage (Valente & MacKinnon, 2017).”
From Transdiagnostic Processes as Mediators of Change in an Internet-Delivered Intervention Based on the Unified Protocol (2022) · cites this paper as Valente & MacKinnon, 2017
The Introduction’s ANCOVA section describes adjusting posttest mediator and outcome scores for their pretest values, and the Discussion recommends ANCOVA for testing mediated effects in a pretest–posttest control-group design. The Results support advantages in Type 1 error rates and power under some conditions, but report that the difference-score model was also unbiased; they do not establish that ANCOVA was superior to it on every listed criterion, including bias and confidence interval coverage. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThis approach has been argued to be the most powerful analysis method as investigated in simulation research comparing multiple methods of assessing the mediated effect in the pre-post control group design.
“This approach has been argued to be the most powerful analysis method as investigated in simulation research comparing multiple methods of assessing the mediated effect in the pre-post control group design (Valente and MacKinnon, 2017).”
From Aversive response to uncertainty as a mediator for the effect of a mindfulness intervention on symptoms of anxiety (2022) · cites this paper as Valente and MacKinnon, 2017
The Abstract and Discussion report a simulation comparing four methods and recommend ANCOVA for estimating the mediated effect. But the Discussion says the cross-sectional model had equal or higher statistical power in some conditions, though its estimates were biased. Thus the paper supports ANCOVA as the preferred method overall, not as uniformly the most powerful. (LLM, conf=0.86)
Experimental — accuracy not yet measuredBaseline measures of the outcome and mediator were included as covariates in the mediation modeling to account for changes over time, as recommended by Hayes and Valente and MacKinnon.
“To take into account changes over time, baseline (pre-intervention period) measures of the outcome and mediator (“lags”) were included in the modeling as covariates in preference to difference scores, as recommended by Hayes (59) and Valente and MacKinnon (62).”
From Group Mindfulness-Integrated Cognitive Behavior Therapy (MiCBT) Reduces Depression and Anxiety and Improves Flourishing in a Transdiagnostic Primary Care Sample Compared to Treatment-as-Usual: A Randomized Controlled Trial (2022) · cites this paper as Valente and MacKinnon (62)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
ANCOVA power is called unaffected despite reported effects
MinorThe simulation summary says predictors did not affect ANCOVA power, but the power results report effects of sample size, mediated-effect size, and their interaction for every model, including ANCOVA.
“The statistical power for the ANCOVA model was unaffected by the predictors in the simulation study”
“For every model there was a significant true value of the mediated effect by sample size two-way interaction (ηp2≥0.01for all models) and main effects of sample size (ηp2≥0.01) and true value of the mediated effect (ηp2≥0.01) on the power to detect the mediated effect.”
Cross-sectional power is described as both decreasing and not decreasing
MinorThe discussion says that unmet model assumptions reduced power for the cross-sectional model, then says in the same paragraph that its power did not decrease as a function of those factors.
“When the assumptions of these models (i.e., assumptions about stabilities and cross-lagged paths) did not hold, the power to detect the mediated effect decreased”
“the power to detect the mediated effect with the ANCOVA and cross-sectional models did not decrease as a function of these factors.”
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,385 · see citing works
(2002) · Cited by 13,211 · see citing works
Mediation in experimental and nonexperimental studies: New procedures and recommendations.
Shrout, P. E., & Bolger, N. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.4.422 · Cited by 11,335 · see citing works
A comparison of methods to test mediation and other intervening variable effects
MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.83 · Cited by 9,406 · see citing works
Self-citationConfidence Limits for the Indirect Effect: Distribution of the Product and Resampling Methods
MacKinnon, D. P., Lockwood, C. M., & Williams, J. (2004) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3901_4 · Cited by 7,806 · see citing works
Self-citationIntroduction to Statistical Mediation Analysis
MacKinnon, D. P. (2012) · doi:10.4324/9780203809556 · Cited by 5,618 · see citing works
Self-citationEquivalence of the Mediation, Confounding and Suppression Effect
MacKinnon, D. P., Krull, J. L., & Lockwood, C. M. (2000) · Prevention Science · doi:10.1023/a:1026595011371 · Cited by 4,188 · see citing works
Self-citationA general approach to causal mediation analysis.
Imai, K., Keele, L., & Tingley, D. (2010) · Psychological Methods · doi:10.1037/a0020761 · Cited by 4,143 · see citing works
(1947) · The American Statistician · doi:10.1080/00031305.1947.10501534 · Cited by 3,745 · see citing works
Cole, D. A., & Maxwell, S. E. (2003) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.112.4.558 · Cited by 3,605 · see citing works
Bias in cross-sectional analyses of longitudinal mediation
Maxwell, S. E., & Cole, D. A. (2007) · Psychological Methods · doi:10.1037/1082-989x.12.1.23 · Cited by 3,401 · see citing works
Flora, D. B., & Curran, P. J. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.466 · Cited by 3,105 · see citing works
Preacher, K. J., & Kelley, K. (2011) · Psychological Methods · doi:10.1037/a0022658 · Cited by 3,060 · see citing works
How we should measure "change": Or should we?
Cronbach, L. J., & Furby, L. (1970) · Psychological Bulletin · doi:10.1037/h0029382 · Cited by 2,456 · see citing works
Statistical Methods for the Social Sciences
EJD, Agresti, A., & Finlay, B. (1998) · Journal of the American Statistical Association · doi:10.2307/2670147 · Cited by 2,133 · see citing works
Valeri, L., & VanderWeele, T. J. (2013) · Psychological Methods · doi:10.1037/a0031034 · Cited by 2,132 · see citing works
Latent Variable Modeling of Differences and Changes with Longitudinal Data
McArdle, J. J. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163612 · Cited by 1,952 · see citing works
Estimating Mediated Effects in Prevention Studies
MacKinnon, D. P., & Dwyer, J. H. (1993) · Evaluation Review · doi:10.1177/0193841x9301700202 · Cited by 1,936 · see citing works
Self-citationStatistical Strategies for Small Sample Research
Hoyle, R. H. (1999) · Cited by 1,896 · see citing works
Statistical Methods for Social Scientists
(1977) · Elsevier eBooks · doi:10.1016/c2009-0-22083-6 · Cited by 1,880 · see citing works
A Simulation Study of Mediated Effect Measures
MacKinnon, D. P., Warsi, G. M., & Dwyer, J. H. (1995) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3001_3 · Cited by 1,715 · see citing works
Self-citationKraemer, H. C., Stice, E., Kazdin, A., Offord, D., & Kupfer, D. (2001) · American Journal of Psychiatry · doi:10.1176/appi.ajp.158.6.848 · Cited by 1,712 · see citing works
Identification, Inference and Sensitivity Analysis for Causal Mediation Effects
Imai, K., Keele, L., & Yamamoto, T. (2010) · Statistical Science · doi:10.1214/10-sts321 · Cited by 1,628 · see citing works
Maxwell, S. E., Cole, D. A., & Mitchell, M. A. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.606716 · Cited by 1,391 · see citing works
Distribution of the product confidence limits for the indirect effect: Program PRODCLIN
MacKinnon, D. P., Fritz, M. S., Williams, J. G., & Lockwood, C. M. (2007) · Behavior Research Methods · doi:10.3758/bf03193007 · Cited by 1,328 · see citing works
Self-citationCounterfactuals and Causal Inference: Methods and Principles for Social Research
Morgan, S. L., & Winship, C. (2014) · Cambridge University Press · doi:10.1017/cbo9781107587991 · Cited by 1,037 · see citing works
Designing Experiments and Analyzing Data: A Model Comparison Perspective, Third Edition
Maxwell, S. E., & Delaney, H. D. (1999) · Medical Entomology and Zoology · Cited by 975 · see citing works
Identification, Inference, and Sensitivity Analysis for Causal Mediation Effects
Imai, K., & Keele, L. J. (2008) · Cited by 887 · see citing works
MacCoun, R. J. (2003) · Journal of Policy Analysis and Management · doi:10.1002/pam.10129 · Cited by 828 · see citing works
Van Breukelen, G. J. P. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.02.007 · Cited by 765 · see citing works
Conceptual issues concerning mediation, interventions and composition
VanderWeele, T. J., & Vansteelandt, S. (2009) · Statistics and Its Interface · doi:10.4310/sii.2009.v2.n4.a7 · Cited by 731 · see citing works
Designing experiments and analyzing data: a model comparison perspective
(1990) · Choice Reviews Online · doi:10.5860/choice.28-2192 · Cited by 673 · see citing works
A paradox in the interpretation of group comparisons.
Lord, F. M. (1967) · Psychological Bulletin · doi:10.1037/h0025105 · Cited by 614 · see citing works
Interpretation and identification of causal mediation
Pearl, J. (2014) · Psychological Methods · doi:10.1037/a0036434 · Cited by 554 · cited 1× · see citing works
“Pearl, 2014, for alternate assumptions regarding identification of causal mediated effects).”
BackgroundEstimating the effects of social interventions
Judd, C. M., & Kenny, D. A. (1981) · Medical Entomology and Zoology · Cited by 538 · see citing works
Jang, H., Kim, E. J., & Reeve, J. (2012) · Journal of Educational Psychology · doi:10.1037/a0028089 · Cited by 529 · see citing works
The Phantom Menace: Omitted Variable Bias in Econometric Research
Clarke, K. A. (2005) · Conflict Management and Peace Science · doi:10.1080/07388940500339183 · Cited by 484 · cited 1× · see citing works
“The difference in the b path across models can be thought of as an unmeasured confounder problem by applying results from Clark (2005) and Hanushek and Jackson (1977).”
MethodA Primer on Regression Artifacts
Campbell, D. T., & Kenny, D. A. (1999) · Medical Entomology and Zoology · Cited by 482 · see citing works
Investigation of Mediational Processes Using Parallel Process Latent Growth Curve Modeling
Cheong, J., MacKinnon, D. P., & Khoo, S. T. (2003) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem1002_5 · Cited by 442 · see citing works
Self-citationBias Formulas for Sensitivity Analysis for Direct and Indirect Effects
VanderWeele, T. J. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181df191c · Cited by 435 · cited 1× · see citing works
“…effects of unmeasured confounders on the M to Y relation in the single mediator model (Cox et al., 2013; Imai, Keele, & Yamamoto, 2010; Mauro, 1990; VanderWeele, 2010), sensitivity analyses can be used to assess the potential effect of an unmeasured pretest correlation between M and Y, unmeasured…”
BackgroundMyths about longitudinal research.
Rogosa, D. (1988) · Cited by 383 · see citing works
Monotonicity of effect sizes: questioning kappa-squared as mediation effect size measure
Wen, Z., & Fan, X. (2015) · Psychological Methods · doi:10.1037/met0000029 · Cited by 350 · cited 1× · see citing works
“…limit estimation (Cheung, 2007, 2009; MacKinnon, Lockwood, & Williams, 2004; MacKinnon, Warsi, & Dwyer, 1995), effect size (Preacher & Kelly, 2011; Wen & Fan, 2015), influence of omitted variables (Cox, Kisbu-Sakarya, Miočević, & MacKinnon, 2013; James, 1980; Jo, 2008; MacKinnon & Pirlott, 2015),…”
MacKinnon, D. P., Johnson, C. A., Pentz, M. A., Dwyer, J. H., Hansen, W. B., Flay, B. R., & Wang, E. Y.-I. (1991) · Health Psychology · doi:10.1037/0278-6133.10.3.164 · Cited by 311 · see citing works
Self-citationAnalysis of Pretest-Posttest Designs
Bonate, P. L. (2000) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420035926 · Cited by 305 · see citing works
Statistical models for the social and behavioral sciences
Dwyer, J. H. (1983) · Oxford University Press eBooks · Cited by 299 · see citing works
Comparison of methods for constructing confidence intervals of standardized indirect effects
Cheung, M. W. ‐. (2009) · Behavior Research Methods · doi:10.3758/brm.41.2.425 · Cited by 279 · see citing works
Analysis of Mediating Variables in Prevention and Intervention Research
MacKinnon, D. P. (1994) · PsycEXTRA Dataset · doi:10.1037/e495892006-009 · Cited by 274 · see citing works
Self-citationEffect Analysis and Causation in Linear Structural Equation Models
Sobel, M. E. (1990) · Psychometrika · doi:10.1007/bf02294763 · Cited by 264 · see citing works
Cheung, M. W. ‐. (2007) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705510709336745 · Cited by 258 · see citing works
MacKinnon, D. P., & Pirlott, A. G. (2014) · Personality and Social Psychology Review · doi:10.1177/1088868314542878 · Cited by 257 · see citing works
Self-citationGoldberg, L., Elliot, D., Clarke, G., Mackinnon, D., Moe, E. L., Zoref, L., Green, C., Wolf, S., Greffrath, E., Miller, D. J., et al. (1996) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.276.19.1555 · Cited by 218 · see citing works
Self-citationInterpreting and estimating indirect effects assuming time lags really matter.
Gollob, H. F., & Reichardt, C. S. (1991) · Best methods for the analysis of change: Recent advances, unanswered questions, future directions. · doi:10.1037/10099-015 · Cited by 198 · see citing works
ANCOVA Versus CHANGE From Baseline in Nonrandomized Studies: The Difference
van Breukelen, G. (2013) · Multivariate Behavioral Research · doi:10.1080/00273171.2013.831743 · Cited by 196 · see citing works
Jansen, B. R. J., Louwerse, J., Straatemeier, M., Van der Ven, S. H. G., Klinkenberg, S., & Van der Maas, H. L. J. (2013) · Learning and Individual Differences · doi:10.1016/j.lindif.2012.12.014 · Cited by 193 · cited 2× · see citing works
“Four statistical models can be used to investigate the mediated effect: the cross-sectional model, the difference score model (Jansen et al., 2013; MacKinnon et al., 1991), the residualized change score model (Miller, Trost, & Brown, 2002; Reid & Aiken, 2013), and the ANCOVA model (Jang et al.,…”
MethodTein, J.-Y., Sandler, I. N., MacKinnon, D. P., & Wolchik, S. A. (2004) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.72.4.617 · Cited by 186 · see citing works
Self-citationMediators of physical activity behavior change among women with young children
Miller, Y. D., Trost, S. G., & Brown, W. J. (2002) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(02)00484-1 · Cited by 180 · see citing works
Schmiege, S. J., Broaddus, M., Levin, M. E., & Bryan, A. D. (2009) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0014513 · Cited by 153 · see citing works
Methodological Issues in Aging Research
Bergeman, C. S., & Boker, S. M. (2015) · Psychology Press eBooks · doi:10.4324/9781315820989 · Cited by 147 · see citing works
MacKinnon, D. P., Goldberg, L., Clarke, G. N., Elliot, D. L., Cheong, J., Lapin, A., MOE, E. L., & Krull, J. L. (2001) · Prevention Science · doi:10.1023/a:1010082828000 · Cited by 143 · see citing works
Self-citationThe Language of Social Research; a Reader in the Methodology of Social Research.
Morgenbesser, S., Lazarsfeld, P. F., & Rosenberg, M. (1956) · The Journal of Philosophy · doi:10.2307/2022744 · Cited by 143 · see citing works
Causal inference in randomized experiments with mediational processes.
Jo, B. (2008) · Psychological Methods · doi:10.1037/a0014207 · Cited by 137 · see citing works
Comparing groups in a before–after design: When t test and ANCOVA produce different results
Wright, D. B. (2006) · British Journal of Educational Psychology · doi:10.1348/000709905x52210 · Cited by 135 · see citing works
Further Comparative Analyses of Pretest-Posttest Research Designs
Laird, N. M. (1983) · The American Statistician · doi:10.1080/00031305.1983.10483133 · Cited by 133 · see citing works
Mauro, R. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.2.314 · Cited by 130 · see citing works
The unmeasured variables problem in path analysis.
James, L. R. (1980) · Journal of Applied Psychology · doi:10.1037/0021-9010.65.4.415 · Cited by 128 · see citing works
Effects of a Multidimensional Anabolic Steroid Prevention Intervention
Goldberg, L. (1996) · JAMA · doi:10.1001/jama.1996.03540190027025 · Cited by 128 · see citing works
Reid, A. E., & Aiken, L. S. (2013) · Health Psychology · doi:10.1037/a0028140 · Cited by 110 · see citing works
The Combined Effects of Measurement Error and Omitting Confounders in the Single-Mediator Model
Fritz, M. S., Kenny, D. A., & MacKinnon, D. P. (2016) · Multivariate Behavioral Research · doi:10.1080/00273171.2016.1224154 · Cited by 104 · see citing works
Self-citationSensitivity Plots for Confounder Bias in the Single Mediator Model
Cox, M. G., Kisbu‐Sakarya, Y., Miočević, M., & MacKinnon, D. P. (2013) · Evaluation Review · doi:10.1177/0193841x14524576 · Cited by 70 · see citing works
Self-citationA Note on Testing Mediated Effects in Structural Equation Models
Valente, M. J., González, O., Miočević, M., & MacKinnon, D. P. (2015) · Educational and Psychological Measurement · doi:10.1177/0013164415618992 · Cited by 70 · see citing works
Self-citationThe Use of Propensity Scores in Mediation Analysis
Jo, B., Stuart, E. A., MacKinnon, D. P., & Vinokur, A. D. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.576624 · Cited by 69 · see citing works
Self-citationKisbu-Sakarya, Y., Mackinnon, D., & Aiken, L. (2012) · Educational and Psychological Measurement · doi:10.1177/0013164412450574 · Cited by 59 · see citing works
Self-citationDealing with baseline differences: two principles and two dilemmas
Jamieson, J. (1999) · International Journal of Psychophysiology · doi:10.1016/s0167-8760(98)00048-8 · Cited by 49 · see citing works
Anova of Residuals and Ancova: Correcting an Illusion by Using Model Comparisons and Graphs
Maxwell, S. E., Delaney, H. D., & Manheimer, J. M. (1985) · Journal of Educational Statistics · doi:10.3102/10769986010003197 · Cited by 33 · see citing works
Psychophysical system identification by correlation methods
Eijkman, E. G. J. (1978) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1978.tb00587.x · Cited by 30 · see citing works
An Exponential Decay Model for Mediation
Fritz, M. S. (2013) · Prevention Science · doi:10.1007/s11121-013-0390-x · Cited by 22 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- 10.1111/bmsp.1978.31.issue-2 · DOI
- McArdle, J. J. (2001). A latent difference score approach to longitudinal dynamic structure analysis. In R. Cudeck, S. Du Toit, & D. Sorbom (Eds.), Structural equation modeling: Present and future. A Festschrift in honor of Karl Jöreskog (pp. 341–380). Washington, DC: American Psychological Association.
- Miočević M. MacKinnon D. P. & Levy R. (2015 June). Bayesian longitudinal mediation for stable processes. Poster presented at O-Bayes workshop Valencia Spain.
- Muthén, L. K., & Muthén, B. O. (1998–2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.
Cited by
Sorted by citations
Dzhambov, A. M. (2018) · Archives of Industrial Hygiene and Toxicology · doi:10.2478/aiht-2018-69-3166 · Cited by 74 · see citing works
Fonseca, A., Monteiro, F., Alves, S., Gorayeb, R., & Canavarro, M. C. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00265 · Cited by 67 · see citing works
Behrendt, D., Ebert, D. D., Spiegelhalder, K., & Lehr, D. (2020) · Journal of Medical Internet Research · doi:10.2196/13346 · Cited by 52 · see citing works
Francis, S. E. B., Shawyer, F., Cayoun, B., Enticott, J., & Meadows, G. N. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.815170 · Cited by 45 · see citing works
“To take into account changes over time, baseline (pre-intervention period) measures of the outcome and mediator (“lags”) were included in the modeling as covariates in preference to difference scores, as recommended by Hayes (59) and Valente and MacKinnon (62).”
MethodHanson, J. L., Gillmore, A. D., Yu, T., Holmes, C. J., Hallowell, E. S., Barton, A. W., Beach, S. R. H., Galván, A., MacKillop, J., Windle, M., et al. (2019) · Child Development · doi:10.1111/cdev.13154 · Cited by 41 · see citing works
Fishbein, J. N., Judd, C. M., Genung, S., Stanton, A. L., & Arch, J. J. (2022) · Behaviour Research and Therapy · doi:10.1016/j.brat.2022.104103 · Cited by 28 · see citing works
Carona, C., Pereira, M., Araújo-Pedrosa, A., Canavarro, M. C., & Fonseca, A. (2023) · JMIR Mental Health · doi:10.2196/39253 · Cited by 33 · see citing works
“Finally, to examine whether changes in psychological processes (emotion regulation, self-compassion, and psychological flexibility) were associated with changes in women’s adjustment outcomes (depressive and anxiety symptoms), a 2-wave LCS (2W-LCS) model was tested [30,33,34].”
Papenfuss, I., Lommen, M. J. J., Huisman, M., & Ostafin, B. D. (2022) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2022.06.014 · Cited by 27 · see citing works
Fishbein, J. N., Haslbeck, J., & Arch, J. J. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104266 · Cited by 24 · see citing works
Schaeuffele, C., Bär, J., Buengener, I., Grafiadeli, R., Heuthaler, E., Strieder, J., Ziehn, P., Knaevelsrud, C., Renneberg, B., & Boettcher, J. (2022) · Cognitive Therapy and Research · doi:10.1007/s10608-021-10272-y · Cited by 24 · see citing works
Morin, L., Grégoire, S., & Lachance, L. (2021) · Journal of American College Health · doi:10.1080/07448481.2019.1705828 · Cited by 26 · see citing works
- Authors
- Valente, M. J., & MacKinnon, D. P.
- Affiliations
- Arizona State University (US)
- Corresponding author
- Valente, Matthew J.
- Venue
- Structural Equation Modeling: A Multidisciplinary Journal
- Publisher
- Informa UK Limited
- Published
- 2017-02-08
- Volume
- 24
- Issue
- 3
- Pages
- 428-450
- ISSN
- 1070-5511 (ISSN-L)
- 1532-8007
- PMC
- PMC5568008
- OpenAlex
- W2587551696
- Cited by
- 173
- References
- 77
- Citations by year
- 2026: 14 · 2025: 31 · 2024: 18 · 2023: 33 · 2022: 19 · 2021: 22 · 2020: 8 · 2019: 16 · 2018: 8 · 2017: 4
- FWCI
- 6.85
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Structural Equation Modeling A Multidisciplinary Journal (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute on Drug Abuse (R37 DA09757)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- mediation analysis, pretest-posttest control group design, latent change score models, covariance analysis, type I error, statistical power, bias, confidence interval coverage
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read under definition version 3 (extract-v3) on 2026-09-28.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
