Impact of experimental design on effect size: Findings from the research literature on training.

Carlson, Kevin D., Schmidt, Frank L.·1999·Journal of Applied Psychology

Abstract

This article examines whether differences in the equations commonly used to calculate effect size for single group pretest-posttest (SGPP) designs versus those for control group designs can account for the finding that SGPP designs yield larger mean effect sizes (e.g., M. S. Lipsey & D. B. Wilson, 1993). It was found that the assumptions of no control group effect and the equivalence of pretraining and posttraining dependent variable standard deviations required for these equations to produce equivalent estimates of effect size were violated for some dependent variable types. Results indicate that control group effects and inflation in the standard deviation of the posttraining dependent variable measure account for most of the observed difference in effect size. The most severe violations occurred when the dependent variable was a knowledge assessment. Methods for including data from SGPP designs in meta-analyses that minimize potential biases are discussed. In an extensive review of psychological, educational, and behavioral treatment research, Lipsey and Wilson (1993) examined results from meta-analyses reporting effect size estimates for studies subgrouped by research design. Data from studies employing single group pretest-posttest (SGPP) designs were compared with data from studies using control or comparison group designs (i.e., pretestposttest with control, PPWC; and posttest only with control, POWC). Across 45 meta-analyses , mean effect sizes (dvalues) for SGPP designs were 0.29 SDs larger (0.76 versus 0.47) than effect sizes for studies using control or comparison group designs. Research by Guzzo, Jette, and Katzell (1985) has found similar differences. The common conclusion among these researchers is that data from SGPP designs clearly are capable of upwardly biasing the mean treatment effect estimates derived from meta-analysis (Lipsey & Wilson, 1993, p. 1194). Lipsey and Wilson's (1993) solution to the perceived upward bias in effect size for SGPP designs was not to

Cites (2)

  1. The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis.
    Lipsey, Mark W., Wilson, David B. · 1993 · American Psychologist
  2. Synthesizing standardized mean‐change measures
    Becker, Betsy Jane · 1988 · British Journal of Mathematical and Statistical Psychology

Show all 2 references

Cited by (35)

  1. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs.
    Morris, Scott B., DeShon, Richard P. · 2002 · Psychological Methods
  2. Estimating effect sizes from pretest-posttest-control group designs
    Morris, Scott B. · 2008 · Organizational Research Methods
  3. The Triple P-Positive Parenting Program: a systematic review and meta-analysis of a multi-level system of parenting support
    Sanders, Matthew R., Kirby, James N., Tellegen, Cassandra L., Day, Jamin J. · 2014 · Clinical Psychology Review
  4. A Meta-Analysis of Compassion-Based Interventions: Current State of Knowledge and Future Directions
    Kirby, James N., Tellegen, Cassandra L., Steindl, Stanley R. · 2017 · Behavior Therapy
  5. Efficacy of Neurofeedback Treatment in ADHD: The Effects on Inattention, Impulsivity and Hyperactivity: A Meta-Analysis
    Arns, Martijn, de Ridder, Sabine, Strehl, Ute, Breteler, Marinus, Coenen, Anton · 2009 · Clinical EEG and Neuroscience

See all 202

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Impact of experimental design on effect size: Findings from the research literature on training. https://app.cochat.ai/papers/10.1037//0021-9010.84.6.851
The paper
Carlson, K. D., & Schmidt, F. L. (1999). Impact of experimental design on effect size: Findings from the research literature on training. Journal of Applied Psychology. https://doi.org/10.1037//0021-9010.84.6.851

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.