Using a repeated measures ANOVA to analyze the data from a pretest–posttest design: a potentially confusing task
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
The pretest-posttest control group design (or an extension of it) is a highly prestigious experimental design. A popular analytic strategy involves subjecting the data provided, by this design to a repeated measures analysis of variance (ANOVA). Unfortunately, the statistical results yielded by this type of analysis can easily be misinterpreted, since the score model underlying the analysis is not correct. Examples from recently published articles are used to demonstrate that this statistical procedure has led to (a) incorrect statements regarding treatment effects, (b) completely redundant reanalyses of the same data, and (c) problems with respect to post hoc investigations; Two alternative strategies—gain scores and covariance—are discussed and compared.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Sessions will be videotaped and transcribed, and content analysis will be used to identify and code the major themes in the data [108].”
From LIBER8 design and methods: An integrative intervention for loss of control eating among African American and White adolescent girls (2013) · cites this paper as [108]
Citations
Cites
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 26,755
Experimental Design: Procedures for the Behavioral Sciences
Kirk, R. (2013) · doi:10.4135/9781483384733 · Cited by 8,539
Experimental Design in Psychological Research
(2002) · Foundations of Cognitive Psychology · doi:10.7551/mitpress/3080.003.0011 · Cited by 2,465
Experimental and quasi-experimental designs for research on teaching
Campbell, D. T. (1963) · Medical Entomology and Zoology · Cited by 1,541
Marascuilo, L. A., & Levin, J. R. (1970) · American Educational Research Journal · doi:10.3102/00028312007003397 · Cited by 177
Analysis of Covariance: A Delicate Instrument
ELASHOFF, J. D. (1969) · American Educational Research Journal · doi:10.2307/1161858 · Cited by 176
Leventhal, G. S., Michaels, J. W., & Sanford, C. (1972) · Journal of Personality and Social Psychology · doi:10.1037/h0032877 · Cited by 159
A Comparison of the Precision of Three Experimental Designs Employing a Concomitant Variable
Feldt, L. S. (1958) · Psychometrika · doi:10.1007/bf02289783 · Cited by 112 · cited 1×
“1 For a more detailed comparison of covariance versus gain scores regarding the issue of power, see Feldt (1958) and Gourlay (1953).”
BackgroundAtiqullah, M. (1962) · Biometrika · doi:10.1093/biomet/49.1-2.83 · Cited by 60
Type IV errors and interactions.
Levin, J. R., & Marascuilo, L. A. (1972) · Psychological Bulletin · doi:10.1037/h0033455 · Cited by 52
Games, P. A. (1973) · Psychological Bulletin · doi:10.1037/h0034832 · Cited by 28 · cited 1×
“Games (1973), however, has argued that (a) tests of simple main effects represent a completely legitimate post hoc procedure and (b) "the Marascuilo and Levin 'simple effects design' is a poor substitute for the common factorial analysis of variance logical sequence" (p.”
BackgroundSex of teacher and student reading achievement.
Asher, S. R., & Gottman, J. M. (1973) · Journal of Educational Psychology · doi:10.1037/h0034986 · Cited by 26 · cited 1×
“Consider two recent investigations, one by Asher and Gottman (1973), the other by Roth and McManis (1972).”
BackgroundLevin, J. R., & Marascuilo, L. A. (1973) · Psychological Bulletin · doi:10.1037/h0034833 · Cited by 26
Regions of Significant Criterion Differences in Aptitude-Treatment-Interaction Research
Cahen, L. S., & Linn, R. L. (1971) · American Educational Research Journal · doi:10.3102/00028312008003521 · Cited by 22
COVARIANCE ANALYSIS AND ITS APPLICATIONS IN PSYCHOLOGICAL RESEARCH
Gourlay, N. (1953) · British Journal of Statistical Psychology · doi:10.1111/j.2044-8317.1953.tb00128.x · Cited by 17
Use of direct expectancy to modify academic performance and attitudes of college students.
Meichenbaum, D. H., & Smart, I. (1971) · Journal of Counseling Psychology · doi:10.1037/h0031749 · Cited by 16
Group counseling with reading disabled children.
McCollum, P. S., & Anderson, R. P. (1974) · Journal of Counseling Psychology · doi:10.1037/h0036284 · Cited by 12
Social reinforcement effects on Block Design performance of organic and nonorganic retarded adults.
Roth, G., & McManis, D. L. (1972) · PubMed · Cited by 6
Gross, S. J., & Schwebel, A. I. (1973) · Psychological Reports · doi:10.2466/pr0.1973.33.2.623 · Cited by 5 · cited 1×
“Consider a recent study by Gross and Schwebel (1973) in which a repeated measures ANOVA was applied to a pretest-post-”
Method
Cited by
Unrealistic optimism about future life events
Weinstein, N. D. (1980) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.39.5.806 · Cited by 4,999
“These changes were studied in additional analyses that treated pretreatment event responses as covariates and tested for group differences on treatment and posttreatment events (Huck & McLean, 1975).”
MethodPretest-posttest designs and measurement of change
Dimitrov, D. M., & Rumrill, Jr. (2003) · WORK: A Journal of Prevention, Assessment & Rehabilitation · doi:10.3233/wor-2003-00285 · Cited by 908
“A very little known fact is also that the F statistic for the interaction between the treatment factor and the pretest-posttest factor is identical to the F statistic for the treatment main effect with a one-way ANOVA on gain scores [ 10 ].”
MethodResultSTRATEGIC CHANGE: THE EFFECTS OF FOUNDING AND HISTORY.
Boeker, W. (1989) · Academy of Management Journal · doi:10.2307/256432 · Cited by 833
Howard, G. S., Ralph, K. M., Gulanick, N. A., Maxwell, S. E., Nance, D. W., & Gerber, S. K. (1979) · Applied Psychological Measurement · doi:10.1177/014662167900300101 · Cited by 462
When our identities are mistaken: Reaffirming self-conceptions through social interaction.
Swann, W. B., & Hill, C. A. (1982) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.43.1.59 · Cited by 459
“We report difference-score.analyses instead of the more sensitive analysis of covariance (ANCOVA; e.g., Huck & McLean, 1975), because it allowed us to contrast the effects of consistent versus discrepant feedback directly and clearly.”
Method
Metadata
- Authors
- Huck, S. W., & McLean, R. A.
- Affiliations
- University of Tennessee at Knoxville (US)
- Venue
- Psychological Bulletin
- Published
- 1975-07
- Volume
- 82
- Issue
- 4
- Pages
- 511-518
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2072435510
- Cited by
- 449
- References
- 19
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 6 · 2023: 8 · 2022: 13 · 2021: 14 · 2020: 18 · 2019: 17 · 2018: 15 · 2017: 11 · 2016: 21 · 2015: 20 · 2014: 15 · 2013: 20 · 2012: 11
- FWCI
- 15.23
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- pretest-posttest control group design, repeated measures ANOVA, SCORE risk model, treatment effect estimation, misinterpretation of statistical results, reanalysis of data, post hoc comparisons, gain score, covariance analysis, experimental design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.