A generally robust approach to hypothesis testing in independent and correlated groups designs
Abstract
Standard least squares analysis of variance methods suffer from poor power under arbitrarily small departures from normality and fail to control the probability of a Type I error when standard assumptions are violated. These problems are vastly reduced when using a robust measure of location; incorporating bootstrap methods can result in additional benefits. This paper illustrates the use of trimmed means with an approximate degrees of freedom heteroskedastic statistic for independent and correlated groups designs in order to achieve robustness to the biasing effects of nonnormality and variance heterogeneity. As well, we indicate when a boostrap methodology can be effectively employed to provide improved Type I error control. We also illustrate, with examples from the psychophysiological literature, the use of a new computer program to obtain numerical results for these solutions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A winsorization procedure was applied to subjects' individual mean response times per condition and target cue to account for outliers before further statistical analyses.
“A winsorization procedure was applied to subjects’ individual mean response times (RT) per condition and target cue to account for outliers before further statistical analyses was conducted.”
From Event-related potentials of automatic imitation are modulated by ethnicity during stimulus processing, but not during motor execution (2018) · cites this paper as [59]
Including outlier values in the analysis violates assumptions of general linear model estimations.
“Including outlier values in the analysis violates assumptions of general linear model estimations59,61.”
From Event-related potentials of automatic imitation are modulated by ethnicity during stimulus processing, but not during motor execution (2018) · cites this paper as [59]
Cites
Introduction to Robust Estimation and Hypothesis Testing
Wilcox, R. R. (2012) · Elsevier eBooks · doi:10.1016/c2010-0-67044-1 · Cited by 2,086 · see citing works
Self-citationON THE COMPARISON OF SEVERAL MEAN VALUES: AN ALTERNATIVE APPROACH
Welch, B. L. (1951) · Biometrika · doi:10.1093/biomet/38.3-4.330 · Cited by 1,531 · see citing works
THE SIGNIFICANCE OF THE DIFFERENCE BETWEEN TWO MEANS WHEN THE POPULATION VARIANCES ARE UNEQUAL
Welch, B. L. (1938) · Biometrika · doi:10.1093/biomet/29.3-4.350 · Cited by 1,506 · see citing works
Resampling-Based Multiple Testing.
Booth, J. G., Westfall, P. H., & Young, S. S. (1994) · Journal of the American Statistical Association · doi:10.2307/2291234 · Cited by 914 · see citing works
Contributions to Probability and Statistics
Gleser, L. J., Perlman, M. D., Press, S. J., & Sampson, A. R. (1989) · Springer eBooks · doi:10.1007/978-1-4612-3678-8 · Cited by 612 · see citing works
The two-sample trimmed t for unequal population variances
Yuen, K. K. (1974) · Biometrika · doi:10.1093/biomet/61.1.165 · Cited by 507 · see citing works
Harwell, M. R., Rubinstein, E. N., Hayes, W. S., & Olds, C. C. (1992) · Journal of Educational Statistics · doi:10.3102/10769986017004315 · Cited by 485 · see citing works
Robust Estimation and Testing.
Iglewicz, B., Staudte, R. G., & Sheather, S. J. (1991) · Journal of the American Statistical Association · doi:10.2307/2290433 · Cited by 454 · see citing works
Introduction to Robust Estimation and Hypothesis Testing
Stahel, W. A., & Wilcox, R. R. (1998) · Journal of the American Statistical Association · doi:10.2307/2669876 · Cited by 437 · see citing works
Self-citationFundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy
Wilcox, R. R. (2010) · doi:10.1007/978-1-4419-5525-8 · Cited by 404 · see citing works
Self-citationThe Significance of the Difference Between Two Means when the Population Variances are Unequal
Welch, B. L. (1938) · Biometrika · doi:10.2307/2332010 · Cited by 374 · see citing works
On the Bootstrap and Confidence Intervals
Hall, P. A. (1986) · The Annals of Statistics · doi:10.1214/aos/1176350168 · Cited by 325 · see citing works
On the Number of Bootstrap Simulations Required to Construct a Confidence Interval
Hall, P. A. (1986) · The Annals of Statistics · doi:10.1214/aos/1176350169 · Cited by 246 · see citing works
Fundamentals of Modern Statistical Methods
Wilcox, R. R. (2001) · Springer eBooks · doi:10.1007/978-1-4757-3522-2 · Cited by 227 · see citing works
Self-citationKeselman, H. J., & Rogan, J. C. (1980) · Psychophysiology · doi:10.1111/j.1469-8986.1980.tb00190.x · Cited by 208 · see citing works
Self-citationJohansen, S. (1980) · Biometrika · doi:10.1093/biomet/67.1.85 · Cited by 199 · see citing works
Bootstrapping: Applications to Psychophysiology
Wasserman, S. S., & Böckenholt, U. (1989) · Psychophysiology · doi:10.1111/j.1469-8986.1989.tb03159.x · Cited by 161 · see citing works
Keselman, H. J. (1998) · Psychophysiology · doi:10.1111/1469-8986.3540470 · Cited by 158 · see citing works
Self-citationJohansen, S. (1980) · Biometrika · doi:10.2307/2335320 · Cited by 132 · see citing works
Jonkman, L. M., Kemner, C., Verbaten, M. N., van Engeland, H., Kenemans, J. L., Camfferman, G., Buitelaar, J. K., & Koelega, H. S. (1999) · Psychophysiology · doi:10.1111/1469-8986.3640419 · Cited by 128 · see citing works
ANOVA: A Paradigm for Low Power and Misleading Measures of Effect Size?
Wilcox, R. R. (1995) · Review of Educational Research · doi:10.3102/00346543065001051 · Cited by 106 · see citing works
Self-citationHarwell, M. R., Rubinstein, E. N., Hayes, W. S., & Olds, C. C. (1992) · Journal of Educational Statistics · doi:10.2307/1165127 · Cited by 105 · see citing works
Phillips, G. C., Jones, G. E., Rieger, E. J., & Snell, J. B. (1999) · Psychophysiology · doi:10.1017/s0048577299980071 · Cited by 100 · see citing works
To Trim or Not to Trim: Tests of Location Equality Under Heteroscedasticityand Nonnormality
Lix, L. M., & Keselman, H. J. (1998) · Educational and Psychological Measurement · doi:10.1177/0013164498058003004 · Cited by 94 · see citing works
Self-citationERP indices of auditory selective attention in aging and Parkinson's disease
Karayanidis, F., Andrews, S., Ward, P. B., & Michie, P. T. (1995) · Psychophysiology · doi:10.1111/j.1469-8986.1995.tb01216.x · Cited by 88 · see citing works
Approximate degrees of freedom tests: A unified perspective on testing for mean equality.
Lix, L. M., & Keselman, H. J. (1995) · Psychological Bulletin · doi:10.1037/0033-2909.117.3.547 · Cited by 79 · see citing works
Self-citationRobustness properties of nonorthogonal analysis of variance.
Milligan, G. W., Wong, D., & Thompson, P. A. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.101.3.464 · Cited by 77 · see citing works
Testing Repeated Measures Hypotheses When Covariance Matrices are Heterogeneous
Keselman, H. J., Carriere, K. C., & Lix, L. M. (1993) · Journal of Educational Statistics · doi:10.3102/10769986018004305 · Cited by 69 · see citing works
Self-citationCan tests for treatment group equality be improved?: The bootstrap and trimmed means conjecture
Wilcox, R. R., Keselman, H. J., & Kowalchuk, R. K. (1998) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1998.tb00670.x · Cited by 65 · see citing works
Self-citationMultiple comparison procedures for trimmed means.
Keselman, H. J., Lix, L. M., & Kowalchuk, R. K. (1998) · Psychological Methods · doi:10.1037//1082-989x.3.1.123 · Cited by 48 · see citing works
Self-citationTesting treatment effects in repeated measures designs: Trimmed means and bootstrapping
Keselman, H. J., Kowalchuk, R. K., Algina, J. J., Lix, L. M., & Wilcox, R. R. (2000) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711000159286 · Cited by 46 · see citing works
Self-citationRobust Nonorthogonal Analyses Revisited: An Update Based on Trimmed Means
Keselman, H. J., Kowalchuk, R. K., & Lix, L. M. (1998) · Psychometrika · doi:10.1007/bf02294772 · Cited by 45 · see citing works
Self-citationRobust and Powerful Nonorthogonal Analyses
Keselman, H. J., Carriere, K. C., & Lix, L. M. (1995) · Psychometrika · doi:10.1007/bf02294383 · Cited by 42 · see citing works
Self-citationKeselman, H. J., Algina, J. J., Wilcox, R. R., & Kowa, R. K. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970998 · Cited by 41 · see citing works
Self-citationRepeated measures ANOVA: Some new results on comparing trimmed means and means
Wilcox, R. R., Keselman, H. J., Muska, J., et al. (2000) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711000159187 · Cited by 40 · see citing works
Self-citationAlgina, J. J., & Keselman, H. J. (1998) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986023002152 · Cited by 39 · see citing works
Self-citationMultiple comparison procedures for trimmed means.
Keselman, H. J., Lix, L. M., & Kowalchuk, R. K. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.1.123 · Cited by 38 · see citing works
Self-citationLix, L. M., Algina, J. J., & Keselman, H. J. (2003) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3804_1 · Cited by 37 · see citing works
Self-citationImplementing the Welch-James Procedure with Factorial Designs
Algina, J. J., & Olejnik, S. F. (1984) · Educational and Psychological Measurement · doi:10.1177/0013164484441004 · Cited by 35 · see citing works
Algina, J. J., & Keselman, H. J. (1997) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3203_2 · Cited by 25 · see citing works
Self-citationAnalysing multivariate repeated measures designs when covariance matrices are heterogeneous
Keselman, H. J., & Lix, L. M. (1997) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1997.tb01148.x · Cited by 25 · see citing works
Self-citationPairwise multiple comparisons: A model comparison approach versus stepwise procedures
Cribbie, R. A., & Keselman, H. J. (2003) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711003321645412 · Cited by 25 · see citing works
Self-citationTesting Repeated Measures Hypotheses When Covariance Matrices Are Heterogeneous
Keselman, H. J., Carriere, K. C., & Lix, L. M. (1993) · Journal of Educational Statistics · doi:10.2307/1165157 · Cited by 24 · see citing works
Self-citationInteraction contrasts in repeated measures designs
Lix, L. M., & Keselman, H. J. (1996) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1996.tb01079.x · Cited by 21 · see citing works
Self-citationImproved Repeated Measures Stepwise Multiple Comparison Procedures
Keselman, H. J., & Lix, L. M. (1995) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986020001083 · Cited by 15 · see citing works
Self-citationKeselman, H. J., Carriere, K., & Lix, L. M. (1996) · Psychometrika · doi:10.1007/bf02296967 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Neural responses to social and monetary reward in early adolescence and emerging adulthood
Ethridge, P., Kujawa, A., Dirks, M. A., Arfer, K. B., Kessel, E. M., Klein, D. N., & Weinberg, A. (2017) · Psychophysiology · doi:10.1111/psyp.12957 · Cited by 96 · see citing works
Kujawa, A., Kessel, E. M., Carroll, A., Arfer, K. B., & Klein, D. N. (2017) · Biological Psychology · doi:10.1016/j.biopsycho.2017.07.001 · Cited by 64 · see citing works
Dien, J., Brian, E. S., Molfese, D. L., & Gold, B. T. (2013) · Cortex · doi:10.1016/j.cortex.2013.03.008 · Cited by 59 · see citing works
Rauchbauer, B., Pfabigan, D. M., & Lamm, C. (2018) · Scientific Reports · doi:10.1038/s41598-018-30926-4 · Cited by 12 · see citing works
- Authors
- Keselman, H. J., Wilcox, R. R., & LIX, L. M.
- Affiliations
- University of Manitoba (CA)
- University of Southern California (US)
- Corresponding author
- Keselman, H. J.
- Venue
- Psychophysiology
- Publisher
- Wiley
- Published
- 2003-06-10
- Volume
- 40
- Issue
- 4
- Pages
- 586-596
- ISSN
- 0048-5772 (ISSN-L)
- 1469-8986
- OpenAlex
- W2171549256
- Cited by
- 110
- References
- 46
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 1 · 2023: 4 · 2022: 8 · 2021: 6 · 2020: 3 · 2019: 4 · 2018: 3 · 2017: 6 · 2016: 4 · 2015: 7 · 2014: 7 · 2013: 7 · 2012: 2
- FWCI
- 6.54
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychophysiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- robust hypothesis testing, trimmed mean, bootstrapping, type I error control, non-normality, heteroscedasticity, analysis of variance, psychophysiological data
- MeSH terms
- Algorithms, Data Interpretation, Statistical, Humans, Psychophysiology, Research Design, Reproducibility of Results, Least-Squares Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
