CoChat

The unequal variance t-test is an underused alternative to Student's t-test and the Mann–Whitney U test

Ruxton, Graeme D. · 2006 · Behavioral Ecology

The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.

Abstract

Authors are unaware that Student's t-test is unreliable when variances differ between underlying populations. Authors are aware of this but consider their samples to have similar variances. Authors believe than the Mann–Whitney U test can effectively substitute for Student's t-test when variances are unequal. Because the t distribution tends to the normal distribution for large sample sizes, authors may consider that their sample sizes are sufficiently large for concerns about unequal variance and nonnormality of the samples to be ignored. Histogram of the larger divided by the smaller variance for 35 t-tests and U tests in my sample from Behavioral Ecology for which the variances were provided in the paper. Note for ease of presentation, the following 3 variance ratios were not plotted: 9.0, 9.0, and 21.0. Histogram of the ratio of the highest over the lowest sample size for 61 Student's t-tests and Mann–Whitney U tests in my survey from Behavioral Ecology for which sample sizes were provided. For ease of presentation, the following four ratios were not plotted: 3.1, 3.3, 4.2, and 5.7. The unequal variance t-test does not make the assumption of equal variances. Coombs et al. (1996) presented measured Type I errors obtained by simulated sampling from normal distributions for the Student's t-test and the unequal variance t-test (their result are summarized in Table 1). In the examples in Table 1, we see that the Type I error rate of the unequal variance t-test never deviates far from the nominal 5% value, whereas the Type I error rate for the Student's t-test was over 3 times the nominal rate when the higher variance was associated with the smaller sample size and less than a quarter the nominal rate when the higher variance was associated with the higher sample size. These results concur qualitatively with other studies of these 2 tests (e.g., Zimmerman and Zumbo 1993). Notice that even when the variances are identical, the unequal variance t-test performs just as effectively as the Student's t-test in terms of Type I error. The power of the unequal variance t-test is similar to that of the Student's t-test even when the population variances are equal (e.g., Moser et al. 1989; Moser and Stevens 1992; Coombs et al. 1996). Hence, I suggest that the unequal variance t-test performs as well as, or better than, the Student's t-test in terms of control of both Type I and Type II error rates whenever the underlying distributions are normal. Calculated Type I error rate for the t-test and unequal variance t-test with a nominal α value of 0.05 (adapted from Coombs et al. 1996) Calculated Type I error rate for the t-test and unequal variance t-test with a nominal α value of 0.05 (adapted from Coombs et al. 1996) The unequal variance t-test has no performance benefits over the Student's t-test when the underlying population variances are equal. Hence, you might consider that an effective way to conduct your analysis would be to perform an initial test for homogeneity of variance and then perform either a Student's t-test when the variances are equal or an unequal variance t-test when they are not. The problem with this flexible approach is that the combination of this preliminary test plus whichever of the subsequent tests is ultimately used controls Type I error rates less well than simply always performing an unequal variance t-test on every occasion (Gans 1992; Moser and Stevens 1992), this is one reason why it is generally unwise to decide whether to perform one statistical test on the basis of the outcome of another (Zimmerman 2004 and references therein). There are further reasons for not recommending preliminary tests of variances (e.g., Markowski CA and Markowski EP 1990; Quinn and Keough 2002, p. 42). Hence, I suggest avoiding preliminary tests and adopting the unequal variance t-test unless an argument based on logical, physical, or biological grounds can be made as to why the variances are very likely to be identical for the 2 populations under investigation. It is important to remember that although the unequal variance t-test is more reliable than the Student's t-test in terms of violation of the assumption of homogeneity of variances, it is not necessarily any more reliable than the Student's t-test if the assumption of normality of the underlying populations is violated. However, Zimmerman and Zumbo (1993) argue that the unequal variance t-test performed on ranked data performs just as well as the Mann–Whitney U test (in terms of control of Type I errors) when variances are equal and considerably better than the U test when variances are unequal (see Table 2 for an example). This behavior was found when tested with populations coming from 8 different types of nonnormal distribution. Thus, Zimmerman and Zumbo (1993) suggest that the unequal variance t-test can effectively replace the Mann–Whitney U test if the data are first ranked before the test is applied. There are alternatives to the unequal varianc

Overview

Integrity signals

Registry

Crossref · 10.1093/beheco/ark016

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 1

  1. Zar JH. 1996. Biostatistical analysis. 3rd ed. London: Prentice Hall International.

Cited by

See all 1,988

Metadata

Authors
Ruxton, G. D.
Affiliations
University of Glasgow (GB)
Corresponding author
Ruxton, Graeme D.
Venue
Behavioral Ecology
Published
2006-05-17
Volume
17
Issue
4
Pages
688-690
ISSN
  • 1045-2249 (ISSN-L)
  • 1465-7279
OpenAlex
W2126552603
Cited by
1,988
References
15
Citations by year
2026: 162 · 2025: 178 · 2024: 122 · 2023: 111 · 2022: 129 · 2021: 145 · 2020: 167 · 2019: 131 · 2018: 132 · 2017: 117 · 2016: 124 · 2015: 105 · 2014: 80 · 2013: 66 · 2012: 57
FWCI
19.18
Type
journal article
Language
en
Open access
closed
Locations
  • Behavioral Ecology (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
  • HighWire Press Open Archive (submittedVersion)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
  • Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
Keywords
Student's t-test, Mann-Whitney test, heteroscedasticity, type I error rate, statistical power, sample size, statistical methods in ecology, nonparametric tests, research reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.