CoChat

Using the Student's t-test with extremely small sample sizes

Open access
de Winter, J.C.F. · 2013 · University of Massachusetts Amherst

The DOI matches its DataCite record, with a small difference in title or year; 54 of the 54 references it cites were found in a registry or an index; none of the 10 sources its claims cite is checked for support yet.

Abstract

Researchers occasionally have to work with an extremely small sample size, defined herein as N ≤ 5. Some methodologists have cautioned against using the t-test when the sample size is extremely small, whereas others have suggested that using the t-test is feasible in such a case. The present simulation study estimated the Type I error rate and statistical power of the one- and two-sample t-tests for normally distributed populations and for various distortions such as unequal sample sizes, unequal variances, the combination of unequal sample sizes and unequal variances, and a lognormal population distribution. Ns per group were varied between 2 and 5. Results show that the t-test provides Type I error rates close to the 5% nominal value in most of the cases, and that acceptable power (i.e., 80%) is reached only if the effect size is very large. ... Compared to the regular t-test, the Welch test tends to reduce statistical power and the t-testR yields false positive rates that deviate from 5%. This study further shows that a paired t-test is feasible with extremely small Ns if the within-pair correlation is high. It is concluded that there are no principal.objections to using a t-test with Ns as small as 2. A final cautionary note is made on the credibility of research findings when sample sizes are small. Accessed 123,336 times on https://pareonline.net from August 06, 2013 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right.

Overview

CoChat Summary

TL;DR

This simulation study evaluates the Student's t-test for extremely small samples (N ≤ 5) and finds no fundamental objection to its use, provided the true effect size is very large. Rank-transformed t-tests and the Welch test are generally not recommended for such samples.

Key findings

  • For equal sample sizes, the regular t-test performed acceptably in most simulated cases: Type I error rates were close to 5%, and for N = M = 2 acceptable power (>80%) was reached at D ≥ 12 for the one-sample t-test and D ≥ 6 for the two-sample t-test. For N = M = 3, the null was rejected in >80% of cases at D ≥ 4; for N = M = 5, power was 91.0% (one-sample) and 79.0% (two-sample) at D = 2.
  • The rank-transformed t-test (t-testR) was not recommended: it had 0% power for N = M = 2, and high Type I error rates such as 9.9% for N = M = 3 under normality, 15.7% under unequal variances, and 9.9% under a lognormal distribution.
  • The Welch test generally lost statistical power compared with the regular t-test, though its Type I error rates stayed near or below 5%; it was preferred only when unequal variances and unequal sample sizes occurred with the smaller sample drawn from the smaller-variance population.
  • The paired t-test was feasible with N = 3 when the within-pair correlation was high (r > 0.8); negative correlations diminished power. High false-positive rates occurred for the one-sample t-test on non-normal data (15.3%) and for unequal sample sizes combined with unequal variances when the larger sample came from the low-variance population (t-test and t-testR false-positive rates >27%; Welch power only 76.5% at D = 20).

Methods

  • Simulation study conducted in MATLAB (Version R2010b) with 100,000 repetitions per case; two-tailed p < 0.05 was considered significant.
  • Samples were drawn from normal populations (mean 0, SD 1) with mean shifts D from 0 to 40, for N = 2, 3, and 5; one-sample, two-sample, and paired t-tests were evaluated. Additional scenarios included unequal variances (variance ratio 16), unequal sample sizes (N = 2, M = 5), a lognormal distribution (skewness 5.70), varying skewness across 20 values, and paired N = 3 with within-pair correlations from −0.99 to 0.99.
  • The regular t-test was compared with a t-test on ranked data (t-testR) and the Welch test; a permutation-test follow-up was also mentioned.

Limitations

  • The paper states that a frequentist perspective was used and cautions that from a Bayesian perspective small samples may still be problematic, contributing to false positives and inflated effect sizes; it also states that large samples are always preferred given infinite time and resources.
  • Evident from the text: the conclusions are based on simulations with specific distributions and parameter values, not empirical data, and the recommended power for extremely small samples requires very large effect sizes (e.g., D ≥ 6 for N = M = 2 two-sample), which the paper notes are uncommon in behavioral/psychological research.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.7275/e4r6-dj05

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 1 Not valid · 7 Not checked · 68 Own results and interpretations

49 more claims

Create a free account to see all 76 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 13
  • The quote only partly states this claim. 5
  • Probably not a substantial claim. 5

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 54 references

Research works printed but not linked: 1

  1. % vector of within-pair Pearson correlations DD=[0 ; reps=100000; p1=NaN(length(DD),length(RR),reps);

Cited by

See all 992

Metadata

Authors
de Winter, J. C. F.
Venue
University of Massachusetts Amherst
Publisher
University of Massachusetts Amherst
Published
2013
Volume
18
Issue
10
Pages
10
OpenAlex
W2441124914
Cited by
992
References
48
Citations by year
2026: 12 · 2025: 59 · 2024: 118 · 2023: 108 · 2022: 92 · 2021: 109 · 2020: 72 · 2019: 76 · 2018: 91 · 2017: 88 · 2016: 84 · 2015: 61 · 2014: 22
FWCI
67.06
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • ScholarWorks@UMassAmherst (University of Massachusetts Amherst) (submittedVersion, cc-by-nc-nd)
  • Research Repository (Delft University of Technology) (submittedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • University of Massachusetts (UMass) Amherst
  • Practical assessment, research & evaluation
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Keywords
Student's t-test, type I error rate, statistical power, Welch test, paired t-test, unequal sample sizes, heteroscedasticity, lognormal distribution, effect size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Using the Student's t-test with extremely small sample sizes. https://app.cochat.ai/papers/10.7275/e4r6-dj05
The paper
de Winter, J. C. F. (2013). Using the Student's t-test with extremely small sample sizes. University of Massachusetts Amherst. https://doi.org/10.7275/e4r6-dj05

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.