Using the Student's t-test with extremely small sample sizes
Open accessThe DOI matches its DataCite record, with a small difference in title or year; 54 of the 54 references it cites were found in a registry or an index; none of the 10 sources its claims cite is checked for support yet.
Abstract
Researchers occasionally have to work with an extremely small sample size, defined herein as N ≤ 5. Some methodologists have cautioned against using the t-test when the sample size is extremely small, whereas others have suggested that using the t-test is feasible in such a case. The present simulation study estimated the Type I error rate and statistical power of the one- and two-sample t-tests for normally distributed populations and for various distortions such as unequal sample sizes, unequal variances, the combination of unequal sample sizes and unequal variances, and a lognormal population distribution. Ns per group were varied between 2 and 5. Results show that the t-test provides Type I error rates close to the 5% nominal value in most of the cases, and that acceptable power (i.e., 80%) is reached only if the effect size is very large. ... Compared to the regular t-test, the Welch test tends to reduce statistical power and the t-testR yields false positive rates that deviate from 5%. This study further shows that a paired t-test is feasible with extremely small Ns if the within-pair correlation is high. It is concluded that there are no principal.objections to using a t-test with Ns as small as 2. A final cautionary note is made on the credibility of research findings when sample sizes are small. Accessed 123,336 times on https://pareonline.net from August 06, 2013 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right.
Overview
CoChat Summary
TL;DR
This simulation study evaluates the Student's t-test for extremely small samples (N ≤ 5) and finds no fundamental objection to its use, provided the true effect size is very large. Rank-transformed t-tests and the Welch test are generally not recommended for such samples.
Key findings
- For equal sample sizes, the regular t-test performed acceptably in most simulated cases: Type I error rates were close to 5%, and for N = M = 2 acceptable power (>80%) was reached at D ≥ 12 for the one-sample t-test and D ≥ 6 for the two-sample t-test. For N = M = 3, the null was rejected in >80% of cases at D ≥ 4; for N = M = 5, power was 91.0% (one-sample) and 79.0% (two-sample) at D = 2.
- The rank-transformed t-test (t-testR) was not recommended: it had 0% power for N = M = 2, and high Type I error rates such as 9.9% for N = M = 3 under normality, 15.7% under unequal variances, and 9.9% under a lognormal distribution.
- The Welch test generally lost statistical power compared with the regular t-test, though its Type I error rates stayed near or below 5%; it was preferred only when unequal variances and unequal sample sizes occurred with the smaller sample drawn from the smaller-variance population.
- The paired t-test was feasible with N = 3 when the within-pair correlation was high (r > 0.8); negative correlations diminished power. High false-positive rates occurred for the one-sample t-test on non-normal data (15.3%) and for unequal sample sizes combined with unequal variances when the larger sample came from the low-variance population (t-test and t-testR false-positive rates >27%; Welch power only 76.5% at D = 20).
Methods
- Simulation study conducted in MATLAB (Version R2010b) with 100,000 repetitions per case; two-tailed p < 0.05 was considered significant.
- Samples were drawn from normal populations (mean 0, SD 1) with mean shifts D from 0 to 40, for N = 2, 3, and 5; one-sample, two-sample, and paired t-tests were evaluated. Additional scenarios included unequal variances (variance ratio 16), unequal sample sizes (N = 2, M = 5), a lognormal distribution (skewness 5.70), varying skewness across 20 values, and paired N = 3 with within-pair correlations from −0.99 to 0.99.
- The regular t-test was compared with a t-test on ranked data (t-testR) and the Welch test; a permutation-test follow-up was also mentioned.
Limitations
- The paper states that a frequentist perspective was used and cautions that from a Bayesian perspective small samples may still be problematic, contributing to false positives and inflated effect sizes; it also states that large samples are always preferred given infinite time and resources.
- Evident from the text: the conclusions are based on simulations with specific distributions and parameter values, not empirical data, and the recommended power for extremely small samples requires very large effect sizes (e.g., D ≥ 6 for N = M = 2 two-sample), which the paper notes are uncommon in behavioral/psychological research.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 1 Not valid · 7 Not checked · 68 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 1 further shows that the t-testR has zero power for any effect size when N = M = 2.”
§ Using the Student's t-test with extremely small sample sizes > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 also reveal that the Welch t-test is not preferred; statistical power is lower as compared to the regular t-test.”
§ Using the Student's t-test with extremely small sample sizes > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Welch test again reduces statistical power as compared to the t-test.”
§ Using the Student's t-test with extremely small sample sizes > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It can be seen that Type I error rates are relatively independent of the degree of skewness (Figure 2).”
§ Using the Student's t-test with extremely small sample sizes > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, for D = 1, the power is 57.4%, 39.8%, and 30.4%, for the t-testR, regular t-test, and Welch test, respectively.”
§ Using the Student's t-test with extremely small sample sizes > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
49 more claims
Create a free account to see all 76 claims and every source verdict.
- It may state two separate claims as one. 13
- The quote only partly states this claim. 5
- Probably not a substantial claim. 5
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Using the Student's t-test with extremely small sample sizes > Discussion
“To the credit of the Welch test, the Type I error rates never deviated far from the nominal 5% value, and were in several cases substantially below 5%.”
§ Using the Student's t-test with extremely small sample sizes > Results
“N = 2 (small variance), M = 5 (large variance) N = 5 (small variance), M = 2 (large variance) D ttest (2 sample) t-testR (2 sample) Welch (2 sample) D ttest (2 sample) t-testR (2 sample) Welch (2 sample) 0 0.014 0.046 0.041 0 0.271 0.309 0.104”
Citations
Cites
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,245
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Effect size estimates: Current use, calculations, and interpretation.
Fritz, C. O., Morris, P. E., & Richler, J. J. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0024338 · Cited by 5,199
Applied Nonparametric Statistics
Wozniak, P. J., & Daniel, W. W. (1991) · Technometrics · doi:10.2307/1268795 · Cited by 4,987
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
Rank transformations as a bridge between parametric and nonparametric statistics
Conover, W. J., & Iman, R. L. (1981) · The American Statistician · doi:10.1080/00031305.1981.10479327 · Cited by 3,922
Siegel, S. (1957) · The American Statistician · doi:10.1080/00031305.1957.10501091 · Cited by 3,903
The On-Line Encyclopedia of Integer Sequences
Sloane, N. J. A. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-73086-6_12 · Cited by 3,225
NONPARAMETRIC STATISTICS FOR THE BEHAVIORAL SCIENCES
Sidney, S. (1957) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-195707000-00032 · Cited by 2,230
Research works printed but not linked: 1
- % vector of within-pair Pearson correlations DD=[0 ; reps=100000; p1=NaN(length(DD),length(RR),reps);
Cited by
Statistics in Corpus Linguistics: A Practical Guide
Brezina, V. (2018) · Cambridge University Press eBooks · doi:10.1017/9781316410899 · Cited by 648
Excess TGF-β mediates muscle weakness associated with bone metastases in mice
Waning, D. L., Mohammad, K. S., Reiken, S., Xie, W., Andersson, D. C., John, S., Chiechi, A., Wright, L. E., Umanskaya, A., Niewolna, M., et al. (2015) · Nature Medicine · doi:10.1038/nm.3961 · Cited by 378
The failing heart utilizes 3-hydroxybutyrate as a metabolic stress defense
Horton, J. L., Davidson, M. T., Kurishima, C., Vega, R. B., Powers, J. C., Matsuura, T. R., Petucci, C., Lewandowski, E. D., Crawford, P. A., Muoio, D. M., et al. (2019) · JCI Insight · doi:10.1172/jci.insight.124079 · Cited by 364
Dwivedi, A. K., Mallawaarachchi, I., & Alvarado, L. A. (2017) · Statistics in Medicine · doi:10.1002/sim.7263 · Cited by 334
Galanos, P., Vougas, K., Walter, D., Polyzos, A., Maya-Mendoza, A., Haagensen, E. J., Kokkalis, A., Roumelioti, F.-M., Gagos, S., Tzetis, M., et al. (2016) · Nature Cell Biology · doi:10.1038/ncb3378 · Cited by 308
Metadata
- Authors
- de Winter, J. C. F.
- Venue
- University of Massachusetts Amherst
- Publisher
- University of Massachusetts Amherst
- Published
- 2013
- Volume
- 18
- Issue
- 10
- Pages
- 10
- OpenAlex
- W2441124914
- Cited by
- 992
- References
- 48
- Citations by year
- 2026: 12 · 2025: 59 · 2024: 118 · 2023: 108 · 2022: 92 · 2021: 109 · 2020: 72 · 2019: 76 · 2018: 91 · 2017: 88 · 2016: 84 · 2015: 61 · 2014: 22
- FWCI
- 67.06
- Type
- journal article
- Language
- en
- Open access
- Licence
- cc-by-nc-nd
- Locations
- ScholarWorks@UMassAmherst (University of Massachusetts Amherst) (submittedVersion, cc-by-nc-nd)
- Research Repository (Delft University of Technology) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- University of Massachusetts (UMass) Amherst
- Practical assessment, research & evaluation
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- Student's t-test, type I error rate, statistical power, Welch test, paired t-test, unequal sample sizes, heteroscedasticity, lognormal distribution, effect size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using the Student's t-test with extremely small sample sizes. https://app.cochat.ai/papers/10.7275/e4r6-dj05
- The paper
- de Winter, J. C. F. (2013). Using the Student's t-test with extremely small sample sizes. University of Massachusetts Amherst. https://doi.org/10.7275/e4r6-dj05
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.