Comparing two small samples with an unstable, treatment-independent baseline

Abstract

Due to time and resource constraints, small samples (N=3-7 cases per group) are often used in neurobiological studies that employ multiple techniques. In a simulation study, five statistical tests were used to compare two small samples (treated and control) with an unstable, additive baseline. These five tests differed in the way that they used the baseline variable (B) to adjust or normalize the variable affected by the treatment (Y). We conclude that, if N=3 or 4, the independent t-test on Y-B tends to have the highest power; if N> or =7, ANCOVA on Y with B as the covariate tends to have the highest power; and both tests have comparably high power if N=5 or 6. The Wilcoxon rank-sum test (or, equivalently, the Mann-Whitney test) has precisely zero power if one group has 3 cases and the other has 3 or 4 cases. Some other problems of small-sample analysis are considered.