Evaluation of Experiments with Adaptive Interim Analyses

Abstract

A general method for statistical testing in experiments with an adaptive interim analysis is proposed. The method is based on the observed error probabilities from the disjoint subsamples before and after the interim analysis. Formally, an intersection of individual null hypotheses is tested by combining the two p-values into a global test statistic. Stopping rules for Fisher's product criterion in terms of critical limits for the p-value in the first subsample are introduced, including early stopping in the case of missing effects. The control of qualitative treatment-stage interactions is considered. A generalization to three stages is outlined. The loss of power when using the product criterion instead of the optimal classical test on the whole sample is calculated for the test of the mean of a normal distribution, depending on increasing proportions of the first subsample in relation to the total sample size. An upper bound on the loss of power due to early stopping is derived. A general example is presented and rules for assessing the sample size in the second stage of the trial are given. The problems of interpretation and precautions to be taken for applications are discussed. Finally, the sources of bias for estimation in such designs are described.