MULTIPLE COMPARISON PROCEDURES UPDATED

Abstract

SUMMARY: 1. A common statistical flaw in articles submitted to or published in biomedical research journals is to test multiple null hypotheses that originate from the results of a single experiment without correcting for the inflated risk of type 1 error (false positive statistical inference) that results from this. Multiple comparison procedures (MCP) are designed to minimize this risk. The present review focuses on pairwise contrasts, the most common sort of multiple comparisons made by biomedical investigators. 2. In an earlier review a variety of MCP were described and evaluated. It was concluded that an effective MCP should control the risk of family‐wise type 1 error, so as to ensure that not more than one hypothesis within a single family is falsely rejected. One‐step procedures based on the Bonferroni or Sidak inequalities do this. For continuous data and under normal distribution theory, so does the Tukey‐Kramer procedure for all possible pairwise contrasts of means and the Dunnett procedure for all possible pairwise contrasts of means with a control mean. 3. There is now a new class of MCP, based on the Bonferroni or Šidák inequalities but performed in a step‐wise fashion. The members of this class have certain desirable properties. They: (i) control the family‐wise type 1 error rate as effectively as the one‐step procedures; (ii) are more powerful than the one‐step Bonferroni or Sidak procedures, especially when hypotheses are logically related; and (iii) can be applied not only to continuous data but also to ordinal or categorical data. 4. Of the new step‐wise MCP, Holm's step‐down procedures are commended for their combination of accuracy, power and versatility. They also have the virtue of simplicity. Given the raw P values that result from conventional tests of significance, the adjustments for multiple comparisons can be made by hand or hand‐held calculator. 5. Despite the corrective abilities of the new step‐wise MCP, investigators should try to design their experiments and analyses to test a single, global hypothesis rather than multiple ones.