What is the proper way to apply the multiple comparison test?

Abstract

Multiple Comparison Test and Its ImitationsWe are not always interested in comparison of two groups per experiment.Sometimes (in practice, very often), we may have to determine whether differences exist among the means of three or more groups.The most common analytical method used for such determinations is analysis of variance (ANO-VA).1) When the null hypothesis (H 0 ) is rejected after ANOVA, that is, in the case of three groups, H 0 : μ A = μ B = μ C , we do not know how one group differs from a certain group.The result of ANOVA does not provide detailed information regarding the differences among various combinations of groups.Therefore, researchers usually perform additional analysis to clarify the differences between particular pairs of experimental groups.If the null hypothesis (H 0 ) is rejected in the ANOVA for the three groups, the following cases are considered:In which of these cases is the null hypothesis rejected?The only way to answer this question is to apply the 'multiple comparison test' (MCT), which is sometimes also called a 'post-hoc test.' Statistical RoundMultiple comparisons tests (MCTs) are performed several times on the mean of experimental conditions.When the null hypothesis is rejected in a validation, MCTs are performed when certain experimental conditions have a statistically significant mean difference or there is a specific aspect between the group means.A problem occurs if the error rate increases while multiple hypothesis tests are performed simultaneously.Consequently, in an MCT, it is necessary to control the error rate to an appropriate level.In this paper, we discuss how to test multiple hypotheses simultaneously while limiting type I error rate, which is caused by α inflation.To choose the appropriate test, we must maintain the balance between statistical power and type I error rate.If the test is too conservative, a type I error is not likely to occur.However, concurrently, the test may have insufficient power resulted in increased probability of type II error occurrence.Most researchers may hope to find the best way of adjusting the type I error rate to discriminate the real differences between observed data without wasting too much statistical power.It is expected that this paper will help researchers understand the differences between MCTs and apply them appropriately.