The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test
Open accessAbstract
Multiple comparisons tests (MCTs) include the statistical tests used to compare groups (treatments) often following a significant effect reported in one of many types of linear models. Due to a variety of data and statistical considerations, several dozen MCTs have been developed over the decades, with tests ranging from very similar to each other to very different from each other. Many scientific disciplines use MCTs, including >40,000 reports of their use in ecological journals in the last 60 years. Despite the ubiquity and utility of MCTs, several issues remain in terms of their correct use and reporting. In this study, we evaluated 17 different MCTs. We first reviewed the published literature for recommendations on their correct use. Second, we created a simulation that evaluated the performance of nine common MCTs. The tests examined in the simulation were those that often overlapped in usage, meaning the selection of the test based on fit to the data is not unique and that the simulations could inform the selection of one or more tests when a researcher has choices. Based on the literature review and recommendations: planned comparisons are overwhelmingly recommended over unplanned comparisons, for planned non-parametric comparisons the Mann-Whitney-Wilcoxon U test is recommended, Scheffé’s S test is recommended for any linear combination of (unplanned) means, Tukey’s HSD and the Bonferroni or the Dunn-Sidak tests are recommended for pairwise comparisons of groups, and that many other tests exist for particular types of data. All code and data used to generate this paper are available at: https://github.com/stevemidway/MultipleComparisons .
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Baltosser, W. H., & Zar, J. H. (1996) · Ecology · doi:10.2307/2265725 · Cited by 35,166
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
Mixed effects models and extensions in ecology with R
Zuur, A. F., Ieno, E. N., Walker, N., Saveliev, A. A., & Smith, G. M. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-87458-6 · Cited by 17,963
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 1×
“…and concerns over the use of these criteria in this manner are well documented in the literature (Stephens et al., 2005; Midway et al. (2020), PeerJ, DOI 10.7717/peerj.10387 16/26 Stephens, Buskirk & Del Rio, 2007;Murtaugh, 2014;Wasserstein & Lazar, 2016;Wasserstein, Schrim & Lazar, 2019).”
BackgroundInfluentialMultiple Comparisons Using Rank Sums
Dunn, O. J. (1964) · Technometrics · doi:10.1080/00401706.1964.10490181 · Cited by 5,085
XV.—The Correlation between Relatives on the Supposition of Mendelian Inheritance.
Fisher, R. A. (1919) · Transactions of the Royal Society of Edinburgh · doi:10.1017/s0080456800012163 · Cited by 4,387
Comparing Individual Means in the Analysis of Variance
Tukey, J. W. (1949) · Biometrics · doi:10.2307/3001913 · Cited by 4,246
Effect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835
Extension of Multiple Range Tests to Group Means with Unequal Numbers of Replications
Kramer, C. Y. (1956) · Biometrics · doi:10.2307/3001469 · Cited by 3,146
Moving to a World Beyond “ p < 0.05”
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019) · The American Statistician · doi:10.1080/00031305.2019.1583913 · Cited by 2,976
Rectangular Confidence Regions for the Means of Multivariate Normal Distributions
Šidák, Z. (1967) · Journal of the American Statistical Association · doi:10.1080/01621459.1967.10482935 · Cited by 2,639
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
Comparisons of Treatments After an Analysis of Variance in Ecology
Day, R. W., & Quinn, G. P. (1989) · Ecological Monographs · doi:10.2307/1943075 · Cited by 1,986
Bayesian estimation supersedes the t-test
Kruschke, J. K. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029146 · Cited by 1,530
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372
Applied Multivariate Statistics for the Social Sciences
Pituch, K. A., & Stevens, J. P. (2015) · Routledge eBooks · doi:10.4324/9781315814919 · Cited by 1,152
Some Practical Guidelines for Effective Sample Size Determination
Lenth, R. V. (2001) · The American Statistician · doi:10.1198/000313001317098149 · Cited by 1,139
Gallagher, C. (2007) · Journal of the American Statistical Association · doi:10.1198/jasa.2007.s238 · Cited by 1,070
Implementing false discovery rate control: increasing your power
Verhoeven, K. J. F., Simonsen, K. L., & McIntyre, L. M. (2005) · Oikos · doi:10.1111/j.0030-1299.2005.13727.x · Cited by 974
Pairwise Multiple Comparison Procedures with Unequal N’s and/or Variances: A Monte Carlo Study
Games, P. A., & Howell, J. F. (1976) · Journal of Educational Statistics · doi:10.3102/10769986001002113 · Cited by 924
A Bayes Rule for the Symmetric Multiple Comparisons Problem
Waller, R. A., & Duncan, D. B. (1969) · Journal of the American Statistical Association · doi:10.1080/01621459.1969.10501073 · Cited by 723
Significance tests for multiple comparison of proportions, variances, and other statistics.
Ryan, T. H. (1960) · Psychological Bulletin · doi:10.1037/h0044320 · Cited by 684
An Introduction to Categorical Data Analysis
Sprent, P. (2007) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2007.00506_2.x · Cited by 679
Time for some a priori thinking about post hoc testing
Ruxton, G. D., & Beauchamp, G. (2008) · Behavioral Ecology · doi:10.1093/beheco/arn020 · Cited by 599
Escaping the Bonferroni iron claw in ecological studies
García, L. V. (2004) · Oikos · doi:10.1111/j.0030-1299.2004.13046.x · Cited by 560
Foundations of Linear and Generalized Linear Models
Agresti, A. (2015) · CERN Document Server (European Organization for Nuclear Research) · Cited by 462
Information theory and hypothesis testing: a call for pluralism
STEPHENS, P. A., BUSKIRK, S. W., HAYWARD, G. D., & MARTÍNEZ DEL RIO, C. (2005) · Journal of Applied Ecology · doi:10.1111/j.1365-2664.2005.01002.x · Cited by 360
Robust Rank Procedures for the Behrens-Fisher Problem
Fligner, M. A., & Policello, G. E. (1981) · Journal of the American Statistical Association · doi:10.1080/01621459.1981.10477623 · Cited by 331
Murtaugh, P. A. (2014) · Ecology · doi:10.1890/13-0590.1 · Cited by 317
A Rank Sum Test for Comparing All Pairs of Treatments
Steel, R. G. D. (1960) · Technometrics · doi:10.1080/00401706.1960.10489894 · Cited by 316
Effect Size in Ecological Experiments: The Application of Biological Models in Meta‐Analysis
Osenberg, C. W., Sarnelle, O., & Cooper, S. D. (1997) · The American Naturalist · doi:10.1086/286095 · Cited by 278
Inference in ecology and evolution
Stephens, P. A., Buskirk, S. W., & del Rio, C. M. (2007) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2006.12.003 · Cited by 268
Cortina, J., & Nouri, H. (2000) · Sage eBooks · doi:10.4135/9781412984010 · Cited by 238
Analysis of Messy Data Volume 1: Designed Experiments, Second Edition
Milliken, G. A., & Johnson, D. E. (2009) · Chapman & Hall/CRC eBooks · doi:10.1201/ebk1584883340 · Cited by 223
Controlling the false discovery rate in ecological research
Garcı́a, L. V. (2003) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2003.08.011 · Cited by 177
Introduction to Probability and Statistics
(2012) · Structural Health Monitoring · doi:10.1002/9781118443118.ch6 · Cited by 158
Gelman, A. (2017) · Personality and Social Psychology Bulletin · doi:10.1177/0146167217729162 · Cited by 145
White, T. J. H., van der Ende, J., & Nichols, T. E. (2019) · Conservation Genetics · doi:10.1007/s10592-019-01178-0 · Cited by 110
Pogrow, S. (2019) · The American Statistician · doi:10.1080/00031305.2018.1549101 · Cited by 79
Brinster, R., Köttgen, A., Tayo, B. O., Schumacher, M. C., & Sekula, P. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2081-x · Cited by 36
McCabe, D. J., Hayes-Pontius, E. M., Canepa, A., Berry, K. S., & Levine, B. C. (2012) · Freshwater Science · doi:10.1899/11-080.1 · Cited by 17
Cited by
Biophysical characterization of high-confidence, small human proteins
Whited, A. M., Jungreis, I., Allen, J., Cleveland, C. L., Mudge, J. M., Kellis, M., Rinn, J. L., & Hough, L. E. (2024) · Biophysical Reports · doi:10.1016/j.bpr.2024.100167 · Cited by 5
Kim, Y., Kim, K., Lee, Y., Min, H., Ko, Y., Lee, B., Hur, K., Kim, S., Lee, S., & Jang, C. (2024) · British Journal of Pharmacology · doi:10.1111/bph.16427 · Cited by 4
Metadata
- Authors
- Midway, S., Robertson, M., Flinn, S., & Kaller, M.
- Venue
- PeerJ
- Publisher
- PeerJ
- Published
- 2020-12-04
- Volume
- 8
- Pages
- e10387
- ISSN
- 2167-8359 (ISSN-L)
- OpenAlex
- W3109355298
- Cited by
- 430
- References
- 44
- FWCI
- 20.38
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.