The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test
Open accessAbstract
Multiple comparisons tests (MCTs) include the statistical tests used to compare groups (treatments) often following a significant effect reported in one of many types of linear models. Due to a variety of data and statistical considerations, several dozen MCTs have been developed over the decades, with tests ranging from very similar to each other to very different from each other. Many scientific disciplines use MCTs, including >40,000 reports of their use in ecological journals in the last 60 years. Despite the ubiquity and utility of MCTs, several issues remain in terms of their correct use and reporting. In this study, we evaluated 17 different MCTs. We first reviewed the published literature for recommendations on their correct use. Second, we created a simulation that evaluated the performance of nine common MCTs. The tests examined in the simulation were those that often overlapped in usage, meaning the selection of the test based on fit to the data is not unique and that the simulations could inform the selection of one or more tests when a researcher has choices. Based on the literature review and recommendations: planned comparisons are overwhelmingly recommended over unplanned comparisons, for planned non-parametric comparisons the Mann-Whitney-Wilcoxon U test is recommended, Scheffé’s S test is recommended for any linear combination of (unplanned) means, Tukey’s HSD and the Bonferroni or the Dunn-Sidak tests are recommended for pairwise comparisons of groups, and that many other tests exist for particular types of data. All code and data used to generate this paper are available at: https://github.com/stevemidway/MultipleComparisons .
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Baltosser, W. H., & Zar, J. H. (1996) · Ecology · doi:10.2307/2265725 · Cited by 35,166
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
Mixed effects models and extensions in ecology with R
Zuur, A. F., Ieno, E. N., Walker, N., Saveliev, A. A., & Smith, G. M. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-87458-6 · Cited by 17,963
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 1×
“…and concerns over the use of these criteria in this manner are well documented in the literature (Stephens et al., 2005; Midway et al. (2020), PeerJ, DOI 10.7717/peerj.10387 16/26 Stephens, Buskirk & Del Rio, 2007;Murtaugh, 2014;Wasserstein & Lazar, 2016;Wasserstein, Schrim & Lazar, 2019).”
BackgroundInfluentialMultiple Comparisons Using Rank Sums
Dunn, O. J. (1964) · Technometrics · doi:10.1080/00401706.1964.10490181 · Cited by 5,085
XV.—The Correlation between Relatives on the Supposition of Mendelian Inheritance.
Fisher, R. A. (1919) · Transactions of the Royal Society of Edinburgh · doi:10.1017/s0080456800012163 · Cited by 4,387
Comparing Individual Means in the Analysis of Variance
Tukey, J. W. (1949) · Biometrics · doi:10.2307/3001913 · Cited by 4,246
Cited by
Biophysical characterization of high-confidence, small human proteins
Whited, A. M., Jungreis, I., Allen, J., Cleveland, C. L., Mudge, J. M., Kellis, M., Rinn, J. L., & Hough, L. E. (2024) · Biophysical Reports · doi:10.1016/j.bpr.2024.100167 · Cited by 5
Kim, Y., Kim, K., Lee, Y., Min, H., Ko, Y., Lee, B., Hur, K., Kim, S., Lee, S., & Jang, C. (2024) · British Journal of Pharmacology · doi:10.1111/bph.16427 · Cited by 4
Metadata
- Authors
- Midway, S., Robertson, M., Flinn, S., & Kaller, M.
- Venue
- PeerJ
- Publisher
- PeerJ
- Published
- 2020-12-04
- Volume
- 8
- Pages
- e10387
- ISSN
- 2167-8359 (ISSN-L)
- OpenAlex
- W3109355298
- Cited by
- 430
- References
- 44
- FWCI
- 20.38
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test. https://app.cochat.ai/papers/10.7717/peerj.10387
- The paper
- Midway, S., Robertson, M., Flinn, S., & Kaller, M. (2020). Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test. PeerJ. https://doi.org/10.7717/peerj.10387
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.