The Philosophy of Multiple Comparisons
Open accessAbstract
This paper is based on the 1989 Miller Memorial Lecture at Stanford University. The topic was chosen because of Rupert Miller's long involvement and significant contributions to multiple comparison procedures and theory. Our emphasis will be on the major questions that have received relatively little attention--on what one wants multiple comparisons to do, on why one wants to do that, and on how one can communicate the results. Very little attention will be given to how the results can be calculated--after all, there are books about that (e.g., Miller, 1966, 1981; Hochberg and Tamhane, 1987).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In many cases, researchers aim to decide whether significant results are consistent or inconsistent with directional (dividing) hypotheses.
“in many cases, researchers aim to decide whether significant results are consistent or inconsistent with directional (dividing) hypotheses”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Tukey, 1991, p. 100
The Significance section (p. 1) says the first question should be whether researchers can confidently determine the direction of a difference—up, down, or uncertain. The Directions or Intervals section (p. 5) also discusses interest in confident directions. These passages support the directional focus, but they do not describe deciding whether significant results are consistent or inconsistent with “dividing” hypotheses. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 5
- BRAUN, H. I. and TUKEY, J. W. (1983). Multiple comparisons through orderly partitions: The maximum subrange procedure. In Principals of Modern Psychological Measurement: A Festschrift for Frederic M. Lord (H. Wainer and S. Messick, eds.) 55–65. Erlbaum, Hillsdale, N.J.
- FisHER, R. A. (1926). The arrangement of field experiments. J. Min. Agric. G. Br. 33 503-513. (Also pages 83-94 in Volume 2 of his collected papers.)
- HocHBERG, Y. and TAmHANE, A. C. (1987). Multiple Comparison Procedures. Wiley, New York.
- KuíN, T. S. (1970). The Structure of Scientific Revolutions, 2nd ed. Foundations of the Unity of Science Series 2 (2) Univ. Chicago Press.
- MILLER, R. G. (1966). Simultaneous Statistical Inference, McGraw Hill, New York. (2nd ed., Springer, New York, 1981.)
Cited by
Sorted by citations
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
An effect size primer: A guide for clinicians and researchers.
Ferguson, C. J. (2009) · Professional Psychology: Research and Practice · doi:10.1037/a0015808 · Cited by 3,467 · see citing works
Practical significance: a concept whose time has come
Kirk, R. E. (1996) · Educational and Psychological Measurement · doi:10.1177/0013164496056005002 · Cited by 1,778 · see citing works
<b>Research Commentary</b> —Too Big to Fail: Large Samples and the <i>p</i> -Value Problem
Lin, M., Lucas, H. C., & Shmueli, G. (2013) · Information Systems Research · doi:10.1287/isre.2013.0480 · Cited by 972 · see citing works
McShane, B. B., Böckenholt, U., & Hansen, K. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616662243 · Cited by 398 · see citing works
When should we use one-tailed hypothesis testing?
Ruxton, G. D., & Neuhäuser, M. (2010) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2010.00014.x · Cited by 269 · see citing works
Meta-analysis in Second Language Research: Choices and Challenges
Oswald, F. L., & Plonsky, L. (2010) · Annual Review of Applied Linguistics · doi:10.1017/s0267190510000115 · Cited by 237 · see citing works
Dropping out of school: A review of intervention programs
Prevatt, F., & Kelly, F. D. (2003) · Journal of School Psychology · doi:10.1016/s0022-4405(03)00087-6 · Cited by 136 · see citing works
Misprescription and misuse of one-tailed tests
LOMBARDI, C. M., & HURLBERT, S. H. (2009) · Austral Ecology · doi:10.1111/j.1442-9993.2009.01946.x · Cited by 93 · see citing works
Malignant side effects of null-hypothesis significance testing
Branch, M. (2014) · Theory & Psychology · doi:10.1177/0959354314525282 · Cited by 81 · see citing works
Kriska, A. M., Rockette-Wagner, B., Edelstein, S. L., Bray, G. A., Delahanty, L. M., Hoskin, M. A., Horton, E. S., Venditti, E. M., Knowler, W. C., DPP Research Group, et al. (2021) · Diabetes Care · doi:10.2337/dc20-1129 · Cited by 70 · see citing works
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61 · see citing works
“many cases, researchers aim to decide whether significant results are consistent or inconsistent with directional (dividing) hypotheses (e.g., Cortina & Dunlap, 1997, p. 168; Cox, 1977, p. 51-52; Tukey, 1991, p. 100). In these cases, an optional stopping rule of “test until significant” is independent from the res…”
BackgroundResultA phylogenetic approach to comparative genomics
Dewar, A. E., Belcher, L. J., & West, S. A. (2025) · Nature Reviews Genetics · doi:10.1038/s41576-024-00803-0 · Cited by 41 · see citing works
Single-case experimental research designs.
Kazdin, A. E. (2015) · Methodological issues and strategies in clinical research (4th ed.). · doi:10.1037/14805-029 · Cited by 39 · see citing works
Rockette-Wagner, B., Storti, K. L., Dabelea, D., Edelstein, S., Florez, H., Franks, P. W., Montez, M. G., Pomeroy, J., & Kriska, A. M. (2017) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.10.007 · Cited by 21 · see citing works
Radin, D., Wahbeh, H., Michel, L., & Delorme, A. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.00726 · Cited by 11 · see citing works
- Authors
- Tukey, J. W.
- Corresponding author
- Tukey, John W.
- Venue
- Statistical Science
- Publisher
- Institute of Mathematical Statistics
- Published
- 1991-02-01
- Volume
- 6
- Issue
- 1
- Pages
- 100-116
- ISSN
- 0883-4237 (ISSN-L)
- 2168-8745
- OpenAlex
- W2073997076
- Cited by
- 773
- References
- 0
- Citations by year
- 2026: 14 · 2025: 31 · 2024: 32 · 2023: 26 · 2022: 34 · 2021: 54 · 2020: 44 · 2019: 37 · 2018: 37 · 2017: 25 · 2016: 25 · 2015: 31 · 2014: 21 · 2013: 31 · 2012: 37
- FWCI
- 7.56
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Statistical Science (publishedVersion)
- Project Euclid (Cornell University) (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- multiple comparisons, philosophy of statistics, statistical inference, communication of results, family-wise error rate, significance testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
