The Efficiency of Simulation-Based Multiple Comparisons
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
A frequently encountered problem in practice is that of simultaneous interval estimation of p linear combinations of a parameter beta in the setting of (or equivalent to) a univariate linear model. This problem has been solved adequately only in a few settings when the covariance matrix of the estimator is diagonal; in other cases, conservative solutions can be obtained by the methods of Scheffé, Bonferroni, or Sidák (1967, Journal of the American Statistical Association 62, 626-633). Here we investigate the efficiency of using a simulated critical point for exact intervals, which has been suggested before but never put to serious test. We find the simulation-based method to be completely reliable and essentially exact. Sample size savings are substantial (in our settings): 3-19% over the Sidák method, 4-37% over the Bonferroni method, and 27-33% over the Scheffé method. We illustrate the efficiency and flexibility of the simulation-based method with case studies in physiology and marine ecology.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
David, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
A Multiple Comparison Procedure for Comparing Several Treatments with a Control
Dunnett, C. W. (1955) · Journal of the American Statistical Association · doi:10.1080/01621459.1955.10501294 · Cited by 6,094
Multiple comparisons among means
Dunn, O. J. (1961) · Journal of the American Statistical Association · doi:10.1080/01621459.1961.10482090 · Cited by 5,006
Extension of Multiple Range Tests to Group Means with Unequal Numbers of Replications
Kramer, C. Y. (1956) · Biometrics · doi:10.2307/3001469 · Cited by 3,146
Rectangular Confidence Regions for the Means of Multivariate Normal Distributions
Šidák, Z. (1967) · Journal of the American Statistical Association · doi:10.1080/01621459.1967.10482935 · Cited by 2,639
Modified Randomization Tests for Nonparametric Hypotheses
Dwass, M. (1957) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177707045 · Cited by 737
A Proof of the Conjecture that the Tukey-Kramer Multiple Comparisons Procedure is Conservative
Hayter, A. J. (1984) · The Annals of Statistics · doi:10.1214/aos/1176346392 · Cited by 390
Pairwise Multiple Comparisons in the Homogeneous Variance, Unequal Sample Size Case
Dunnett, C. W. (1980) · Journal of the American Statistical Association · doi:10.1080/01621459.1980.10477551 · Cited by 377
Stoline, M. R. (1981) · The American Statistician · doi:10.1080/00031305.1981.10479331 · Cited by 280
Finite Sample Properties and Asymptotic Efficiency of Monte Carlo Tests
Jockel, K.-H. (1986) · The Annals of Statistics · doi:10.1214/aos/1176349860 · Cited by 158
Juvenile spot (Pisces) and grass shrimp predation on meiobenthos in muddy and sandy substrata
Smith, L. D., & Coull, B. C. (1987) · Journal of Experimental Marine Biology and Ecology · doi:10.1016/0022-0981(87)90167-5 · Cited by 83
Exact Simultaneous Confidence Intervals for Pairwise Comparisons of Three Normal Means
Spurrier, J. D., & Isham, S. P. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10478137 · Cited by 34
Exact Simultaneous Confidence Intervals for Multiple Comparisons Among Three or Four Mean Values
Uusipaikka, E. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10477160 · Cited by 31
Exact simulation-based inference: A survey, with additions
Edwards, D. (1985) · Journal of Statistical Computation and Simulation · doi:10.1080/00949658508810853 · Cited by 28
Self-citationBlood and isoproterenol reduce capillary permeability in cat hindlimb
WATSON, P. D., Wolf, M. B., & Beck-Montgomery, I. S. (1987) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.1987.252.1.h47 · Cited by 10
Simultaneous tests for finite families of hypotheses
Foutz, R. V. (1981) · Communication in Statistics- Theory and Methods · doi:10.1080/03610928108828109 · Cited by 8
Cited by
An Algorithm for a Letter-Based Representation of All-Pairwise Comparisons
Piepho, H.-P. (2004) · Journal of Computational and Graphical Statistics · doi:10.1198/1061860043515 · Cited by 484
A Hitchhiker's Guide to Mixed Models for Randomized Experiments
Piepho, H. P., Büchse, A., & Emrich, K. (2003) · Journal of Agronomy and Crop Science · doi:10.1046/j.1439-037x.2003.00049.x · Cited by 368
4-1BB-mediated immunotherapy of rheumatoid arthritis
Seo, S. K., Choi, J. H., Kim, Y. H., Kang, W. J., Park, H. Y., Suh, J. H., Choi, B. K., Vinay, D. S., & Kwon, B. S. (2004) · Nature Medicine · doi:10.1038/nm1107 · Cited by 325
Brain-to-brain coupling during handholding is associated with pain reduction
Goldstein, P., Weissman-Fogel, I., Dumas, G., & Shamay-Tsoory, S. G. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1703643115 · Cited by 311
“Correlation analysis of pain-related outcomes with clustered interpartner coupling networks was adjusted by simulation-based multiple-tests correction (127) and includes a report of 95% confidence intervals.”
MethodDi Lazzaro, V., Dileone, M., Pilato, F., Capone, F., Musumeci, G., Ranieri, F., Ricci, V., Bria, P., Di Iorio, R., de Waure, C., et al. (2011) · Journal of Neurophysiology · doi:10.1152/jn.00781.2010 · Cited by 303
Metadata
- Authors
- Edwards, D., & Berry, J. J.
- Affiliations
- University of South Carolina (US)
- Venue
- Biometrics
- Publisher
- JSTOR
- Published
- 1987-12
- Volume
- 43
- Issue
- 4
- Pages
- 913
- ISSN
- 0006-341X (ISSN-L)
- OpenAlex
- W1999311530
- Cited by
- 298
- References
- 16
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 8 · 2023: 6 · 2022: 10 · 2021: 10 · 2020: 12 · 2019: 5 · 2018: 16 · 2017: 21 · 2016: 17 · 2015: 18 · 2014: 16 · 2013: 14 · 2012: 10
- FWCI
- 2.12
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Biometrics (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (5r01hl024314-11, 1r01gm034870-01)
- National Heart, Lung, and Blood Institute (R01 HL24314)
- National Institute of General Medical Sciences (GM36977-01, R01 GM34870-01)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Management Science and Operations Research › Simulation Techniques and Applications
- Keywords
- multiple comparisons, linear combinations of parameters, exact confidence intervals, Bonferroni correction, Scheffe method, sample size reduction
- MeSH terms
- Analysis of Variance, Animals, Capillary Permeability, Cats, Computer Simulation, Fishes, Mathematics, Models, Biological, Predatory Behavior, Probability, Research Design, Decapoda
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.