The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Permutation tests for multi-factorial analysis of variance
Open accessAbstract
Several permutation strategies are often possible for tests of individual terms in analysis-of-variance (ANOVA) designs. These include restricted permutations, permutation of whole groups of units, permutation of some form of residuals or some combination of these. It is unclear, especially for complex designs involving random factors, mixed models or nested hierarchies, just which permutation strategy should be used for any particular test. The purpose of this paper is two-fold: (i) we provide a guideline for constructing an exact permutation strategy, where possible, for any individual term in any ANOVA design; and (ii) we provide results of Monte Carlo simulations to compare the level accuracy and power of different permutation strategies in two-way ANOVA, including random and mixed models, nested hierarchies and tests of interaction terms. Simulation results showed that permutation of residuals under a reduced model generally had greater power than the exact test or alternative approximate permutation methods (such as permutation of raw data). In several cases, restricted permutations, in particular, suffered more than other procedures, in terms of loss of power, challenging the conventional wisdom of using this approach. Our simulations also demonstrated that the choice of correct exchangeable units under the null hypothesis, in accordance with the guideline we provide, is essential for any permutation test, whether it be an exact test or an approximate test. For reference, we also provide appropriate permutation strategies for individual terms in any two-way or three-way ANOVA for the exact test (where possible) and for the approximate test using permutation of residuals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 26,755
(2005) · Statistics for Biology and Health · doi:10.1007/0-387-27255-0_9 · Cited by 20,133
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
Non‐parametric multivariate analyses of changes in community structure
CLARKE, K. R. (1993) · Australian Journal of Ecology · doi:10.1111/j.1442-9993.1993.tb00438.x · Cited by 14,128
Excoffier, L., Smouse, P. E., & Quattro, J. M. (1992) · Genetics · doi:10.1093/genetics/131.2.479 · Cited by 13,929
A new method for non-parametric multivariate analysis of variance
Anderson, M. J. (2001) · Austral Ecology · doi:10.1111/j.1442-9993.2001.01070.pp.x · Cited by 13,828
Self-citationter Braak, C. J. F., & Šmilauer, P. (2002) · Cited by 10,579
Self-citationNelder, J. A., & Wedderburn, R. W. M. (1972) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344614 · Cited by 7,194
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
SCHEFFE, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,516
Generalized Linear Models (2nd ed.).
Therneau, T. M., McCullagh, P., & Nelder, J. A. (1993) · Journal of the American Statistical Association · doi:10.2307/2290358 · Cited by 4,907
Approximate Inference in Generalized Linear Mixed Models
Breslow, N. E., & Clayton, D. G. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10594284 · Cited by 4,629
Randomization, Bootstrap and Monte Carlo Methods in Biology
Manly, B. F. J. (2018) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315273075 · Cited by 4,194
FITTING MULTIVARIATE MODELS TO COMMUNITY DATA: A COMMENT ON DISTANCE-BASED REDUNDANCY ANALYSIS
McArdle, B. H., & Anderson, M. J. (2001) · Ecology · doi:10.1890/0012-9658(2001)082[0290:fmmtcd]2.0.co;2 · Cited by 4,001
Self-citationter Braak, C. J. F., & Šmilauer, P. (1998) · Socio-Environmental Systems Modeling · Cited by 3,383
Self-citationEdgington, E., Edgington, E., & Onghena, P. (2007) · doi:10.1201/9781420011814 · Cited by 2,030
Edgington, E. S. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess2169 · Cited by 2,028
Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses
Niknian, M. (1995) · Technometrics · doi:10.1080/00401706.1995.10484341 · Cited by 1,462
Design and Analysis of Experiments
(2007) · Modern Engineering Statistics · doi:10.1002/9780470187548.ch12 · Cited by 1,424
A Simplified Monte Carlo Significance Test Procedure
Hope, A. C. A. (1968) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1968.tb00759.x · Cited by 1,073
QUADRATIC ASSIGNMENT AS A GENERAL DATA ANALYSIS STRATEGY
Hubert, L., & Schultz, J. (1976) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1976.tb00714.x · Cited by 829
Multivariate Permutation Tests : With Applications in Biostatistics
Pesarin, F. (2001) · Cited by 636
Permutation Tests for Linear Models
Anderson, M. J., & Robinson, J. (2001) · Australian & New Zealand Journal of Statistics · doi:10.1111/1467-842x.00156 · Cited by 555
Self-citationComparative Robustness of Six Tests in Multivariate Analysis of Variance
Olson, C. L. (1974) · Journal of the American Statistical Association · doi:10.1080/01621459.1974.10480224 · Cited by 507
A Nonstochastic Interpretation of Reported Significance Levels
Freedman, D., & Lane, D. (1983) · Journal of Business & Economic Statistics · doi:10.1080/07350015.1983.10509354 · Cited by 483
Anderson, M. J., & Legendre, P. (1999) · Journal of Statistical Computation and Simulation · doi:10.1080/00949659908811936 · Cited by 425
Self-citationResampling Methods for Computation-Intensive Data Analysis in Ecology and Evolution
Crowley, P. (1992) · Annual Review of Ecology and Systematics · doi:10.1146/annurev.ecolsys.23.1.405 · Cited by 417
Some Consequences When the Assumptions for the Analysis of Variance are not Satisfied
Cochran, W. G. (1947) · Biometrics · doi:10.2307/3001535 · Cited by 396
A Technique of Nonparametric Multivariate Analysis
Mantel, N., & Valand, R. S. (1970) · Biometrics · doi:10.2307/2529108 · Cited by 366
PITMAN, E. J. G. (1938) · Biometrika · doi:10.1093/biomet/29.3-4.322 · Cited by 328
Multi-response permutation procedures for a priori classifications
Mielke, P. W., Berry, K. J., & Johnson, E. S. (1976) · Communications in Statistics - Theory and Methods · doi:10.1080/03610927608827451 · Cited by 310
Nelder, J. A. (1965) · Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences · doi:10.1098/rspa.1965.0012 · Cited by 305
The Randomization Theory of Experimental Inference*
Kempthorne, O. (1955) · Journal of the American Statistical Association · doi:10.1080/01621459.1955.10501979 · Cited by 258
Edgington, E. (1998) · Technometrics · doi:10.1080/00401706.1998.10485213 · Cited by 243
MARDIA, K. V. (1971) · Biometrika · doi:10.1093/biomet/58.1.105 · Cited by 223
On randomization testing in vegetation science: multifactor comparisons of relevé groups
Pillar, V. D. P., & Orlóci, L. (1996) · Journal of Vegetation Science · doi:10.2307/3236308 · Cited by 217
Nelder, J. A. (1965) · Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences · doi:10.1098/rspa.1965.0013 · Cited by 190
The temporal variability of animal abundances: measures, methods and patterns
Gaston, K. J., & McArdle, B. H. (1994) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.1994.0114 · Cited by 186
Construction of Permutation Tests
Welch, W. J. (1990) · Journal of the American Statistical Association · doi:10.1080/01621459.1990.10474929 · Cited by 167
Multivariate Permutation Tests
Babu, G. J. (2002) · Technometrics · doi:10.1198/004017002320256468 · Cited by 133
Some Aspects of Experimental Inference
Kempthorne, O. (1966) · Journal of the American Statistical Association · doi:10.1080/01621459.1966.10502007 · Cited by 115
Randomization tests for multiple regression
Kennedy, P., & Cade, B. (1996) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610919608813350 · Cited by 111
Using fixed-effects model multivariate analysis of variance in marine biology and ecology
Johnson, C. R., & Field, C. (1993) · UTAS Research Repository · Cited by 110
Boik, R. J. (1987) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1987.tb00865.x · Cited by 99
The approximate randomization test as an alternative to the F test in analysis of variance
Still, A. W., & White, A. P. (1981) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1981.tb00634.x · Cited by 96
Permutation Tests for Least Absolute Deviation Regression
Cade, B. S., & Richards, J. D. (1996) · Biometrics · doi:10.2307/2533050 · Cited by 81
Gower, J. C., & Krzanowski, W. J. (1999) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/1467-9876.00168 · Cited by 74
Smith, E. P., Pontasch, K. W., & Cairns, J. (1990) · Water Research · doi:10.1016/0043-1354(90)90236-y · Cited by 68
Analysis of variance by randomization with small data sets
González, L., & Manly, B. F. J. (1998) · Environmetrics · doi:10.1002/(sici)1099-095x(199801/02)9:1<53::aid-env285>3.0.co;2-# · Cited by 53
Distribution‐Free Methods in Regression1
Brown, B. M., & Maritz, J. S. (1982) · Australian Journal of Statistics · doi:10.1111/j.1467-842x.1982.tb00837.x · Cited by 41
Cited by
Permutation inference for the general linear model
Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Nichols, T. E. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.01.060 · Cited by 3,563
Permutational Multivariate Analysis of Variance ( PERMANOVA )
Anderson, M. J. (2017) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat07841 · Cited by 1,849
Self-citationSensitivity of MRQAP Tests to Collinearity and Autocorrelation Conditions
Dekker, D., Krackhardt, D., & Snijders, T. A. B. (2007) · Psychometrika · doi:10.1007/s11336-007-9016-1 · Cited by 736
Sowell, E. R., Peterson, B. S., Kan, E., Woods, R. P., Yoshii, J., Bansal, R., Xu, D., Zhu, H., Thompson, P. M., & Toga, A. W. (2007) · Cerebral Cortex · doi:10.1093/cercor/bhl066 · Cited by 695
RRPP: An r package for fitting linear models to high‐dimensional data using residual randomization
Collyer, M. L., & Adams, D. C. (2018) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13029 · Cited by 682
Metadata
- Authors
- Anderson, M., & Braak, C. T.
- Venue
- Journal of Statistical Computation and Simulation
- Publisher
- Informa UK Limited
- Published
- 2003-01
- Volume
- 73
- Issue
- 2
- Pages
- 85-113
- ISSN
- 0094-9655 (ISSN-L)
- 1563-5163
- OpenAlex
- W2018135213
- Cited by
- 1,109
- References
- 50
- FWCI
- 12.71
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.