The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Permutation tests for multi-factorial analysis of variance
Open accessAbstract
Several permutation strategies are often possible for tests of individual terms in analysis-of-variance (ANOVA) designs. These include restricted permutations, permutation of whole groups of units, permutation of some form of residuals or some combination of these. It is unclear, especially for complex designs involving random factors, mixed models or nested hierarchies, just which permutation strategy should be used for any particular test. The purpose of this paper is two-fold: (i) we provide a guideline for constructing an exact permutation strategy, where possible, for any individual term in any ANOVA design; and (ii) we provide results of Monte Carlo simulations to compare the level accuracy and power of different permutation strategies in two-way ANOVA, including random and mixed models, nested hierarchies and tests of interaction terms. Simulation results showed that permutation of residuals under a reduced model generally had greater power than the exact test or alternative approximate permutation methods (such as permutation of raw data). In several cases, restricted permutations, in particular, suffered more than other procedures, in terms of loss of power, challenging the conventional wisdom of using this approach. Our simulations also demonstrated that the choice of correct exchangeable units under the null hypothesis, in accordance with the guideline we provide, is essential for any permutation test, whether it be an exact test or an approximate test. For reference, we also provide appropriate permutation strategies for individual terms in any two-way or three-way ANOVA for the exact test (where possible) and for the approximate test using permutation of residuals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 26,755
(2005) · Statistics for Biology and Health · doi:10.1007/0-387-27255-0_9 · Cited by 20,133
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
Non‐parametric multivariate analyses of changes in community structure
CLARKE, K. R. (1993) · Australian Journal of Ecology · doi:10.1111/j.1442-9993.1993.tb00438.x · Cited by 14,128
Excoffier, L., Smouse, P. E., & Quattro, J. M. (1992) · Genetics · doi:10.1093/genetics/131.2.479 · Cited by 13,929
A new method for non-parametric multivariate analysis of variance
Anderson, M. J. (2001) · Austral Ecology · doi:10.1111/j.1442-9993.2001.01070.pp.x · Cited by 13,828
Self-citationter Braak, C. J. F., & Šmilauer, P. (2002) · Cited by 10,579
Self-citationNelder, J. A., & Wedderburn, R. W. M. (1972) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344614 · Cited by 7,194
Snedecor, G. W., & Cochran, W. G. (1967) · Cited by 6,573
SCHEFFE, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,516
Cited by
Permutation inference for the general linear model
Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Nichols, T. E. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.01.060 · Cited by 3,563
Permutational Multivariate Analysis of Variance ( PERMANOVA )
Anderson, M. J. (2017) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat07841 · Cited by 1,849
Self-citationSensitivity of MRQAP Tests to Collinearity and Autocorrelation Conditions
Dekker, D., Krackhardt, D., & Snijders, T. A. B. (2007) · Psychometrika · doi:10.1007/s11336-007-9016-1 · Cited by 736
Sowell, E. R., Peterson, B. S., Kan, E., Woods, R. P., Yoshii, J., Bansal, R., Xu, D., Zhu, H., Thompson, P. M., & Toga, A. W. (2007) · Cerebral Cortex · doi:10.1093/cercor/bhl066 · Cited by 695
RRPP: An r package for fitting linear models to high‐dimensional data using residual randomization
Collyer, M. L., & Adams, D. C. (2018) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13029 · Cited by 682
Community ecology in the age of multivariate multiscale spatial analysis
Dray, S., Pélissier, R., Couteron, P., Fortin, M.-J., Legendre, P., Peres-Neto, P. R., Bellier, E., Bivand, R., Blanchet, F. G., De Cáceres, M., et al. (2012) · Ecological Monographs · doi:10.1890/11-1183.1 · Cited by 679
Multivariate Analysis of Ecological Data Using CANOCO
(2005) · Bulletin of the Ecological Society of America · doi:10.1890/0012-9623(2005)86[201:maoedu]2.0.co;2 · Cited by 536
A method for analysis of phenotypic change for phenotypes described by high-dimensional data
Collyer, M. L., Sekora, D. J., & Adams, D. C. (2015) · Heredity · doi:10.1038/hdy.2014.75 · Cited by 436
Levosimendan for the Prevention of Acute Organ Dysfunction in Sepsis
Gordon, A. C., Perkins, G. D., Singer, M., McAuley, D. F., Orme, R. M. L., Santhakumaran, S., Mason, A. J., Cross, M., Al-Beidh, F., Best-Lane, J., et al. (2016) · New England Journal of Medicine · doi:10.1056/nejmoa1609409 · Cited by 372
Relationships between IQ and Regional Cortical Gray Matter Thickness in Healthy Adults
Narr, K. L., Woods, R. P., Thompson, P. M., Szeszko, P., Robinson, D., Dimtcheva, T., Gurbani, M., Toga, A. W., & Bilder, R. M. (2007) · Cerebral Cortex · doi:10.1093/cercor/bhl125 · Cited by 344
Multivariate Phylogenetic Comparative Methods: Evaluations, Comparisons, and Recommendations
Adams, D. C., & Collyer, M. L. (2018) · Systematic Biology · doi:10.1093/sysbio/syx055 · Cited by 302
Pellegrino, E., Öpik, M., Bonari, E., & Ercoli, L. (2015) · Soil Biology and Biochemistry · doi:10.1016/j.soilbio.2015.02.020 · Cited by 299
Anderson, M. J., & Millar, R. B. (2004) · Journal of Experimental Marine Biology and Ecology · doi:10.1016/j.jembe.2003.12.011 · Cited by 298
Self-citationA GENERAL FRAMEWORK FOR THE ANALYSIS OF PHENOTYPIC TRAJECTORIES IN EVOLUTIONARY STUDIES
Adams, D. C., & Collyer, M. L. (2009) · Evolution · doi:10.1111/j.1558-5646.2009.00649.x · Cited by 297
Fraschetti, S., Terlizzi, A., & Benedetti-Cecchi, L. (2005) · Marine Ecology Progress Series · doi:10.3354/meps296013 · Cited by 289
Fornito, A., Yoon, J., Zalesky, A., Bullmore, E. T., & Carter, C. S. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2011.02.019 · Cited by 277
Claudet, J., Pelletier, D., Jouvenel, J.-Y., Bachet, F., & Galzin, R. (2006) · Biological Conservation · doi:10.1016/j.biocon.2005.12.030 · Cited by 258
Intraspecific variability in leaf traits strongly affects alder leaf decomposition in a stream
Lecerf, A., & Chauvet, E. (2008) · Basic and Applied Ecology · doi:10.1016/j.baae.2007.11.003 · Cited by 257
Comparing microbial water quality in an intermittent and continuous piped water supply
Kumpel, E., & Nelson, K. L. (2013) · Water Research · doi:10.1016/j.watres.2013.05.058 · Cited by 226
The neural dynamics of hierarchical Bayesian causal inference in multisensory perception
Rohe, T., Ehlis, A.-C., & Noppeney, U. (2019) · Nature Communications · doi:10.1038/s41467-019-09664-2 · Cited by 187
Sun, L., Grutzner, C., Bolte, S., Wibral, M., Tozman, T., Schlitt, S., Poustka, F., Singer, W., Freitag, C. M., & Uhlhaas, P. J. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.1073-12.2012 · Cited by 151
Deficits in high- (>60 Hz) gamma-band oscillations during visual processing in schizophrenia
Grützner, C., Wibral, M., Sun, L., Rivolta, D., Singer, W., Maurer, K., & Uhlhaas, P. J. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00088 · Cited by 140
The Faces of Predictive Coding
Brodski, A., Paasch, G.-F., Helbling, S., & Wibral, M. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.1529-14.2015 · Cited by 99
Sun, L., Castellanos, N., Grützner, C., Koethe, D., Rivolta, D., Wibral, M., Kranaster, L., Singer, W., Leweke, M. F., & Uhlhaas, P. J. (2013) · Schizophrenia Research · doi:10.1016/j.schres.2013.08.023 · Cited by 96
Rickman, D. S., Chen, Y.-B., Banerjee, S., Pan, Y., Yu, J., Vuong, T., Perner, S., Lafargue, C. J., Mertz, K. D., Setlur, S. R., et al. (2010) · Neoplasia · doi:10.1593/neo.10866 · Cited by 62
Breakdown of local information processing may underlie isoflurane anesthesia effects
Wollstadt, P., Sellers, K. K., Rudelt, L., Priesemann, V., Hutt, A., Fröhlich, F., & Wibral, M. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005511 · Cited by 49
“When permuting data, it is crucial to only permute data in such a way that the currently investigated effect is destroyed while all other effects are kept intact [65]: for example, consider a two-factorial design with factors A and B—if the main effect of factor A is to be tested, the assignment of levels of A to data…”
Chan, J. S., Wibral, M., Stawowsky, C., Brandl, M., Helbling, S., Naumer, M. J., Kaiser, J., & Wollstadt, P. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.631599 · Cited by 49
Linking the genetic architecture of cytosine modifications with human complex traits
Zhang, X., Moen, E. L., Liu, C., Mu, W., Gamazon, E. R., Delaney, S. M., Wing, C., Godley, L. A., Dolan, M. E., & Zhang, W. (2014) · Human Molecular Genetics · doi:10.1093/hmg/ddu313 · Cited by 41
“FDR was estimated by 100 permutations with an approximate test (59).”
MethodZhang, K., Wu, Z., Tang, D., Lv, C., Luo, K., Zhao, Y., Liu, X., Huang, Y., & Wang, J. (2016) · Frontiers in Plant Science · doi:10.3389/fpls.2016.00223 · Cited by 32
Lončarić, Ž., Bjedov, D., Kovačić, L. S., Ečimović, S., Hollert, H., & Velki, M. (2026) · Environmental Toxicology and Pharmacology · doi:10.1016/j.etap.2026.105097
Metadata
- Authors
- Anderson, M., & Braak, C. T.
- Venue
- Journal of Statistical Computation and Simulation
- Publisher
- Informa UK Limited
- Published
- 2003-01
- Volume
- 73
- Issue
- 2
- Pages
- 85-113
- ISSN
- 0094-9655 (ISSN-L)
- 1563-5163
- OpenAlex
- W2018135213
- Cited by
- 1,109
- References
- 50
- FWCI
- 12.71
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.