Permutation tests for factorially designed neuroimaging experiments
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Permutation methods for analysis of functional neuroimaging data acquired as factorially designed experiments are described and validated. The F ratio was estimated for main effects and interactions at each voxel in standard space. Critical values corresponding to probability thresholds were derived from a null distribution sampled by appropriate permutation of observations. Spatially informed, cluster‐level test statistics were generated by applying a preliminary probability threshold to the voxel F maps and then computing the sum of voxel statistics in each of the resulting three‐dimensional clusters, i.e., cluster “mass.” Using simulations comprising two between‐ or within‐subject factors each with two or three levels, contaminated by Gaussian and non‐normal noise, the voxel‐wise permutation test was compared to the standard parametric F test and to the performance of the spatially informed statistic using receiver operating characteristic (ROC) curves. Validity of the permutation‐testing algorithm and software is endorsed by almost identical performance of parametric and permutation tests of the voxel‐level F statistic. Permutation testing of suprathreshold voxel cluster mass, however, was found to provide consistently superior sensitivity to detect simulated signals than either of the voxel‐level tests. The methods are also illustrated by application to an experimental dataset designed to investigate effects of antidepressant drug treatment on brain activation by implicit sad facial affect perception in patients with major depression. Antidepressant drug effects in left amygdala and ventral striatum were detected by this software for an interaction between time (within‐subject factor) and group (between‐subject factor) in a representative two‐way factorial design. Hum. Brain Mapping 22:193–205, 2004. © 2004 Wiley‐Liss, Inc.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Nonparametric permutation tests for functional neuroimaging: A primer with examples
Nichols, T. E., & Holmes, A. P. (2002) · Human Brain Mapping · doi:10.1002/hbm.1058 · Cited by 6,256
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Forman, S. D., Cohen, J. D., Fitzgerald, M., Eddy, W. F., Mintun, M. A., & Noll, D. C. (1995) · Magnetic Resonance in Medicine · doi:10.1002/mrm.1910330508 · Cited by 3,202
Assessing the significance of focal activations using their spatial extent
Friston, K. J., Worsley, K. J., Frackowiak, R. S. J., Mazziotta, J. C., & Evans, A. C. (1994) · Human Brain Mapping · doi:10.1002/hbm.460010306 · Cited by 1,907 · cited 2×
“Exponential forms of the null distribution for cluster extent statistics have been theoretically derived previously for brain mapping [Cao, 1999; Cao and Worsley, 2001; Friston et al., 1994; Poline et al., 1997], but often may be over-conservative [Ashburner and Friston 2000; Bullmore et al.”
BackgroundSheline, Y. I., Barch, D. M., Donnelly, J. M., Ollinger, J. M., Snyder, A. Z., & Mintun, M. A. (2001) · Biological Psychiatry · doi:10.1016/s0006-3223(01)01263-x · Cited by 1,191 · cited 1×
“In fact, parametric methods are robust to violations of these assumptions and indeed perform slightly better than permutation methods with small samples unless the distribution is highly nonnormal [Gonzalez and Manly, 1998; Routledge, 1997 ].”
MethodBullmore, E. T., Suckling, J., Overmeyer, S., Rabe-Hesketh, S., Taylor, E., & Brammer, M. J. (1999) · IEEE Transactions on Medical Imaging · doi:10.1109/42.750253 · Cited by 1,088 · cited 5×
“Key words: ANOVA; repeated measures; nonparametric; randomisation; pharmacological MRI”
BackgroundMethodInfluentialSelf-citationCombining spatial extent and peak intensity to test for activations in functional imaging
Poline, J.-B., Worsley, K. J., Evans, A. C., & Friston, K. J. (1997) · NeuroImage · doi:10.1006/nimg.1996.0248 · Cited by 670
Validating cluster size inference: random field and permutation methods
Hayasaka, S. (2003) · NeuroImage · doi:10.1016/j.neuroimage.2003.08.003 · Cited by 491
Cited by
Permutation inference for the general linear model
Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Nichols, T. E. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.01.060 · Cited by 3,563
Network-based statistic: identifying differences in brain networks
Zalesky, A., Fornito, A., & Bullmore, E. T. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.06.041 · Cited by 2,750
“The approach can also be straightforwardly generalized to multifactorial designs (Suckling & Bullmore, 2004).”
BackgroundSelf-citationFu, C. H. Y., Williams, S. C. R., Cleare, A. J., Brammer, M. J., Walsh, N. D., Kim, J., Andrew, C. M., Pich, E. M., Williams, P. M., Reed, L. J., et al. (2004) · Archives of General Psychiatry · doi:10.1001/archpsyc.61.9.877 · Cited by 775
Self-citationChamberlain, S. R., Menzies, L., Hampshire, A., Suckling, J., Fineberg, N. A., del Campo, N., Aitken, M., Craig, K., Owen, A. M., Bullmore, E. T., et al. (2008) · Science · doi:10.1126/science.1154433 · Cited by 550
Self-citationOn the Definition of Signal-To-Noise Ratio and Contrast-To-Noise Ratio for fMRI Data
Welvaert, M., & Rosseel, Y. (2013) · PLoS ONE · doi:10.1371/journal.pone.0077089 · Cited by 493
Ersche, K. D., Barnes, A., Jones, P. S., Morein-Zamir, S., Robbins, T. W., & Bullmore, E. T. (2011) · Brain · doi:10.1093/brain/awr138 · Cited by 389
“For statistical inference, we used permutation methods and spatially extended statistics with nominal type I error control and greater sensitivity than voxel-based metrics (Suckling and Bullmore 2004).”
MethodSelf-citationMcDonald, C., Bullmore, E. T., Sham, P. C., Chitnis, X., Wickham, H., Bramon, E., & Murray, R. M. (2004) · Archives of General Psychiatry · doi:10.1001/archpsyc.61.10.974 · Cited by 379
“We tested the null hypotheses of no association between brain structure and genetic risk by permutation at cluster level, as described in detail elsewhere.(14,35,36) Stringent thresholds for statistical significance were derived from the permutation distribution so that the expected number of false-positive test resul…”
MethodSelf-citationChen, C.-H., Ridler, K., Suckling, J., Williams, S., Fu, C. H. Y., Merlo-Pich, E., & Bullmore, E. (2007) · Biological Psychiatry · doi:10.1016/j.biopsych.2006.09.018 · Cited by 354
Self-citationChamberlain, S. R., Hampshire, A., Müller, U., Rubia, K., del Campo, N., Craig, K., Regenthal, R., Suckling, J., Roiser, J. P., Grant, J. E., et al. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.10.014 · Cited by 301
Self-citationFornito, A., Yoon, J., Zalesky, A., Bullmore, E. T., & Carter, C. S. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2011.02.019 · Cited by 277
Self-citationFractional Gaussian noise, functional MRI and Alzheimer's disease
Maxim, V., Şendur, L., Fadili, J., Suckling, J., Gould, R., Howard, R., & Bullmore, E. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.10.044 · Cited by 272
Self-citationConnectivity differences in brain networks
Zalesky, A., Cocchi, L., Fornito, A., Murray, M. M., & Bullmore, E. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.068 · Cited by 260
Self-citationMure, H., Hirano, S., Tang, C. C., Isaias, I. U., Antonini, A., Ma, Y., Dhawan, V., & Eidelberg, D. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.09.028 · Cited by 259
Chen, C.-H., Lennox, B., Jacob, R., Calder, A., Lupson, V., Bisbrown-Chippendale, R., Suckling, J., & Bullmore, E. (2006) · Biological Psychiatry · doi:10.1016/j.biopsych.2005.06.008 · Cited by 235
Self-citationFunctional Coupling of the Amygdala in Depressed Patients Treated with Antidepressant Medication
Chen, C.-H., Suckling, J., Ooi, C., Fu, C. H. Y., Williams, S. C. R., Walsh, N. D., Mitterschiffthaler, M. T., Pich, E. M., & Bullmore, E. (2008) · Neuropsychopharmacology · doi:10.1038/sj.npp.1301593 · Cited by 224
“At these thresholds, we expect less than 1 false-positive cluster per map (Suckling and Bullmore, 2004; Bullmore et al, 1999).”
Self-citationDevelopmental changes in brain network hub connectivity in late adolescence
Baker, S. T. E., Lubman, D. I., Yucel, M., Allen, N. B., Whittle, S., Fulcher, B. D., Zalesky, A., & Fornito, A. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.5043-14.2015 · Cited by 168
“This constraint accounts for the fact that repeated measurements on the same individual will be correlated and cannot be considered exchangeable under the null hypothesis (Suckling and Bullmore, 2004).”
BackgroundA Shift to Randomness of Brain Oscillations in People with Autism
Lai, M.-C., Lombardo, M. V., Chakrabarti, B., Sadek, S. A., Pasco, G., Wheelwright, S. J., Bullmore, E. T., Baron-Cohen, S., & Suckling, J. (2010) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.06.027 · Cited by 153
Self-citationMenzies, L., Ooi, C., Kamath, S., Suckling, J., McKenna, P., Fletcher, P., Bullmore, E., & Stephenson, C. (2007) · Archives of General Psychiatry · doi:10.1001/archpsyc.64.2.156 · Cited by 120
Self-citationThe Faces of Predictive Coding
Brodski, A., Paasch, G.-F., Helbling, S., & Wibral, M. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.1529-14.2015 · Cited by 99
“HRs and RTs were subjected to separate 2 2 repeated-measurements permutation ANOVAs (Anderson and Ter Braak, 2003; Suckling and Bullmore, 2004).”
MethodNeuroanatomical Correlates of Memory Deficits in Tuberous Sclerosis Complex
Ridler, K., Suckling, J., Higgins, N., de Vries, P., Stephenson, C., Bolton, P., & Bullmore, E. (2007) · Cerebral Cortex · doi:10.1093/cercor/bhj144 · Cited by 81
Self-citationBreakdown of local information processing may underlie isoflurane anesthesia effects
Wollstadt, P., Sellers, K. K., Rudelt, L., Priesemann, V., Hutt, A., Fröhlich, F., & Wibral, M. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005511 · Cited by 49
“To test for statistically relevant effects of isoflurane levels and direction of interaction or recording site on estimated measures, we performed two-factorial permutation analyses of variance (pANOVA) for each animal and estimated measure [65, 66, 67, 39].”
Chan, J. S., Wibral, M., Stawowsky, C., Brandl, M., Helbling, S., Naumer, M. J., Kaiser, J., & Wollstadt, P. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.631599 · Cited by 49
A statistical method (cross-validation) for bone loss region detection after spaceflight
Zhao, Q., Li, W., Li, C., Chu, P. W., Kornak, J., Lang, T. F., Fang, J., & Lu, Y. (2010) · Australasian Physical & Engineering Sciences in Medicine · doi:10.1007/s13246-010-0024-6 · Cited by 5
“Researchers face a constant challenge in trying to identify valid thresholds for constructing candidate clusters [10, 22].”
Background
Metadata
- Authors
- Suckling, J., & Bullmore, E.
- Venue
- Human Brain Mapping
- Publisher
- Wiley
- Published
- 2004-04-26
- Volume
- 22
- Issue
- 3
- Pages
- 193-205
- ISSN
- 1065-9471 (ISSN-L)
- 1097-0193
- OpenAlex
- W2049582292
- Cited by
- 94
- References
- 8
- FWCI
- 3.20
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.