The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
Nonparametric permutation tests for functional neuroimaging: A primer with examples
Open accessAbstract
Requiring only minimal assumptions for validity, nonparametric permutation testing provides a flexible and intuitive methodology for the statistical analysis of data from functional neuroimaging experiments, at some computational expense. Introduced into the functional neuroimaging literature by Holmes et al. ([ 1996 ]: J Cereb Blood Flow Metab 16:7–22), the permutation approach readily accounts for the multiple comparisons problem implicit in the standard voxel‐by‐voxel hypothesis testing framework. When the appropriate assumptions hold, the nonparametric permutation approach gives results similar to those obtained from a comparable Statistical Parametric Mapping approach using a general linear model with multiple comparisons corrections derived from random field theory. For analyses with low degrees of freedom, such as single subject PET/SPECT experiments or multi‐subject PET/SPECT or f MRI designs assessed for population effects, the nonparametric approach employing a locally pooled (smoothed) variance estimate can outperform the comparable Statistical Parametric Mapping approach. Thus, these nonparametric techniques can be used to verify the validity of less computationally expensive parametric approaches. Although the theory and relative advantages of permutation approaches have been discussed by various authors, there has been no accessible explication of the method, and no freely distributed software implementing it. Consequently, there have been few practical applications of the technique. This article, and the accompanying MATLAB software, attempts to address these issues. The standard nonparametric randomization and permutation testing ideas are developed at an accessible level, using practical examples from functional neuroimaging, and the extensions for multiple comparisons described. Three worked examples from PET and f MRI are presented, with discussion, and comparisons with standard parametric approaches made where appropriate. Practical considerations are given throughout, and relevant statistical concepts are expounded in appendices. Hum. Brain Mapping 15:1–25, 2001. © 2001 Wiley‐Liss, Inc.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Statistical significance of decoding performance was assessed via permutation test, randomly permuting the sample labels and repeating the decoding analysis 1,000 times, corrected for multiple comparisons using a cluster-based correction (cluster mass inference with cluster forming threshold at P < 0.05)105,106.”
From Adversarial testing of global neuronal workspace and integrated information theories of consciousness (2025) · cites this paper as [105]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The permutation p-value in Equation ( 10) is super-uniform under H 0 (i.e., P H0 (p ≤ α) ≤ α for all α ∈ [0, 1]), a standard consequence of randomization inference (Nichols & Holmes, 2002;Phipson & Smyth, 2010;Good, 2005)”
From Revisiting the Platonic Representation Hypothesis: An Aristotelian View (2026) · cites this paper as Nichols & Holmes, 2002
The cited source says what the claim attributes to it.
In the citing paper
“While resampling-based “maxT” procedures (Westfall & Young, 1993;Nichols & Holmes, 2002) can calibrate such aggregates, this has not yet been applied in representational similarity studies.”
From Revisiting the Platonic Representation Hypothesis: An Aristotelian View (2026) · cites this paper as Nichols & Holmes, 2002
The cited source says what the claim attributes to it.
In the citing paper
“To find the true type I error rate, we performed Monte Carlo simulations that permute the sign of the whole-brain maps from individual subjects58.”
From Anchoring the neural compass: coding of local spatial reference frames in human medial parietal lobe (2014) · cites this paper as [58]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688
Self-citationKendall, M. G. (1957) · Biometrika · doi:10.2307/2333282 · Cited by 6,442
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Forman, S. D., Cohen, J. D., Fitzgerald, M., Eddy, W. F., Mintun, M. A., & Noll, D. C. (1995) · Magnetic Resonance in Medicine · doi:10.1002/mrm.1910330508 · Cited by 3,202
Analysis of fMRI Time-Series Revisited
Friston, K. J., Holmes, A. P., Poline, J., Grasby, P. M., Williams, S., Frackowiak, R. S. J., & Turner, R. (1995) · NeuroImage · doi:10.1006/nimg.1995.1007 · Cited by 2,504
Self-citationBevan, J. M., & Kendall, M. G. (1971) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2986801 · Cited by 2,458
Multiple Comparison Procedures
Hochberg, Y., & Tamhane, A. C. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316672 · Cited by 2,389
Analysis of fMRI Time-Series Revisited—Again
Worsley, K. J., & Friston, K. J. (1995) · NeuroImage · doi:10.1006/nimg.1995.1023 · Cited by 2,198
Multiple Comparison Procedures
Hahs-Vaughn, D., & Lomax, R. (2013) · Routledge eBooks · doi:10.4324/9780203137819-17 · Cited by 2,132
Edgington, E. S., Edgington, E., & Onghena, P. (1964) · Springer eBooks · doi:10.1002/0471667196.ess2169 · Cited by 2,052
A three-dimensional statistical analysis for CBF activation studies in human brain
Worsley, K. J., Evans, A. C., Marrett, S., & Neelin, P. (1992) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.1992.127 · Cited by 2,036
Randomization, Bootstrap and Monte Carlo Methods in Biology.
Kemp, A. W., & Manly, B. F. J. (1997) · Biometrics · doi:10.2307/2533527 · Cited by 1,928
Assessing the significance of focal activations using their spatial extent
Friston, K. J., Worsley, K. J., Frackowiak, R. S. J., Mazziotta, J. C., & Evans, A. C. (1994) · Human Brain Mapping · doi:10.1002/hbm.460010306 · Cited by 1,907
Frackowiak, R. S. J. (1997) · Cited by 1,713
Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment.
Martin, M. A., Westfall, P. H., & Young, S. S. (1994) · Biometrics · doi:10.2307/2533464 · Cited by 1,713
Comparing Functional (PET) Images: The Assessment of Significant Change
Friston, K. J., Frith, C., Liddle, P. F., & Frackowiak, R. S. J. (1991) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.1991.122 · Cited by 1,626
Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses
Niknian, M., & Good, P. (1995) · Technometrics · doi:10.2307/1269919 · Cited by 1,463
Watson, J. D. G., Myers, R., Frackowiak, R. S. J., Hajnal, J. V., Woods, R. P., Mazziotta, J. C., Shipp, S. D., & Zeki, S. (1993) · Cerebral Cortex · doi:10.1093/cercor/3.2.79 · Cited by 1,235
Detecting activations in PET and fMRI: levels of inference and power
Friston, K. J., Holmes, A., Poline, J.-B., Price, C. J., & Frith, C. D. (1996) · NeuroImage · doi:10.1006/nimg.1996.0074 · Cited by 1,148
Self-citationBullmore, E. T., Suckling, J., Overmeyer, S., Rabe-Hesketh, S., Taylor, E., & Brammer, M. J. (1999) · IEEE Transactions on Medical Imaging · doi:10.1109/42.750253 · Cited by 1,088
A functional neuroanatomy of hallucinations in schizophrenia
Silbersweig, D. A., Stern, E., Frith, C., Cahill, C., Holmes, A., Grootoonk, S., Seaward, J., McKenna, P., Chua, S. E., Schnorr, L., et al. (1995) · Nature · doi:10.1038/378176a0 · Cited by 906
Nonparametric Analysis of Statistic Images from Functional Mapping Experiments
Holmes, A. P., Blair, R. C., Watson, J. D. G., & Ford, I. (1996) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1097/00004647-199601000-00002 · Cited by 894
Self-citationGeneralisability, Random Effects & Population Inference
Holmes, A. P., & Friston, K. J. (1998) · NeuroImage · doi:10.1016/s1053-8119(18)31587-8 · Cited by 886
Self-citationModified Randomization Tests for Nonparametric Hypotheses
Dwass, M. (1957) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177707045 · Cited by 744
Significance Tests Which May be Applied to Samples from Any Populations
Pitman, E. J. G. (1937) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.2307/2984124 · Cited by 743
Statistical methods of estimation and inference for functional MR image analysis
Bullmore, E. T., Brammer, M. J., Williams, S., Rabe‐Hesketh, S., Janot, N., David, A. S., Mellers, J. D. C., Howard, R. J., & Sham, P. C. (1996) · Magnetic Resonance in Medicine · doi:10.1002/mrm.1910350219 · Cited by 683
Combining spatial extent and peak intensity to test for activations in functional imaging
Poline, J.-B., Worsley, K. J., Evans, A. C., & Friston, K. J. (1997) · NeuroImage · doi:10.1006/nimg.1996.0248 · Cited by 670
Andreasen, N. C., O'Leary, D. S., Cizadlo, T., Arndt, S., Rezai, K., Ponto, L. L., Watkins, G. L., & Hichwa, R. D. (1996) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.93.18.9985 · Cited by 625
Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment.
Thomas, G. E., Westfall, P. H., & Young, S. S. (1994) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348369 · Cited by 522
Local Maxima and the Expected Euler Characteristic of Excursion Sets of χ 2, F and t Fields
Worsley, K. J. (1994) · Advances in Applied Probability · doi:10.2307/1427576 · Cited by 355
Rank Correlation Methods (5th ed.).
Marden, J. I., Kendall, M. G., & Gibbons, J. D. (1992) · Journal of the American Statistical Association · doi:10.2307/2290477 · Cited by 344
Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses.
Modarres, R., & Good, P. (1995) · Journal of the American Statistical Association · doi:10.2307/2291167 · Cited by 340
Pitman, E. J. G. (1938) · Biometrika · doi:10.1093/biomet/29.3-4.322 · Cited by 314
Investigation of Low Frequency Drift in fMRI Signal
Smith, A. M., Lewis, B. K., Ruttimann, U. E., Ye, F. Q., Sinnwell, T., Yang, Y., Duyn, J. H., & Frank, J. A. (1999) · NeuroImage · doi:10.1006/nimg.1999.0435 · Cited by 313
Local Maxima and the Expected Euler Characteristic of Excursion Sets of χ 2, F and t Fields
Worsley, K. J. (1994) · Advances in Applied Probability · doi:10.1017/s0001867800025970 · Cited by 310
Statistical Methods, Experimental Design, and Scientific Inference.
Fisher, R. A. (1991) · Biometrics · doi:10.2307/2532685 · Cited by 254
Edgington, E. S. (1998) · Technometrics · doi:10.1080/00401706.1998.10485213 · Cited by 245
Order Information in Working Memory: fMRI Evidence for Parietal and Prefrontal Mechanisms
Marshuetz, C., Smith, E. E., Jonides, J., Degutis, J. M., & Chenevert, T. L. (2000) · Journal of Cognitive Neuroscience · doi:10.1162/08989290051137459 · Cited by 243
Approximate Randomization Tests
Edgington, E. S. (1969) · The Journal of Psychology · doi:10.1080/00223980.1969.10543491 · Cited by 191
Finite Sample Properties and Asymptotic Efficiency of Monte Carlo Tests
Jöckel, K. (1986) · The Annals of Statistics · doi:10.1214/aos/1176349860 · Cited by 161
Poline, J., & Mazoyer, B. M. (1993) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.1993.57 · Cited by 155
Three‐dimensional analysis of clustered voxels in 15O‐butanol brain activation images
Roland, P. E., Levin, B., Kawashima, R., & Åkerman, S. (1993) · Human Brain Mapping · doi:10.1002/hbm.460010103 · Cited by 121
The size of the connected components of excursion sets of χ2, t and F fields
Cao, J. L. (1999) · Advances in Applied Probability · doi:10.1239/aap/1029955192 · Cited by 114
Statistical inference : the distribution-free approach
Edgington, E. S. (1969) · McGraw-Hill eBooks · Cited by 108
Worsley, K. J. (1996) · CHANCE · doi:10.1080/09332480.1996.10542483 · Cited by 102
Tests for Comparing Images Based on Randomization and Permutation Methods
Arndt, S. V., Cizadlo, T., Andreasen, N. C., Heckel, D., Gold, S. M., & OʼLeary, D. S. (1996) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1097/00004647-199611000-00023 · Cited by 92
Silbersweig, D., Stern, E. R., Schnorr, L., Frith, C., Ashburner, J., Cahill, C., Frackowiak, R. S. J., & Jones, T. (1994) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.1994.98 · Cited by 47
Intensity-adaptive segmentation of single-echo T1-weighted magnetic resonance images
Momenan, R., Hommer, D. W., Rawlings, R. R., Ruttimann, U. E., Kerich, M. J., & Rio, D. E. (1997) · Human Brain Mapping · doi:10.1002/(sici)1097-0193(1997)5:3 · Cited by 39
A Linear Spatial Correlation Model, With Applications to Positron Emission Tomography
Worsley, K. J., Evans, A. C., Strother, S. C., & Tyler, J. L. (1991) · Journal of the American Statistical Association · doi:10.1080/01621459.1991.10475004 · Cited by 38
Performance of a Randomization Test for Single-Subject 15 O-Water PET Activation Studies
Halber, M., Herholz, K., Wienhard, K., Pawlik, G., & Heiss, W. (1997) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1097/00004647-199710000-00004 · Cited by 15
Holmes and Watson on ‘Sherlock’
Holmes, A. P., Watson, J. D. G., & Nichols, T. E. (1998) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1097/00004647-199806000-00012 · Cited by 7
Self-citationSmoothness and degrees of freedom restrictions when using SPM99
Stoeckel, J., Poline, J., Malandain, G., Ayache, N., & Darcourt, J. (2001) · NeuroImage · doi:10.1016/s1053-8119(01)91602-7 · Cited by 6
Smoothness and degrees of freedom restrictions when using SPM 99
Stoeckel, J., Poline, J., Malandain, G., Ayache, N., & Darcourt, J. (2001) · HAL · Cited by 5
Comparison of activation response using functional PET and MRI
Noll, D. C., Kinahan, P. E., Mintun, M. A., Thulborn, K. R., & Townsend, D. W. (1996) · NeuroImage · doi:10.1016/s1053-8119(96)80036-x · Cited by 1
Worsley (1996) · doi:10.1002/(sici)1097-0193(1996)4:1
Research works printed but not linked: 5
- 10.1080/00223980.1964.9916711 · DOI
- Fisher RA, 1990, Bennett JH
- HolmesAP(1994): Statistical issues in functional brain mapping PhD thesis. University of Glasgow.http://www.fil.ion.ucl.ac.uk/spm/papers/APH_thesis
- Pitman EJG, 1937, Significance tests which may be applied to samples from any population. II. The correlation coefficient test, J R Stat Soc, 4, 224
- Worsley KJ, 1995, A unified statistical approach for determining significant signals in images of cerebral activation, Hum Brain Mapp, 4, 58, 10.1002/(SICI)1097-0193(1996)4:1<58::AID-HBM4>3.0.CO;2-O · DOI
Cited by
Sorted by citations
Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W., & Smith, S. M. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.09.015 · Cited by 11,314
Nonparametric statistical testing of EEG- and MEG-data
Maris, E., & Oostenveld, R. (2007) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2007.03.024 · Cited by 8,496
Tract-based spatial statistics: Voxelwise analysis of multi-subject diffusion data
Smith, S. M., Jenkinson, M., Johansen‐Berg, H., Rueckert, D., Nichols, T. E., Mackay, C. E., Watkins, K. E., Ciccarelli, O., Cader, M. Z., Matthews, P. M., et al. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.02.024 · Cited by 6,598
Self-citationMEG and EEG data analysis with MNE-Python
Gramfort, A. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00267 · Cited by 3,882
Cluster failure: why fMRI inferences for spatial extent have inflated false-positive rates
Eklund, A., Nichols, T. E., & Knutsson, H. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1602413113 · Cited by 3,487
Self-citation
Metadata
- Authors
- Nichols, T. E., & Holmes, A. P.
- Affiliations
- University of Michigan (US)
- UCL Queen Square Institute of Neurology (GB)
- ORCID
- Nichols, Thomas E.
- Venue
- Human Brain Mapping
- Publisher
- Wiley
- Published
- 2001-10-15
- Volume
- 15
- Issue
- 1
- Pages
- 1-25
- ISSN
- 1065-9471 (ISSN-L)
- 1097-0193
- PMC
- PMC6871862
- OpenAlex
- W2138790588
- Cited by
- 6,256
- References
- 56
- Citations by year
- 2026: 223 · 2025: 264 · 2024: 255 · 2023: 275 · 2022: 321 · 2021: 350 · 2020: 392 · 2019: 368 · 2018: 381 · 2017: 368 · 2016: 343 · 2015: 385 · 2014: 365 · 2013: 356 · 2012: 314
- FWCI
- 16.77
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Human Brain Mapping (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Deep Blue (University of Michigan) (submittedVersion)
- Funders
- Wellcome Trust
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Medical Imaging Techniques and Applications
- Keywords
- permutation tests, functional neuroimaging, multiple testing correction, statistical parametric mapping, random field theory, PET, functional MRI, population effects, MATLAB software
- MeSH terms
- Brain, Brain Mapping, Data Interpretation, Statistical, Humans, Image Processing, Computer-Assisted, Magnetic Resonance Imaging, Signal Processing, Computer-Assisted, Tomography, Emission-Computed, Statistical Distributions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
