The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Faster permutation inference in brain imaging
Open accessAbstract
Permutation tests are increasingly being used as a reliable method for inference in neuroimaging analysis. However, they are computationally intensive. For small, non-imaging datasets, recomputing a model thousands of times is seldom a problem, but for large, complex models this can be prohibitively slow, even with the availability of inexpensive computing power. Here we exploit properties of statistics used with the general linear model (GLM) and their distributions to obtain accelerations irrespective of generic software or hardware improvements. We compare the following approaches: (i) performing a small number of permutations; (ii) estimating the p-value as a parameter of a negative binomial distribution; (iii) fitting a generalised Pareto distribution to the tail of the permutation distribution; (iv) computing p-values based on the expected moments of the permutation distribution, approximated from a gamma distribution; (v) direct fitting of a gamma distribution to the empirical permutation distribution; and (vi) permuting a reduced number of voxels, with completion of the remainder using low rank matrix theory. Using synthetic data we assessed the different methods in terms of their error rates, power, agreement with a reference result, and the risk of taking a different decision regarding the rejection of the null hypotheses (known as the resampling risk). We also conducted a re-analysis of a voxel-based morphometry study as a real-data example. All methods yielded exact error rates. Likewise, power was similar across methods. Resampling risk was higher for methods (i), (iii) and (v). For comparable resampling risks, the method in which no permutations are done (iv) was the absolute fastest. All methods produced visually similar maps for the real data, with stronger effects being detected in the family-wise error rate corrected maps by (iii) and (v), and generally similar to the results seen in the reference set. Overall, for uncorrected p-values, method (iv) was found the best as long as symmetric errors can be assumed. In all other settings, including for familywise error corrected p-values, we recommend the tail approximation (iii). The methods considered are freely available in the tool PALM - Permutation Analysis of Linear Models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
Advances in functional and structural MR image analysis and implementation as FSL
Smith, S. M., Jenkinson, M., Woolrich, M. W., Beckmann, C. F., Behrens, T. E. J., Johansen-Berg, H., Bannister, P. R., De Luca, M., Drobnjak, I., Flitney, D. E., et al. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.07.051 · Cited by 13,630 · cited 2×
“, 2001) carried out with the fmrib Software Library (fsl; Smith et al., 2004).”
MethodSelf-citationCramming More Components onto Integrated Circuits (1965)
Moore, G. (2021) · Ideas That Created the Future · doi:10.7551/mitpress/12274.003.0027 · Cited by 6,241
SCHEFFE, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,516
SMITH, S., & NICHOLS, T. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.03.061 · Cited by 5,250
Self-citationTHE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
Genovese, C. R., Lazar, N. A., & Nichols, T. (2002) · NeuroImage · doi:10.1006/nimg.2001.1037 · Cited by 4,923
Self-citationExact Matrix Completion via Convex Optimization
Candès, E. J., & Recht, B. (2009) · Foundations of Computational Mathematics · doi:10.1007/s10208-009-9045-5 · Cited by 4,877
A Voxel-Based Morphometric Study of Ageing in 465 Normal Adult Human Brains
Good, C. D., Johnsrude, I. S., Ashburner, J., Henson, R. N. A., Friston, K. J., & Frackowiak, R. S. J. (2001) · NeuroImage · doi:10.1006/nimg.2001.0786 · Cited by 4,561
Probable Inference, the Law of Succession, and Statistical Inference
Wilson, E. B. (1927) · Journal of the American Statistical Association · doi:10.1080/01621459.1927.10502953 · Cited by 4,476
Missing Value Estimation Methods for DNA Microarrays
Troyanskaya, O., Cantor, M., Sherlock, G., Brown, P., Hastie, T., Tibshirani, R., Botstein, D., & Altman, R. B. (2001) · Bioinformatics · doi:10.1093/bioinformatics/17.6.520 · Cited by 4,230
Statistical Inference Using Extreme Order Statistics
III, J. P. (1975) · The Annals of Statistics · doi:10.1214/aos/1176343003 · Cited by 3,567
Permutation inference for the general linear model
Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Nichols, T. E. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.01.060 · Cited by 3,563
Self-citationAsymptotic Theory of Certain "Goodness of Fit" Criteria Based on Stochastic Processes
Anderson, T. W., & Darling, D. A. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729437 · Cited by 3,542 · cited 1×
“Goodness of fit can be assessed with the Anderson– Darling test (Anderson and Darling, 1952; Choulakian and Stephens, 2001; Knijnenburg et al., 2009).”
MethodInterval Estimation for a Binomial Proportion
Brown, L. D., Cai, T. T., & DasGupta, A. (2001) · Statistical Science · doi:10.1214/ss/1009213286 · Cited by 3,538
Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480550 · Cited by 3,223
Extremes and Related Properties of Random Sequences and Processes
Leadbetter, M. R., Lindgren, G., & Rootzén, H. (1983) · Springer Series in Statistics · doi:10.1007/978-1-4612-5449-2 · Cited by 2,873
The chips are down for Moore’s law
Waldrop, M. M. (2016) · Nature · doi:10.1038/530144a · Cited by 2,397
Sur La Distribution Limite Du Terme Maximum D'Une Serie Aleatoire
Gnedenko, B. (1943) · The Annals of Mathematics · doi:10.2307/1968974 · Cited by 2,258
The Power of Convex Relaxation: Near-Optimal Matrix Completion
Candes, E. J., & Tao, T. (2010) · IEEE Transactions on Information Theory · doi:10.1109/tit.2010.2044061 · Cited by 2,060
Assessing the significance of focal activations using their spatial extent
Friston, K. J., Worsley, K. J., Frackowiak, R. S. J., Mazziotta, J. C., & Evans, A. C. (1994) · Human Brain Mapping · doi:10.1002/hbm.460010306 · Cited by 1,907
Extremes and Related Properties of Random Sequences and Processes.
Davis, R. A., Leadbetter, M. R., Lindgren, G., & Rootzen, H. (1985) · Journal of the American Statistical Association · doi:10.2307/2288102 · Cited by 1,725
III. Contributions to the mathematical theory of evolution
Pearson, K. (1894) · Philosophical Transactions of the Royal Society of London. (A.) · doi:10.1098/rsta.1894.0003 · Cited by 1,550
Parameter and Quantile Estimation for the Generalized Pareto Distribution
Hosking, J. R. M., & Wallis, J. R. (1987) · Technometrics · doi:10.1080/00401706.1987.10488243 · Cited by 1,359
X. Contributions to the mathematical theory of evolution.—II. Skew variation in homogeneous material
Pearson, K. (1895) · Philosophical Transactions of the Royal Society of London. (A.) · doi:10.1098/rsta.1895.0010 · Cited by 1,226
Bullmore, E. T., Suckling, J., Overmeyer, S., Rabe-Hesketh, S., Taylor, E., & Brammer, M. J. (1999) · IEEE Transactions on Medical Imaging · doi:10.1109/42.750253 · Cited by 1,088
Anatomically related grey and white matter abnormalities in adolescent-onset schizophrenia
Douaud, G., Smith, S., Jenkinson, M., Behrens, T., Johansen-Berg, H., Vickers, J., James, S., Voets, N., Watkins, K., Matthews, P. M., et al. (2007) · Brain · doi:10.1093/brain/awm184 · Cited by 771
Self-citationSensitivity of MRQAP Tests to Collinearity and Autocorrelation Conditions
Dekker, D., Krackhardt, D., & Snijders, T. A. B. (2007) · Psychometrika · doi:10.1007/s11336-007-9016-1 · Cited by 736
Combining spatial extent and peak intensity to test for activations in functional imaging
Poline, J.-B., Worsley, K. J., Evans, A. C., & Friston, K. J. (1997) · NeuroImage · doi:10.1006/nimg.1996.0248 · Cited by 670
Nonstationary cluster-size inference with random field and permutation methods
Hayasaka, S., Phan, K. L., Liberzon, I., Worsley, K. J., & Nichols, T. E. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.01.041 · Cited by 661
Self-citationPermutation Tests for Linear Models
Anderson, M. J., & Robinson, J. (2001) · Australian & New Zealand Journal of Statistics · doi:10.1111/1467-842x.00156 · Cited by 555
Bootstrap tests: how many bootstraps?
Davidson, R., & MacKinnon, J. G. (2000) · Econometric Reviews · doi:10.1080/07474930008800459 · Cited by 460
Some New Test Criteria in Multivariate Analysis
Pillai, K. C. S. (1955) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728599 · Cited by 431
Anderson, M. J., & Legendre, P. (1999) · Journal of Statistical Computation and Simulation · doi:10.1080/00949659908811936 · Cited by 425
Medical image processing on the GPU – Past, present and future
Eklund, A., Dufort, P., Forsberg, D., & LaConte, S. M. (2013) · Medical Image Analysis · doi:10.1016/j.media.2013.05.008 · Cited by 397
A Three-step Method for Choosing the Number of Bootstrap Repetitions
Andrews, D. W. K., & Buchinsky, M. (2000) · Econometrica · doi:10.1111/1468-0262.00092 · Cited by 306
ON A METHOD OF ESTIMATING FREQUENCIES
HALDANE, J. B. S. (1945) · Biometrika · doi:10.1093/biomet/33.3.222 · Cited by 285
Distributional assumptions in voxel-based morphometry
Salmond, C. H., Ashburner, J., Vargha-Khadem, F., Connelly, A., Gadian, D. G., & Friston, K. J. (2002) · NeuroImage · doi:10.1006/nimg.2002.1153 · Cited by 256
Non‐parametric combination and related permutation tests for neuroimaging
Winkler, A. M., Webster, M. A., Brooks, J. C., Tracey, I., Smith, S. M., & Nichols, T. E. (2016) · Human Brain Mapping · doi:10.1002/hbm.23115 · Cited by 245
Self-citationWinkler, A. M., Webster, M. A., Vidaurre, D., Nichols, T. E., & Smith, S. M. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.05.092 · Cited by 243
Self-citationMARDIA, K. V. (1971) · Biometrika · doi:10.1093/biomet/58.1.105 · Cited by 223
Fewer permutations, more accurate P-values
Knijnenburg, T. A., Wessels, L. F. A., Reinders, M. J. T., & Shmulevich, I. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp211 · Cited by 221
Sequential Monte Carlo p-values
BESAG, J., & CLIFFORD, P. (1991) · Biometrika · doi:10.1093/biomet/78.2.301 · Cited by 194
Hernández, M., Guerrero, G. D., Cecilia, J. M., García, J. M., Inuggi, A., Jbabdi, S., Behrens, T. E. J., & Sotiropoulos, S. N. (2013) · PLoS ONE · doi:10.1371/journal.pone.0061892 · Cited by 193
Davison, A. C., & Huser, R. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020133 · Cited by 187
Adjusting the effect of nonstationarity in cluster-based and TFCE inference
Salimi‐Khorshidi, G., Smith, S. M., & Nichols, T. E. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.09.088 · Cited by 166
Self-citationResolving the sign ambiguity in the singular value decomposition
Bro, R., Acar, E., & Kolda, T. G. (2008) · Journal of Chemometrics · doi:10.1002/cem.1122 · Cited by 160
Robustness to non-normality of regression tests
BOX, G. E. P., & WATSON, G. S. (1962) · Biometrika · doi:10.1093/biomet/49.1-2.93 · Cited by 149
́ndez, M. H., Guerrero, G. D., Cecilia, J. M., García, J. M., Inuggi, A., & Sotiropoulos, S. N. (2012) · Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP) · doi:10.1109/pdp.2012.46 · Cited by 141
Refined approximations to permutation tests for multivariate inference
Kazi-Aoual, F., Hitier, S., Sabatier, R., & Lebreton, J. (1995) · Computational Statistics & Data Analysis · doi:10.1016/0167-9473(94)00064-2 · Cited by 106
Ge, T., Feng, J., Hibar, D. P., Thompson, P. M., & Nichols, T. E. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.07.012 · Cited by 94
Self-citationGilbert, P. B. (2004) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/j.1467-9876.2005.00475.x · Cited by 75
Meaningful design and contrast estimability in FMRI
Smith, S. M., Jenkinson, M., Beckmann, C. F., Miller, K. L., & Woolrich, M. W. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.09.019 · Cited by 68
Self-citationfMRI analysis on the GPU—Possibilities and challenges
Eklund, A., Andersson, M., & Knutsson, H. E. (2011) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2011.07.007 · Cited by 62
Sequential Implementation of Monte Carlo Tests With Uniformly Bounded Resampling Risk
Gandy, A. (2009) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.tm08368 · Cited by 52
Improving the reporting of P ‐values generated by randomization methods
Ruxton, G. D., & Neuhäuser, M. (2013) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12102 · Cited by 51
Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling RiskPreprint
Gandy, A. (2008) · Cited by 49
Sequential Monte Carlo multiple testing
Sandve, G. K., Ferkingstad, E., & Nygård, S. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr568 · Cited by 44
Designing Monte Carlo Implementations of Permutation or Bootstrap Hypothesis Tests
Fay, M. P., & Follmann, D. A. (2002) · The American Statistician · doi:10.1198/000313002753631385 · Cited by 33
Non-normality and transformations of random fields, with an application to voxel-based morphometry
Viviani, R., Beschoner, P., Ehrhard, K., Schmitz, B., & Thöne, J. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2006.11.037 · Cited by 31
Fay, M. P., Kim, H., & Hachey, M. (2007) · Journal of Computational and Graphical Statistics · doi:10.1198/106186007x257025 · Cited by 30
Deriving Statistical Significance Maps for SVM Based Image Classification and Group Comparisons
Gaonkar, B., & Davatzikos, C. A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33415-3_89 · Cited by 24
Distance-based analysis of variance: Approximate inference
Minas, C., & Montana, G. (2014) · Statistical Analysis and Data Mining The ASA Data Science Journal · doi:10.1002/sam.11227 · Cited by 13
Speeding up Permutation Testing in Neuroimaging
Chris Hinrichs, A., Ithapu, V. K., Sun, Q., Johnson, S. C., & Singh, V. (2015) · PubMed · doi:10.48550/arxiv.1502.03536 · Cited by 8
Cited by
NetCoMi: network construction and comparison for microbiome data in R
Peschel, S., Müller, C. L., von Mutius, E., Boulesteix, A.-L., & Depner, M. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa290 · Cited by 604
Chen, X., Lu, B., & Yan, C. (2018) · Human Brain Mapping · doi:10.1002/hbm.23843 · Cited by 340
Statistical Challenges in “Big Data” Human Neuroimaging
Smith, S. M., & Nichols, T. E. (2018) · Neuron · doi:10.1016/j.neuron.2017.12.018 · Cited by 330
Self-citationRegional, circuit and network heterogeneity of brain abnormalities in psychiatric disorders
Segal, A., Parkes, L., Aquino, K., Kia, S. M., Wolfers, T., Franke, B., Hoogman, M., Beckmann, C. F., Westlye, L. T., Andreassen, O. A., et al. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01404-6 · Cited by 271
Brain Heterogeneity in Schizophrenia and Its Association With Polygenic Risk
Alnæs, D., Kaufmann, T., van der Meer, D., Córdova-Palomera, A., Rokicki, J., Moberget, T., Bettella, F., Agartz, I., Barch, D. M., Bertolino, A., et al. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.0257 · Cited by 264
Metadata
- Authors
- Winkler, A. M., Ridgway, G. R., Douaud, G., Nichols, T. E., & Smith, S. M.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2016-11
- Volume
- 141
- Pages
- 502-516
- ISSN
- 1053-8119 (ISSN-L)
- OpenAlex
- W2418410189
- Cited by
- 329
- References
- 65
- FWCI
- 16.45
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.