Statistical inferences under the Null hypothesis: common mistakes and pitfalls in neuroimaging studies
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Published studies using functional and structural MRI include many errors in the way data are analyzed and conclusions reported. This was observed when working on a comprehensive review of the neural bases of synesthesia, but these errors are probably endemic to neuroimaging studies. All studies reviewed had based their conclusions using Null Hypothesis Significance Tests (NHST). NHST have yet been criticized since their inception because they are more appropriate for taking decisions related to a Null hypothesis (like in manufacturing) than for making inferences about behavioral and neuronal processes. Here I focus on a few key problems of NHST related to brain imaging techniques, and explain why or when we should not rely on "significance" tests. I also observed that, often, the ill-posed logic of NHST was even not correctly applied, and describe what I identified as common mistakes or at least problematic practices in published papers, in light of what could be considered as the very basics of statistical inference. MRI statistics also involve much more complex issues than standard statistical inference. Analysis pipelines vary a lot between studies, even for those using the same software, and there is no consensus which pipeline is the best. I propose a synthetic view of the logic behind the possible methodological choices, and warn against the usage and interpretation of two statistical methods popular in brain imaging studies, the false discovery rate (FDR) procedure and permutation tests. I suggest that current models for the analysis of brain imaging data suffer from serious limitations and call for a revision taking into account the "new statistics" (confidence intervals) logic.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 2×
“Most studies in neuroimaging (Yarkoni et al., 2010) and even neurosciences (Button et al., 2013) are underpowered.”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
SMITH, S., & NICHOLS, T. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.03.061 · Cited by 5,250
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · cited 1×
“But a circular error is made when using the same data to choose the “interesting” voxels and to test them (Kriegeskorte et al., 2009, 2010; Vul et al., 2009): the so-called a priori information is, in fact, defined a posteriori.”
MethodCluster-extent based thresholding in fMRI analyses: Pitfalls and recommendations
Woo, C.-W., Krishnan, A., & Wager, T. D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.12.058 · Cited by 1,253 · cited 1×
“…statistics, t or z-statistics thresholds should therefore be at least above 3 (Poline et al., 1997) 7http://www.fil.ion.ucl.ac.uk/spm/ Frontiers in Neuroscience | Brain Imaging Methods February 2015 | Volume 9 | Article 18 | 6 or 4 (Smith and Nichols, 2009; see also Woo et al., 2014).”
BackgroundBeyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034
Theory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962
Cited by
Van Dam, N. T., van Vugt, M. K., Vago, D. R., Schmalzl, L., Saron, C. D., Olendzki, A., Meissner, T., Lazar, S. W., Kerr, C. E., Gorchov, J., et al. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617709589 · Cited by 1,506
Fox, K. C. R., Dixon, M. L., Nijeboer, S., Girn, M., Floman, J. L., Lifshitz, M., Ellamil, M., Sedlmeier, P., & Christoff, K. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.03.021 · Cited by 656
The Resource Identification Initiative: A cultural shift in publishing
Bandrowski, A., Brush, M., Grethe, J. S., Haendel, M. A., Kennedy, D. N., Hill, S., Hof, P. R., Martone, M. E., Pols, M., Tan, S., et al. (2015) · F1000Research · doi:10.12688/f1000research.6555.2 · Cited by 80
Metacognitive ability correlates with hippocampal and prefrontal microstructure
Allen, M., Glen, J. C., Müllensiefen, D., Schwarzkopf, D. S., Fardo, F., Frank, D., Callaghan, M. F., & Rees, G. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.02.008 · Cited by 85
The resource identification initiative: a cultural shift in publishing
Bandrowski, A., Brush, M., Grethe, J. S., Haendel, M. A., Kennedy, D. N., Hill, S., Hof, P. R., Martone, M. E., Pols, M., Tan, S., et al. (2015) · F1000Research · doi:10.12688/f1000research.6555.1 · Cited by 50
Metadata
- Authors
- Hupé, J.-M.
- Venue
- Frontiers in Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2015-02-19
- Volume
- 9
- Pages
- 18-18
- ISSN
- 1662-453X (ISSN-L)
- OpenAlex
- W2134576402
- Cited by
- 44
- References
- 23
- FWCI
- 4.06
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Statistical inferences under the Null hypothesis: common mistakes and pitfalls in neuroimaging studies. https://app.cochat.ai/papers/10.3389/fnins.2015.00018
- The paper
- Hupé, J.-M. (2015). Statistical inferences under the Null hypothesis: common mistakes and pitfalls in neuroimaging studies. Frontiers in Neuroscience. https://doi.org/10.3389/fnins.2015.00018
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.