Avoiding non-independence in fMRI data analysis: Leave one subject out
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Concerns regarding certain fMRI data analysis practices have recently evoked lively debate. The principal concern regards the issue of non-independence, in which an initial statistical test is followed by further non-independent statistical tests. In this report, we propose a simple, practical solution to reduce bias in secondary tests due to nonindependence using a leave-one-subject-out (LOSO) approach. We provide examples of this method, show how it reduces effect size inflation, and suggest that it can serve as a functional localizer when within-subject methods are impractical.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The fusiform face area: a module in human extrastriate cortex specialized for face perception
Kanwisher, N., McDermott, J., & Chun, M. M. (1997) · The Journal of Neuroscience · doi:10.1523/jneurosci.17-11-04302.1997 · Cited by 7,738 · cited 2×
“…in the middle fusiform gyrus, consistent with the left and right fusiform face area (FFA; Sergent et al., 1992; Allison et al., 1994; Puce et al., 1995; Kanwisher et al., 1997) and another face-selective region in the right superior temporal sulcus (STS; Puce et al., 1998; Haxby et al., 1999).”
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696 · cited 1×
“Another recent article (Kriegeskorte et al., 2009) addressed the problem of non-independence in a broader range of neuroimaging analyses and suggested a step-by-step procedure for assessing and avoiding these kinds of errors.”
Linear Systems Analysis of Functional Magnetic Resonance Imaging in Human V1
Boynton, G. M., Engel, S. A., Glover, G. H., & Heeger, D. J. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-13-04207.1996 · Cited by 2,314
FUNCTIONAL NEUROANATOMY OF FACE AND OBJECT PROCESSING: A POSITRON EMISSION TOMOGRAPHY STUDY
SERGENT, J., OHTA, S., & MACDONALD, B. (1992) · Brain · doi:10.1093/brain/115.1.15 · Cited by 1,517
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · cited 1×
“Although this procedure will radically reduce inflation of effect size, simulations suggest that there remains the possibility of some residual inflation (Ed Vul, 2009, personal communication), especially when considering a small number of voxels.”
Temporal Cortex Activation in Humans Viewing Eye and Mouth Movements
Puce, A., Allison, T., Bentin, S., Gore, J. C., & McCarthy, G. (1998) · The Journal of Neuroscience · doi:10.1523/jneurosci.18-06-02188.1998 · Cited by 1,077
The fusiform “face area” is part of a network that processes faces at the individual level
Gauthier, I., Tarr, M. J., Moylan, J., Skudlarski, P., Gore, J. C., & Anderson, A. W. (2000) · Journal of Cognitive Neuroscience · doi:10.1162/089892900562165 · Cited by 859
Rossion, B. (2003) · Brain · doi:10.1093/brain/awg241 · Cited by 716
Face-sensitive regions in human extrastriate cortex studied by functional MRI
Puce, A., Allison, T., Gore, J. C., & McCarthy, G. (1995) · Journal of Neurophysiology · doi:10.1152/jn.1995.74.3.1192 · Cited by 690
The Effect of Face Inversion on Activity in Human Neural Systems for Face and Object Perception
Haxby, J. V., Ungerleider, L. G., Clark, V. P., Schouten, J. L., Hoffman, E. A., & Martin, A. (1999) · Neuron · doi:10.1016/s0896-6273(00)80690-x · Cited by 657
Cited by
Neural computations underlying arbitration between model-based and model-free learning
Lee, S. W., Shimojo, S., & O’Doherty, J. P. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.028 · Cited by 642
Foundations of human reasoning in the prefrontal cortex
Donoso, M., Collins, A. G. E., & Koechlin, E. (2014) · Science · doi:10.1126/science.1252254 · Cited by 406
Activity Recognition Using a Single Accelerometer Placed at the Wrist or Ankle
MANNINI, A., INTILLE, S. S., ROSENBERGER, M., SABATINI, A. M., & HASKELL, W. (2013) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0b013e31829736d6 · Cited by 383
Mannini, A., Trojaniello, D., Cereatti, A., & Sabatini, A. (2016) · Sensors · doi:10.3390/s16010134 · Cited by 242
“Such a validation approach is particularly useful in evaluating the inter-subject generalization capability of the proposed methodology [8,35].”
Social Manipulation of Preference in the Human Brain
Izuma, K., & Adolphs, R. (2013) · Neuron · doi:10.1016/j.neuron.2013.03.023 · Cited by 224
“analyses were carried out using a leave-one-subject-out (LOSO) crossvalidation procedure (Esterman et al., 2010) in order to eliminate nonindependence bias for plots of parameter estimates.”
Stimulus-specific enhancement of fear extinction during slow-wave sleep
Hauner, K. K., Howard, J. D., Zelano, C., & Gottfried, J. A. (2013) · Nature Neuroscience · doi:10.1038/nn.3527 · Cited by 173
How Self-Determined Choice Facilitates Performance: A Key Role of the Ventromedial Prefrontal Cortex
Murayama, K., Matsumoto, M., Izuma, K., Sugiura, A., Ryan, R. M., Deci, E. L., & Matsumoto, K. (2015) · Cerebral Cortex · doi:10.1093/cercor/bht317 · Cited by 143
Evans, S., & Davis, M. H. (2015) · Cerebral Cortex · doi:10.1093/cercor/bhv136 · Cited by 130
Automatic versus Choice-Dependent Value Representations in the Human Brain
Grueschow, M., Polania, R., Hare, T. A., & Ruff, C. C. (2015) · Neuron · doi:10.1016/j.neuron.2014.12.054 · Cited by 130
Liljeholm, M., Tricomi, E., O'Doherty, J. P., & Balleine, B. W. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.3354-10.2011 · Cited by 130
“To eliminate nonindependence bias for plots of parameter estimates, a leave-one-subject-out (LOSO) (Esterman et al., 2010) approach was used in which 19 general linear models (GLMs) were run with one subject left out in each and with each GLM defining the voxel cluster for the subject that was left…”
BackgroundMethodDriving Performance Under Violations of Traffic Rules: Novice vs. Experienced Drivers
Xu, J., Guo, K., & Sun, P. Z. H. (2022) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2022.3200592 · Cited by 116
Real-world stress resilience is associated with the responsivity of the locus coeruleus
Grueschow, M., Stenz, N., Thörn, H., Ehlert, U., Breckwoldt, J., Brodmann Maeder, M., Exadaktylos, A. K., Bingisser, R., Ruff, C. C., & Kleim, B. (2021) · Nature Communications · doi:10.1038/s41467-021-22509-1 · Cited by 116
Dopamine Modulates Adaptive Prediction Error Coding in the Human Midbrain and Striatum
Diederen, K. M. J., Ziauddeen, H., Vestergaard, M. D., Spencer, T., Schultz, W., & Fletcher, P. C. (2017) · The Journal of Neuroscience · doi:10.1523/jneurosci.1979-16.2016 · Cited by 110
“As we had no strong a priori hypotheses about brain areas encoding the instructional cues that predicted reward variability, we explored the effect of dopaminergic modulation on the neural responses to the cues using a leave-onesubject-out approach (Esterman et al., 2010).”
MethodAcupuncture treatment modulates the corticostriatal reward circuitry in major depressive disorder
Wang, Z., Wang, X., Liu, J., Chen, J., Liu, X., Nie, G., Jorgenson, K., Sohn, K. C., Huang, R., Liu, M., et al. (2017) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2016.09.014 · Cited by 108
Subjective confidence reflects representation of Bayesian probability in cortex
Geurts, L. S., Cooke, J. R. H., van Bergen, R. S., & Jehee, J. F. M. (2022) · Nature Human Behaviour · doi:10.1038/s41562-021-01247-w · Cited by 104
Increased Activity Imbalance in Fronto-Subcortical Circuits in Adolescents with Major Depression
Jiao, Q., Ding, J., Lu, G., Su, L., Zhang, Z., Wang, Z., Zhong, Y., Li, K., Ding, M., & Liu, Y. (2011) · PLoS ONE · doi:10.1371/journal.pone.0025159 · Cited by 83
Distinct Regions within Medial Prefrontal Cortex Process Pain and Cognition
Jahn, A., Nee, D. E., Alexander, W. H., & Brown, J. W. (2016) · The Journal of Neuroscience · doi:10.1523/jneurosci.2180-16.2016 · Cited by 75
“To create data-driven, independent ROIs to examine each effect, we used a leave-one-out cross-validation method (Esterman et al., 2010).”
MethodPresynaptic Serotoninergic Regulation of Emotional Processing: A Multimodal Brain Imaging Study
Selvaraj, S., Mouchlianitis, E., Faulkner, P., Turkheimer, F., Cowen, P. J., Roiser, J. P., & Howes, O. (2015) · Biological Psychiatry · doi:10.1016/j.biopsych.2014.04.011 · Cited by 29
Metadata
- Authors
- Esterman, M., Tamber-Rosenau, B. J., Chiu, Y.-C., & Yantis, S.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2010-04
- Volume
- 50
- Issue
- 2
- Pages
- 572-576
- ISSN
- 1053-8119 (ISSN-L)
- OpenAlex
- W2114667458
- Cited by
- 261
- References
- 15
- FWCI
- 1.83
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.