Robust is not necessarily reliable: From within-subjects fMRI contrasts to between-subjects comparisons
Open accessAbstract
Advances in cognitive and affective neuroscience come largely from within-subjects comparisons, in which the functional significance of neural activity is determined by contrasting two or more experimental conditions. Clinical and social neuroscience studies have attempted to leverage between-subject variability in such condition differences to better understand psychopathology and other individual differences. Shifting from within-to between-subjects comparisons requires that measures have adequate internal consistency to function as individual difference variables. This is particularly relevant for difference scores-which have lower reliability. The field has assumed reasonable internal consistency of neural measures based on consistent findings across studies (i.e., if a within-subject difference in neural activity is robust, then it must be reliable). Using one of the most common fMRI paradigms in the clinical neuroscience literature (i.e., a face- and shape-matching task), in a large sample of adolescents (N = 139) we replicate a robust finding: amygdala activation is greater for faces than shapes. Moreover, we demonstrate that the internal consistency of the amygdala in face and shape blocks was excellent (Spearman-Brown corrected reliability [SB] > .94). However, the internal consistency of the activation difference between faces and shapes was nearly zero (SB = -.06). This reflected the fact that the amygdala response to faces and shapes was highly correlated (r = .97) across individuals. Increased neural activation to faces versus shapes could not possibly function as an individual difference measure in these data-illustrating how neural activation can be robust within subjects, but unreliable as an individual difference measure. Strong and reproducible condition differences in neural activity are not necessarily well-suited for individual differences research-and neuroimaging studies should always report the internal consistency of, and correlations between, activations used in individual differences research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reliability of difference scores is always lower than reliability of their individual parts, which has been shown to diminish reliability of task-based fMRI and hinder mechanistic research.
“This contrast was used (cf. anticipation of social reward minus neutral outcomes) because reliability of difference scores is always lower than reliability of their individual parts,(53) which has been shown to diminish reliability of task-based fMRI(54) (see also(55)) and hinder mechanistic research.(56)”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
Genovese, C. R., Lazar, N. A., & Nichols, T. (2002) · NeuroImage · doi:10.1006/nimg.2001.1037 · Cited by 4,923 · see citing works
How we should measure "change": Or should we?
Cronbach, L. J., & Furby, L. (1970) · Psychological Bulletin · doi:10.1037/h0029382 · Cited by 2,456 · see citing works
Region of interest analysis using an SPM toolbox
Brett, M. (2002) · NeuroImage · Cited by 2,273 · see citing works
The Emotional Brain, Fear, and the Amygdala
LeDoux, J. (2003) · Cellular and Molecular Neurobiology · doi:10.1023/a:1025048802629 · Cited by 1,818 · see citing works
Neuronal circuits for fear and anxiety
Tovote, P., Fadok, J. P., & Lüthi, A. (2015) · Nature Reviews Neuroscience · doi:10.1038/nrn3945 · Cited by 1,786 · see citing works
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457 · see citing works
Correction for geometric distortion in echo planar images from B0 field variations
Jezzard, P., & Balaban, R. S. (1995) · Magnetic Resonance in Medicine · doi:10.1002/mrm.1910340111 · Cited by 1,428 · see citing works
The Amygdala Response to Emotional Stimuli: A Comparison of Faces and Scenes
Hariri, A. R., Tessitore, A., Mattay, V. S., Fera, F., & Weinberger, D. R. (2002) · NeuroImage · doi:10.1006/nimg.2002.1179 · Cited by 984 · see citing works
Sergerie, K., Chochol, C., & Armony, J. L. (2008) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2007.12.002 · Cited by 917 · see citing works
Increased Amygdala and Insula Activation During Emotion Processing in Anxiety-Prone Subjects
Stein, M. B., Simmons, A. N., Feinstein, J. S., & Paulus, M. P. (2007) · American Journal of Psychiatry · doi:10.1176/ajp.2007.164.2.318 · Cited by 851 · see citing works
A Susceptibility Gene for Affective Disorders and the Response of the Human Amygdala
Hariri, A. R., Drabant, E. M., Munoz, K. E., Kolachana, B. S., Mattay, V. S., Egan, M. F., & Weinberger, D. R. (2005) · Archives of General Psychiatry · doi:10.1001/archpsyc.62.2.146 · Cited by 815 · see citing works
Developing constructs for psychopathology research: Research domain criteria
Sanislow, C. A., Pine, D. S., Quinn, K. J., Kozak, M. J., Garvey, M. A., Heinssen, R. K., Wang, P. S.-E., & Cuthbert, B. N. (2010) · Journal of Abnormal Psychology · doi:10.1037/a0020909 · Cited by 793 · see citing works
The NIMH research domain criteria initiative: Background, issues, and pragmatics
Kozak, M. J., & Cuthbert, B. N. (2016) · Psychophysiology · doi:10.1111/psyp.12518 · Cited by 655 · see citing works
PITMAN, E. J. G. (1939) · Biometrika · doi:10.1093/biomet/31.1-2.9 · Cited by 480 · see citing works
Questions and Answers in the Measurement of Change
Willett, J. B. (1988) · Review of Research in Education · doi:10.2307/1167368 · Cited by 461 · see citing works
A fast pathway for fear in human amygdala
Méndez‐Bértolo, C., Moratti, S., Toledano, R., López-Sosa, F., Martínez-Alvarez, R., Mah, Y., Vuilleumier, P. O., Gil‐Nagel, A., & Strange, B. A. (2016) · Nature Neuroscience · doi:10.1038/nn.4324 · Cited by 396 · see citing works
The Influence of Working-Memory Demand and Subject Performance on Prefrontal Cortical Activity
Rypma, B., Berger, J. S., & D’Esposito, M. (2002) · Journal of Cognitive Neuroscience · doi:10.1162/08989290260138627 · Cited by 364 · see citing works
EGGER, H. L., Pine, D. S., Nelson, E. E., Leibenluft, E., Monique Ernst, D., Towbin, K. E., & Angold, A. C. (2011) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.343 · Cited by 295 · see citing works
Dopamine Modulates the Response of the Human Amygdala: A Study in Parkinson's Disease
Tessitore, A., Hariri, A. R., Fera, F., Smith, W. G., Chase, T. N., Hyde, T. M., Weinberger, D. R., & Mattay, V. S. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-20-09099.2002 · Cited by 293 · see citing works
Individual differences in typical reappraisal use predict amygdala and prefrontal responses
Drabant, E. M., McRae, K., Manuck, S. B., Hariri, A. R., & Gross, J. J. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.09.007 · Cited by 270 · see citing works
Meyer, A., Lerner, M. D., De Los Reyes, A., Laird, R. D., & Hajcak, G. (2017) · Psychophysiology · doi:10.1111/psyp.12664 · Cited by 251 · see citing works
Self-citationLord, F. M. (1956) · Educational and Psychological Measurement · doi:10.1177/001316445601600401 · Cited by 247 · see citing works
Assessment of spatial normalization of whole‐brain magnetic resonance images in children
Wilke, M., Schmithorst, V. J., & Holland, S. K. (2002) · Human Brain Mapping · doi:10.1002/hbm.10053 · Cited by 225 · see citing works
Bertolino, A., Arciero, G., Rubino, V., Latorre, V., Candia, M. D., Mazzola, V., Blasi, G., Caforio, G., Hariri, A. R., Kolachana, B. S., et al. (2005) · Biological Psychiatry · doi:10.1016/j.biopsych.2005.02.031 · Cited by 223 · see citing works
Morgan, W. A. (1939) · Biometrika · doi:10.1093/biomet/31.1-2.13 · Cited by 215 · see citing works
Kleinhans, N. M., Richards, T., Weaver, K., Johnson, L. C., Greenson, J., Dawson, G., & Aylward, E. (2010) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2010.07.022 · Cited by 187 · see citing works
Chapter 9: Questions and Answers in the Measurement of Change
Willett, J. B. (1988) · Review of Research in Education · doi:10.3102/0091732x015001345 · Cited by 135 · see citing works
Rasetti, R., Mattay, V. S., Wiedholz, L. M., Kolachana, B. S., Hariri, A. R., Callicott, J. H., Meyer‐Lindenberg, A., & Weinberger, D. R. (2008) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2008.08020261 · Cited by 123 · see citing works
Hajcak, G., Meyer, A. M., & Kotov, R. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000274 · Cited by 116 · see citing works
Self-citationRDoC: Translating promise into progress
Patrick, C. J., & Hajcak, G. (2016) · Psychophysiology · doi:10.1111/psyp.12612 · Cited by 114 · see citing works
Self-citationLuking, K. R., Nelson, B. D., Infantolino, Z. P., Sauder, C. L., & Hajcak, G. (2016) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2016.12.004 · Cited by 85 · see citing works
Self-citationReliability of composite-task measurements of holistic face processing
Ross, D. A., Richler, J. J., & Gauthier, L. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0497-4 · Cited by 54 · see citing works
Marusak, H. A., Carré, J., & Thomason, M. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.07.002 · Cited by 49 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Chiou JS Spreng RA 1996 The reliability of difference scores: A re-examination Journal of Consumer Satisfaction, Dissatisfaction, and Complaining Behavior 9 158 167
- Zumbo BD 1999 The simple difference score as an inherently poor measure of change: Some reality, much mythology Thompson B Advances in Social Science Methodoloy 269 304 Greenwich JAI Press
Cited by
Sorted by citations
Elliott, M. L., Knodt, A. R., Ireland, D., Morris, M. L., Poulton, R., Ramrakha, S., Sison, M. L., Moffitt, T. E., Caspi, A., & Hariri, A. R. (2020) · Psychological Science · doi:10.1177/0956797620916786 · Cited by 714 · see citing works
Taylor, C. T., Stein, M. B., Simmons, A. N., He, F., Oveis, C., Shakya, H. B., Sieber, W. J., Fowler, J. H., & Jain, S. (2024) · Biological Psychiatry · doi:10.1016/j.biopsych.2023.07.024 · Cited by 53 · see citing works
Clayson, P. E., Kappenman, E. S., Gehring, W. J., Miller, G. A., & Larson, M. J. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.117932 · Cited by 28 · see citing works
- Authors
- Infantolino, Z. P., Luking, K. R., Sauder, C. L., Curtin, J. J., & Hajcak, G.
- Affiliations
- Stony Brook University (US)
- The University of Texas at San Antonio Health Science Center (US)
- University of Wisconsin–Madison (US)
- Florida State University (US)
- Corresponding author
- Infantolino, Zachary P.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2018-06
- Volume
- 173
- Pages
- 146-152
- ISSN
- 1053-8119 (ISSN-L)
- PMC
- PMC5912348
- OpenAlex
- W2793110972
- Cited by
- 108
- References
- 33
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 13 · 2023: 14 · 2022: 15 · 2021: 26 · 2020: 12 · 2019: 11 · 2018: 2
- FWCI
- 7.59
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- NeuroImage (acceptedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (R01 MH097767)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- internal consistency, individual differences, amygdala activation, face processing, adolescents
- MeSH terms
- Adolescent, Brain, Brain Mapping, Child, Female, Humans, Image Processing, Computer-Assisted, Magnetic Resonance Imaging, Pattern Recognition, Visual, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
