What is the test–retest reliability of common task-functional MRI measures? New empirical evidence and a meta-analysis
Open accessAbstract
Identifying brain biomarkers of disease risk is a growing priority in neuroscience. The ability to identify meaningful biomarkers is limited by measurement reliability; unreliable measures are unsuitable for predicting clinical outcomes. Measuring brain activity using task functional MRI (fMRI) is a major focus of biomarker development; however, the reliability of task fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. First, a meta-analysis of 90 experiments ( N = 1,008) revealed poor overall reliability—mean intraclass correlation coefficient (ICC) = .397. Second, the test-retest reliabilities of activity in a priori regions of interest across 11 common fMRI tasks collected by the Human Connectome Project ( N = 45) and the Dunedin Study ( N = 20) were poor (ICCs = .067–.485). Collectively, these findings demonstrate that common task-fMRI measures are not currently suitable for brain biomarker discovery or for individual-differences research. We review how this state of affairs came to be and highlight avenues for improving task-fMRI reliability.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reliability of difference scores is always lower than reliability of their individual parts, which has been shown to diminish reliability of task-based fMRI and hinder mechanistic research.
“This contrast was used (cf. anticipation of social reward minus neutral outcomes) because reliability of difference scores is always lower than reliability of their individual parts,(53) which has been shown to diminish reliability of task-based fMRI(54) (see also(55)) and hinder mechanistic research.(56)”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · see citing works
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799 · see citing works
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1× · see citing works
“It has been suggested that neuroscience is generally an underpowered enterprise and that small sample sizes undermine fMRI research in particular (Button et al., 2013).”
Neurophysiological investigation of the basis of the fMRI signal
Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., & Oeltermann, A. (2001) · Nature · doi:10.1038/35084005 · Cited by 6,542 · see citing works
The WU-Minn Human Connectome Project: An overview
Van Essen, D. C., Smith, S. M., Barch, D. M., Behrens, T. E. J., Yacoub, E., & Ugurbil, K. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.05.041 · Cited by 5,651 · see citing works
A multi-modal parcellation of human cerebral cortex
Glasser, M. F., Coalson, T. S., Robinson, E. C., Hacker, C. D., Harwell, J., Yacoub, E., Ugurbil, K., Andersson, J., Beckmann, C. F., Jenkinson, M., et al. (2016) · Nature · doi:10.1038/nature18933 · Cited by 5,111 · see citing works
Introduction to Classical and Modern Test Theory
Crocker, L. M., & Algina, J. J. (1986) · Cited by 4,803 · see citing works
Evaluating Effect Size in Psychological Research: Sense and Nonsense
Funder, D. C., & Ozer, D. J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847202 · Cited by 3,295 · cited 1× · see citing works
“For an r of .1 (a small effect size common in psychological research; Funder & Ozer, 2019), adequately powered studies require a total sample of more than 2,000.”
The two disciplines of scientific psychology.
Cronbach, L. J. (1957) · American Psychologist · doi:10.1037/h0043943 · Cited by 2,556 · see citing works
Beyond mind-reading: multi-voxel pattern analysis of fMRI data
Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.07.005 · Cited by 2,271 · see citing works
(1981) · PubMed · Cited by 2,253 · see citing works
At what sample size do correlations stabilize?
Schönbrodt, F. D., & Perugini, M. (2013) · Journal of Research in Personality · doi:10.1016/j.jrp.2013.05.009 · Cited by 2,232 · see citing works
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,930 · see citing works
Function in the human connectome: task-fMRI and individual differences in behavior
Barch, D. M., Burgess, G. C., Harms, M. P., Petersen, S. E., Schlaggar, B. L., Corbetta, M., Glasser, M. F., Curtiss, S., Dixit, S., Feldt, C., et al. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.05.033 · Cited by 1,529 · see citing works
Han, X., Jovicich, J., Salat, D., van der Kouwe, A., Quinn, B., Czanner, S., Busa, E., Pacheco, J., Albert, M., Killiany, R., et al. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.02.051 · Cited by 1,508 · see citing works
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · see citing works
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457 · see citing works
Scanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410 · see citing works
Precision Functional Mapping of Individual Human Brains
Gordon, E. M., Laumann, T. O., Gilmore, A. W., Newbold, D. J., Greene, D. J., Berg, J. J., Ortega, M., Hoyt-Drazen, C., Gratton, C., Sun, H., et al. (2017) · Neuron · doi:10.1016/j.neuron.2017.07.011 · Cited by 1,335 · see citing works
Building better biomarkers: brain models in translational neuroimaging
Woo, C.-W., Chang, L. J., Lindquist, M. A., & Wager, T. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4478 · Cited by 1,056 · see citing works
Introduction to Psychological Measurement
Nunnally, J. C. (1970) · Medical Entomology and Zoology · Cited by 1,045 · see citing works
Gratton, C., Laumann, T. O., Nielsen, A. N., Greene, D. J., Gordon, E. M., Gilmore, A. W., Nelson, S. M., Coalson, R. S., Snyder, A. Z., Schlaggar, B. L., et al. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.035 · Cited by 846 · see citing works
Schäfer, T., & Schwarz, M. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00813 · Cited by 774 · see citing works
Building a Science of Individual Differences from fMRI
Dubois, J., & Adolphs, R. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.03.014 · Cited by 610 · see citing works
How reliable are the results from functional magnetic resonance imaging?
Bennett, C. M., & Miller, M. B. (2010) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2010.05446.x · Cited by 579 · see citing works
Yarkoni, T. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01127.x · Cited by 549 · see citing works
Linear mixed-effects modeling approach to FMRI group analysis
Chen, G., Saad, Z. S., Britton, J. C., Pine, D. S., & Cox, R. W. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.01.047 · Cited by 440 · see citing works
Poulton, R., Moffitt, T. E., & Silva, P. A. (2015) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-015-1048-8 · Cited by 394 · see citing works
Self-citationElliott, M. L., Knodt, A. R., Cooke, M. E., Kim, M. J., Melzer, T. R., Keenan, R. J., Ireland, D. R., Ramrakha, S., Poulton, R., Caspi, A., et al. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.01.068 · Cited by 306 · see citing works
Self-citationMovies in the magnet: Naturalistic paradigms in developmental functional neuroimaging
Vanderwal, T., Eilbott, J. A., & Castellanos, F. X. (2018) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2018.10.004 · Cited by 270 · see citing works
A Neural Biomarker of Psychological Vulnerability to Future Life Stress
Swartz, J. R., Knodt, A. R., Radtke, S. R., & Hariri, A. R. (2015) · Neuron · doi:10.1016/j.neuron.2014.12.055 · Cited by 216 · see citing works
Self-citationEverything You Never Wanted to Know about Circular Analysis, but Were Afraid to Ask
Kriegeskorte, N., Lindquist, M. A., Nichols, T. E., Poldrack, R. A., & Vul, E. (2010) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2010.86 · Cited by 214 · see citing works
Intraclass correlation: Improved modeling approaches and applications for neuroimaging
Chen, G., Taylor, P. A., Haller, S. P., Kircanski, K., Stoddard, J., Pine, D. S., Leibenluft, E., Brotman, M. A., & Cox, R. W. (2017) · Human Brain Mapping · doi:10.1002/hbm.23909 · Cited by 135 · see citing works
Hajcak, G., Meyer, A. M., & Kotov, R. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000274 · Cited by 116 · see citing works
Test-retest reliability of longitudinal task-based fMRI: Implications for developmental studies
Herting, M. M., Gautam, P., Chen, Z., Mezher, A. F., & Vetter, N. C. (2017) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2017.07.001 · Cited by 112 · see citing works
Temporal stability of individual differences in amygdala reactivity
MANUCK, S. B., BROWN, S. M., FORBES, E. E., & HARIRI, A. R. (2007) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2007.07040609 · Cited by 111 · see citing works
Self-citationInfantolino, Z. P., Luking, K. R., Sauder, C. L., Curtin, J. J., & Hajcak, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.02.024 · Cited by 108 · see citing works
Unreliability of putative fMRI biomarkers during emotional face processing
Nord, C. L., Gray, A., Charpentier, C. J., Robinson, O. J., & Roiser, J. P. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.05.024 · Cited by 96 · see citing works
Fröhner, J. H., Teckentrup, V., Smolka, M. N., & Kroemer, N. B. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.03.053 · Cited by 92 · see citing works
Neuroimaging of individual differences: A latent variable modeling perspective
Cooper, S. R., Jackson, J. J., Barch, D. M., & Braver, T. S. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.12.022 · Cited by 57 · see citing works
Developmental Cognitive Neuroscience
Bowman, L. C. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-809324-5.23066-6 · Cited by 52 · see citing works
A Polygenic Score for Higher Educational Attainment is Associated with Larger Brains
Elliott, M. L., Belsky, D. W., Anderson, K., Corcoran, D. L., Ge, T., Knodt, A., Prinz, J. A., Sugden, K., Williams, B., Ireland, D., et al. (2019) · Cerebral Cortex · doi:10.1093/cercor/bhy219 · Cited by 50 · see citing works
Self-citationTHE AXIOMS AND PRINCIPAL RESULTS OF CLASSICAL TEST THEORY
Novick, M. R. (1965) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1965.tb00132.x · Cited by 34 · see citing works
Advanced Research Methods for the Social and Behavioral Sciences
Edlund, J. E., & Nichols, A. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108349383 · Cited by 32 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Grubbs, J. B., Hoagland, K. C., Lee, B. N., Grant, J. T., Davison, P., Reid, R. C., & Kraus, S. W. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101925 · Cited by 253 · see citing works
Bickman, L. (2020) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-020-01065-8 · Cited by 184 · see citing works
Genetic variation, brain, and intelligence differences
Deary, I. J., Cox, S. R., & Hill, W. D. (2022) · Molecular Psychiatry · doi:10.1038/s41380-021-01027-y · Cited by 137 · see citing works
Biomarkers of Pathological Dissociation: A Systematic Review
Roydeva, M. I., & Reinders, A. A. T. S. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.11.019 · Cited by 116 · see citing works
Dysregulated resting state functional connectivity and obesity: A systematic review
Syan, S. K., McIntyre-Wood, C., Minuzzi, L., Hall, G., McCabe, R. E., & MacKillop, J. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.08.019 · Cited by 66 · see citing works
Taylor, C. T., Stein, M. B., Simmons, A. N., He, F., Oveis, C., Shakya, H. B., Sieber, W. J., Fowler, J. H., & Jain, S. (2024) · Biological Psychiatry · doi:10.1016/j.biopsych.2023.07.024 · Cited by 53 · see citing works
Moment-to-Moment Brain Signal Variability Reliably Predicts Psychiatric Treatment Outcome
Månsson, K. N. T., Waschke, L., Manzouri, A., Furmark, T., Fischer, H., & Garrett, D. D. (2022) · Biological Psychiatry · doi:10.1016/j.biopsych.2021.09.026 · Cited by 49 · see citing works
Neurorights, mental privacy, and mind reading
Brown, C. M. L. (2024) · Neuroethics · doi:10.1007/s12152-024-09568-z · Cited by 33 · see citing works
Tamm, S., Harmer, C. J., Schiel, J., Holub, F., Rutter, M. K., Spiegelhalder, K., & Kyle, S. D. (2022) · American Journal of Psychiatry · doi:10.1176/appi.ajp.21050466 · Cited by 22 · see citing works
Crone, E. A., & van Drunen, L. (2024) · Human Development · doi:10.1159/000539844 · Cited by 17 · see citing works
“Thus, when interpreting change scores with low and high variability, it is crucial to not too hastily conclude that measures with high variability lack meaningfulness in the understanding of growth (Elliott et al., 2020).”
Is Adolescence a Sensitive Period for the Development of Incentive-Reward Motivation?
Luciana, M., & Collins, P. F. (2021) · Current Topics in Behavioral Neurosciences · doi:10.1007/7854_2021_275 · Cited by 7 · see citing works
- Authors
- Elliott, M. L., Knodt, A. R., Ireland, D., Morris, M. L., Poulton, R., Ramrakha, S., Sison, M. L., Moffitt, T. E., Caspi, A., & Hariri, A. R.
- Affiliations
- Duke University (US)
- University of Otago (NZ)
- Corresponding author
- Hariri, Ahmad R.
- ORCID
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2020-06-03
- Volume
- 31
- Issue
- 7
- Pages
- 792-806
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- PMC
- PMC7370246
- OpenAlex
- W3032707732
- Cited by
- 714
- References
- 47
- Citations by year
- 2026: 73 · 2025: 82 · 2024: 107 · 2023: 116 · 2022: 129 · 2021: 146 · 2020: 55 · 2019: 4 · 2018: 1
- FWCI
- 15.06
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Psychological Science (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Research Portal (King's College London) (submittedVersion)
- Funders
- U.K. Medical Research Council (P005918)
- National Institute of Aging (R01AG032282, R01AG049789)
- National Science Foundation (NSF DGE-1644868)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
- Keywords
- test-retest reliability, measurement reliability, brain biomarkers, intraclass correlation coefficient, region of interest, individual differences, Human Connectome Project, Dunedin Study
- MeSH terms
- Brain, Cognition, Humans, Individuality, Magnetic Resonance Imaging, Reproducibility of Results, Connectome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
