Big correlations in little studies: Inflated fMRI correlations reflect low statistical power – Commentary on Vul et al. (2009)
Abstract
Vul, Harris, Winkielman, and Pashler (2009), (this issue) argue that correlations in many cognitive neuroscience studies are grossly inflated due to a widespread tendency to use nonindependent analyses. In this article, I argue that Vul et al.'s primary conclusion is correct, but for different reasons than they suggest. I demonstrate that the primary cause of grossly inflated correlations in whole-brain fMRI analyses is not nonindependence, but the pernicious combination of small sample sizes and stringent alpha-correction levels. Far from defusing Vul et al.'s conclusions, the simulations presented suggest that the level of inflation may be even worse than Vul et al.'s empirical analysis would suggest.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Individual neuroimaging studies are often underpowered and exhibit relatively high false positive rates.
“individual neuroimaging studies are often underpowered and exhibit relatively high false positive rate2-4”
From Large-scale automated synthesis of human functional neuroimaging data. (2011) · cites this paper as [2]
Effect sizes for statistically significant effects typically vary inversely with sample size, because when power is relatively low, one must capitalize on chance in order to obtain statistically significant results.
“It is important to remember, however, that effect sizes for statistically significant effects typically vary inversely with sample size, because when power is relatively low, one must capitalize on chance in order to obtain statistically significant results (Ioannidis, 2008; Yarkoni, 2009).”
From Personality in 100,000 Words: A large-scale analysis of personality and word use among bloggers (2010) · cites this paper as Yarkoni, 2009
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771 · see citing works
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · cited 1× · see citing works
“Inflated fMRI Correlations Reflect Low Statistical Power— Commentary on Vul et al. (2009)”
BackgroundPsychological testing and psychological assessment: A review of evidence and issues.
Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., Eisman, E. J., Kubiszyn, T. W., & Reed, G. M. (2001) · American Psychologist · doi:10.1037//0003-066x.56.2.128 · Cited by 1,469 · see citing works
QUANTITATIVE METHODS IN PSYCHOLOGY A Power Primer
Cohen, J. (1992) · Cited by 1,040 · see citing works
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · see citing works
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655 · see citing works
Handbook of Individual Differences in Cognition
Gruszka, A., Matthews, G., & Szymura, B. (2010) · Plenum series on human exceptionality · doi:10.1007/978-1-4419-1210-7 · Cited by 232 · see citing works
Affective personality differences in neural processing efficiency confirmed using fMRI
Gray, J. R., Burgess, G. C., Schaefer, A., Yarkoni, T., LARSEN, R. J., & Braver, T. S. (2005) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.5.2.182 · Cited by 210 · see citing works
Self-citationPersonality predicts working-memory-related activation in the caudal anterior cingulate cortex
GRAY, J. R., & BRAVER, T. S. (2002) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.2.1.64 · Cited by 175 · see citing works
Yarkoni, T., & Braver, T. S. (2010) · Plenum series on human exceptionality · doi:10.1007/978-1-4419-1210-7_6 · Cited by 140 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Large-scale automated synthesis of human functional neuroimaging data.
Yarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C., & Wager, T. D. (2011) · Nature Methods · doi:10.1038/nmeth.1635 · Cited by 3,686 · see citing works
Self-citationThe brain basis of emotion: a meta-analytic review
Lindquist, K. A., Wager, T. D., Kober, H., Bliss-Moreau, E., & Barrett, L. F. (2012) · Behavioral and Brain Sciences · doi:10.1017/s0140525x11000446 · Cited by 2,320 · see citing works
“First, metaanalysis can weed out false positives (which are frequent in neuroimaging studies; see Wager et al. 2007; Yarkoni 2009) and focus interpretation on the activations that are consistently observed.”
BackgroundScanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410 · see citing works
Self-citationPuzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · see citing works
Szucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761 · see citing works
Elliott, M. L., Knodt, A. R., Ireland, D., Morris, M. L., Poulton, R., Ramrakha, S., Sison, M. L., Moffitt, T. E., Caspi, A., & Hariri, A. R. (2020) · Psychological Science · doi:10.1177/0956797620916786 · Cited by 714 · see citing works
Fox, K. C. R., Dixon, M. L., Nijeboer, S., Girn, M., Floman, J. L., Lifshitz, M., Ellamil, M., Sedlmeier, P., & Christoff, K. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.03.021 · Cited by 656 · see citing works
Evidence for a large-scale brain system supporting allostasis and interoception in humans
Kleckner, I. R., Zhang, J., Touroutoglou, A., Chanes, L., Xia, C., Simmons, W. K., Quigley, K. S., Dickerson, B. C., & Feldman Barrett, L. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0069 · Cited by 641 · see citing works
Global Connectivity of Prefrontal Cortex Predicts Cognitive Control and Intelligence
Cole, M. W., Yarkoni, T., Repovs, G., Anticevic, A., & Braver, T. S. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.0536-12.2012 · Cited by 613 · see citing works
Self-citationPersonality in 100,000 Words: A large-scale analysis of personality and word use among bloggers
Yarkoni, T. (2010) · Journal of Research in Personality · doi:10.1016/j.jrp.2010.04.001 · Cited by 637 · see citing works
Self-citationYarkoni, T. (2022) · Behavioral and Brain Sciences · doi:10.1017/s0140525x20001685 · Cited by 701 · see citing works
Self-citationThe ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455 · see citing works
Wager, T. D., Atlas, L. Y., Leotti, L. A., & Rilling, J. K. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.3420-10.2011 · Cited by 328 · see citing works
Evaluation of the Social Motivation Hypothesis of Autism: A Systematic Review and Meta-analysis
Clements, C. C., Zoltowski, A. R., Yankowitz, L. D., Yerys, B. E., Schultz, R. T., & Herrington, J. D. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.1100 · Cited by 316 · see citing works
Implementing Reproducible Research
Stodden, V. (2014) · doi:10.1201/b16868 · Cited by 314 · see citing works
Structural disconnections explain brain network dysfunction after stroke
Griffis, J. C., Metcalf, N. V., Corbetta, M., & Shulman, G. L. (2019) · Cell Reports · doi:10.1016/j.celrep.2019.07.100 · Cited by 213 · see citing works
Goodman, M., Carpenter, D., Tang, C. Y., Goldstein, K. E., Avedon, J., Fernandez, N., Mascitelli, K. A., Blair, N. J., New, A. S., Triebwasser, J., et al. (2014) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2014.06.020 · Cited by 198 · see citing works
Cognitive neuroscience 2.0: building a cumulative science of human brain function
Yarkoni, T., Poldrack, R. A., Van Essen, D. C., & Wager, T. D. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.08.004 · Cited by 174 · see citing works
Self-citationA power calculation guide for fMRI studies
Mumford, J. A. (2012) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nss059 · Cited by 171 · see citing works
Schonberg, T., Fox, C. R., Mumford, J. A., Congdon, E., Trepel, C., & Poldrack, R. A. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00080 · Cited by 162 · see citing works
Failure to filter: Anxious individuals show inefficient gating of threat from working memory
Stout, D. M., Shackman, A. J., & Larson, C. L. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00058 · Cited by 133 · see citing works
Mirror neuron system involvement in empathy: A critical look at the evidence
Baird, A. D., Scheffer, I. E., & Wilson, S. J. (2011) · Social Neuroscience · doi:10.1080/17470919.2010.547085 · Cited by 137 · see citing works
Vive les differences! Individual variation in neural mechanisms of executive control
Braver, T. S., Cole, M. W., & Yarkoni, T. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.03.002 · Cited by 130 · see citing works
Self-citationGreening, S. G., Osuch, E. A., Williamson, P. C., & Mitchell, D. G. V. (2014) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nst027 · Cited by 99 · see citing works
Using fMRI to study reward processing in humans: past, present, and future
Wang, K. S., Smith, D. V., & Delgado, M. R. (2016) · Journal of Neurophysiology · doi:10.1152/jn.00333.2015 · Cited by 105 · see citing works
Progress toward openness, transparency, and reproducibility in cognitive neuroscience
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13325 · Cited by 95 · see citing works
Self-citationOxytocin receptor gene and racial ingroup Bias in empathy-related brain activity
Luo, S., Li, B., Ma, Y., Zhang, W., Rao, Y., & Han, S. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.01.042 · Cited by 85 · see citing works
Kober, H., Brewer, J. A., Height, K. L., & Sinha, R. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.09.042 · Cited by 75 · see citing works
Barch, D. M., & Yarkoni, T. (2013) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-013-0201-7 · Cited by 67 · see citing works
Self-citationIntrinsic functional connectivity is organized as three interdependent gradients
Zhang, J., Abiose, O., Katsumi, Y., Touroutoglou, A., Dickerson, B. C., & Barrett, L. F. (2019) · Scientific Reports · doi:10.1038/s41598-019-51793-7 · Cited by 47 · see citing works
Puberty and functional brain development in humans: convergence in findings?
Dai, J., & Scherf, K. S. (2019) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2019.100690 · Cited by 51 · see citing works
Neural basis of negativity bias in the perception of ambiguous facial expression
Ito, T., Yokokawa, K., Yahata, N., Isato, A., Suhara, T., & Yamada, M. (2017) · Scientific Reports · doi:10.1038/s41598-017-00502-3 · Cited by 45 · see citing works
Cremers, H. R., Veer, I. M., Spinhoven, P., Rombouts, S. A. R. B., Yarkoni, T., Wager, T. D., & Roelofs, K. (2015) · Psychological Medicine · doi:10.1017/s0033291714002657 · Cited by 38 · see citing works
Self-citationShame on the brain: Neural correlates of moral injury event recall in posttraumatic stress disorder
Lloyd, C. S., Nicholson, A. A., Densmore, M., Théberge, J., Neufeld, R. W. J., Jetly, R., McKinnon, M. C., & Lanius, R. A. (2021) · Depression and Anxiety · doi:10.1002/da.23128 · Cited by 35 · see citing works
Neurocognitive Phenotypes and Genetic Dissection of Disorders of Brain and Behavior
Congdon, E., Poldrack, R. A., & Freimer, N. B. (2010) · Neuron · doi:10.1016/j.neuron.2010.10.007 · Cited by 22 · see citing works
The Cognitive Neuroscience of Psychological Treatment Action in Depression and Anxiety
Young, K. S., & Craske, M. G. (2018) · Current Behavioral Neuroscience Reports · doi:10.1007/s40473-018-0137-x · Cited by 13 · see citing works
Common and distinct neural representations of imagined and perceived speech
Lu, L., Han, M., Zou, G., Zheng, L., & Gao, J.-H. (2023) · Cerebral Cortex · doi:10.1093/cercor/bhac519 · Cited by 11 · see citing works
- Authors
- Yarkoni, T.
- Affiliations
- Washington University in St. Louis (US)
- Corresponding author
- Yarkoni, Tal
- ORCID
- Yarkoni, Tal
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2009-05
- Volume
- 4
- Issue
- 3
- Pages
- 294-298
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2036231449
- Cited by
- 549
- References
- 12
- Citations by year
- 2026: 15 · 2025: 27 · 2024: 17 · 2023: 42 · 2022: 36 · 2021: 35 · 2020: 85 · 2019: 60 · 2018: 43 · 2017: 35 · 2016: 27 · 2015: 27 · 2014: 17 · 2013: 25 · 2012: 21
- FWCI
- 9.25
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- functional MRI, statistical power, small sample size, whole-brain analysis, replication crisis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
