Reporting details of neuroimaging studies on individual traits prediction: A literature survey
Open accessAbstract
Using machine-learning tools to predict individual phenotypes from neuroimaging data is one of the most promising and hence dynamic fields in systems neuroscience. Here, we perform a literature survey of the rapidly work on phenotype prediction in healthy subjects or general population to sketch out the current state and ongoing developments in terms of data, analysis methods and reporting. Excluding papers on age-prediction and clinical applications, which form a distinct literature, we identified a total 108 papers published since 2007. In these, memory, fluid intelligence and attention were most common phenotypes to be predicted, which resonates with the observation that roughly a quarter of the papers used data from the Human Connectome Project, even though another half recruited their own cohort. Sample size (in terms of training and external test sets) and prediction accuracy (from internal and external validation respectively) did not show significant temporal trends. Prediction accuracy was negatively correlated with sample size of the training set, but not the external test set. While known to be optimistic, leave-one-out cross-validation (LOO CV) was the prevalent strategy for model validation (n = 48). Meanwhile, 27 studies used external validation with external test set. Both numbers showed no significant temporal trends. The most popular learning algorithm was connectome-based predictive modeling introduced by the Yale team. Other common learning algorithms were linear regression, relevance vector regression (RVR), support vector regression (SVR), least absolute shrinkage and selection operator (LASSO), and elastic net. Meanwhile, the amount of data from self-recruiting studies (but not studies using open, shared dataset) was positively correlated with internal validation prediction accuracy. At the same time, self-recruiting studies also reported a significantly higher internal validation prediction accuracy than those using open, shared datasets. Data type and participant age did not significantly influence prediction accuracy. Confound control also did not influence prediction accuracy after adjusted for other factors. To conclude, most of the current literature is probably quite optimistic with internal validation using LOO CV. More efforts should be made to encourage the use of external validation with external test sets to further improve generalizability of the models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The types of models used in this study are the most common in functional connectivity brain-phenotype studies.
“In addition, we used the types of models most common in functional connectivity brain-phenotype studies51.”
From Data leakage inflates prediction performance in connectome-based machine learning models (2024) · cites this paper as [51]
Pearson's r is the most common performance metric in neuroimaging trait prediction studies.
“we focused primarily on r and secondarily on q2 because r is the most common performance metric in neuroimaging trait prediction studies51.”
From Data leakage inflates prediction performance in connectome-based machine learning models (2024) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., Mazoyer, B., & Joliot, M. (2002) · NeuroImage · doi:10.1006/nimg.2001.0978 · Cited by 16,319 · see citing works
Desikan, R. S., Ségonne, F., Fischl, B., Quinn, B. T., Dickerson, B. C., Blacker, D., Buckner, R. L., Dale, A. M., Maguire, R. P., Hyman, B. T., et al. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.021 · Cited by 13,324 · see citing works
The organization of the human cerebral cortex estimated by intrinsic functional connectivity
Thomas Yeo, B. T., Krienen, F. M., Sepulcre, J., Sabuncu, M. R., Lashkari, D., Hollinshead, M., Roffman, J. L., Smoller, J. W., Zöllei, L., Polimeni, J. R., et al. (2011) · Journal of Neurophysiology · doi:10.1152/jn.00338.2011 · Cited by 9,182 · see citing works
Fischl, B., Salat, D. H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., van der Kouwe, A., Killiany, R., Kennedy, D., Klaveness, S., et al. (2002) · Neuron · doi:10.1016/s0896-6273(02)00569-x · Cited by 8,610 · see citing works
The organization of the human cerebellum estimated by intrinsic functional connectivity
Buckner, R. L., Krienen, F. M., Castellanos, A., Diaz, J. C., & Yeo, B. T. T. (2011) · Journal of Neurophysiology · doi:10.1152/jn.00339.2011 · Cited by 7,304 · see citing works
A multi-modal parcellation of human cerebral cortex
Glasser, M. F., Coalson, T. S., Robinson, E. C., Hacker, C. D., Harwell, J., Yacoub, E., Ugurbil, K., Andersson, J., Beckmann, C. F., Jenkinson, M., et al. (2016) · Nature · doi:10.1038/nature18933 · Cited by 5,111 · see citing works
Functional Network Organization of the Human Brain
Power, J. D., Cohen, A. L., Nelson, S. M., Wig, G. S., Barnes, K. A., Church, J. A., Vogel, A. C., Laumann, T. O., Miezin, F. M., Schlaggar, B. L., et al. (2011) · Neuron · doi:10.1016/j.neuron.2011.09.006 · Cited by 4,124 · see citing works
Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI
Schaefer, A., Kong, R., Gordon, E. M., Laumann, T. O., Zuo, X.-N., Holmes, A. J., Eickhoff, S. B., & Yeo, B. T. T. (2018) · Cerebral Cortex · doi:10.1093/cercor/bhx179 · Cited by 3,657 · see citing works
Self-citationAutomatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature
Destrieux, C., Fischl, B., Dale, A., & Halgren, E. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.06.010 · Cited by 2,956 · see citing works
The Human Brainnetome Atlas: A New Brain Atlas Based on Connectional Architecture
Fan, L., Li, H., Zhuo, J., Zhang, Y., Wang, J., Chen, L., Yang, Z., Chu, C., Xie, S., Laird, A. R., et al. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhw157 · Cited by 2,956 · see citing works
Self-citationFunctional connectome fingerprinting: Identifying individuals using patterns of brain connectivity
Finn, E. S., Shen, X., Scheinost, D., Rosenberg, M. D., Huang, J., Chun, M. M., Papademetris, X., & Constable, R. T. (2015) · Nature Neuroscience · doi:10.1038/nn.4135 · Cited by 2,747 · see citing works
Prediction of individual brain maturity using fMRI
Dosenbach, N. U. F., Nardos, B., Cohen, A. L., Fair, D. A., Power, J. D., Church, J. A., Nelson, S. M., Wig, G. S., Vogel, A. C., Lessov-Schlaggar, C. N., et al. (2010) · Science · doi:10.1126/science.1194144 · Cited by 2,183 · see citing works
Generation and evaluation of a cortical area parcellation from resting-state correlations
Gordon, E. M., Laumann, T. O., Adeyemo, B., Huckins, J. F., Kelley, W. M., & Petersen, S. E. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhu239 · Cited by 1,463 · see citing works
Using connectome-based predictive modeling to predict individual behavior from brain connectivity
Shen, X.-L., Finn, E., Scheinost, D., Rosenberg, M., Chun, M. M., Papademetris, X., & Constable, R. T. (2017) · Nature Protocols · doi:10.1038/nprot.2016.178 · Cited by 1,088 · see citing works
Building better biomarkers: brain models in translational neuroimaging
Woo, C.-W., Chang, L. J., Lindquist, M. A., & Wager, T. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4478 · Cited by 1,056 · see citing works
A neuromarker of sustained attention from whole-brain functional connectivity
Rosenberg, M. D., Finn, E. S., Scheinost, D., Papademetris, X., Shen, X., Constable, R. T., & Chun, M. M. (2015) · Nature Neuroscience · doi:10.1038/nn.4179 · Cited by 1,004 · see citing works
Groupwise whole-brain parcellation from resting-state fMRI data for network node identification
Shen, X., Tokoglu, F., Papademetris, X., & Constable, R. T. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.05.081 · Cited by 1,000 · see citing works
A spatially unbiased atlas template of the human cerebellum
Diedrichsen, J. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.05.056 · Cited by 989 · see citing works
Cross-validation failure: Small sample sizes lead to large error bars
Varoquaux, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.061 · Cited by 897 · see citing works
Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines
Varoquaux, G. P., Raamana, P. R., Engemann, D. A., Hoyos-Idrobo, A., Schwartz, Y., & Thirion, B. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.10.038 · Cited by 801 · see citing works
Establishment of Best Practices for Evidence for Prediction: A Review
Poldrack, R. A., Huckins, G., & Varoquaux, G. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.3671 · Cited by 738 · see citing works
Resting-state fMRI confounds and cleanup
Murphy, K., Birn, R. M., & Bandettini, P. A. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.04.001 · Cited by 666 · see citing works
The IMAGEN study: reinforcement-related behaviour in normal brain function and psychopathology
Schumann, G., Loth, E., Banaschewski, T., Barbot, A., Barker, G. J., Büchel, C., Conrod, P., Dalley, J. W., Flor, H., Gallinat, J., et al. (2010) · Molecular Psychiatry · doi:10.1038/mp.2010.4 · Cited by 618 · see citing works
Prediction as a Humanitarian and Pragmatic Contribution from Human Cognitive Neuroscience
Gabrieli, J. D. E., Ghosh, S., & Whitfield, S. (2015) · Neuron · doi:10.1016/j.neuron.2014.10.047 · Cited by 526 · see citing works
Longitudinal Development of Cortical and Subcortical Gray Matter from Birth to 2 Years
Gilmore, J. H., Shi, F., Woolson, S. L., Knickmeyer, R. C., Short, S. J., Lin, W., Zhu, H., Hamer, R. M., Styner, M., & Shen, D. (2012) · Cerebral Cortex · doi:10.1093/cercor/bhr327 · Cited by 456 · see citing works
On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments
Carp, J. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00149 · Cited by 381 · see citing works
Ten simple rules for predictive modeling of individual differences in neuroimaging
Scheinost, D., Noble, S., Horien, C., Greene, A. S., Lake, E. M., Salehi, M., Gao, S., Shen, X., O'Connor, D., Barron, D. S., et al. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.02.057 · Cited by 381 · see citing works
Sui, J., Jiang, R., Bustillo, J., & Calhoun, V. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.02.016 · Cited by 309 · see citing works
He, T., Kong, R., Holmes, A. J., Nguyen, M., Sabuncu, M. R., Eickhoff, S. B., Bzdok, D., Feng, J., & Yeo, B. T. T. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2019.116276 · Cited by 271 · see citing works
Self-citationWhen Optimism Hurts: Inflated Predictions in Psychiatric Neuroimaging
Whelan, R., & Garavan, H. (2014) · Biological Psychiatry · doi:10.1016/j.biopsych.2013.05.014 · Cited by 209 · see citing works
Exploration, Inference, and Prediction in Neuroscience and Biomedicine
Bzdok, D., & Ioannidis, J. P. A. (2019) · Trends in Neurosciences · doi:10.1016/j.tins.2019.02.001 · Cited by 204 · see citing works
Longitudinal test-retest neuroimaging data from healthy young adults in southwest China
Liu, W., Wei, D., Chen, Q., Yang, W., Meng, J., Wu, G., Bi, T., Zhang, Q., Zuo, X., & Qiu, J. (2017) · Scientific Data · doi:10.1038/sdata.2017.17 · Cited by 155 · see citing works
Combining Multiple Connectomes Improves Predictive Modeling of Phenotypic Measures
Gao, S., Greene, A., Constable, R. T., & Scheinost, D. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116038 · Cited by 149 · see citing works
Predictive modelling using neuroimaging data in the presence of confounds
Rao, A. W., Monteiro, J. M., & Mourão-Miranda, J. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.01.066 · Cited by 114 · see citing works
Yip, S. W., Kiluk, B. D., & Scheinost, D. (2019) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2019.11.001 · Cited by 77 · see citing works
A hitchhiker’s guide to working with large, open-source neuroimaging datasets
Horien, C. L., Noble, S., Greene, A. S., Lee, K., Barron, D. S., Gao, S., O’Connor, D. A., Salehi, M., Dadashkarimi, J., Shen, X., et al. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-01005-4 · Cited by 67 · see citing works
Age-specific gray and white matter DTI atlas for human brain at 33, 36 and 39 postmenstrual weeks
Feng, L., Li, H., Oishi, K., Mishra, V. R., Song, L., Peng, Q., Ouyang, M., Wang, J., Slinger, M. A., Jeon, T., et al. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.06.069 · Cited by 58 · see citing works
Yamashita, M., Kawato, M., & Imamizu, H. (2015) · Scientific Reports · doi:10.1038/srep07622 · Cited by 53 · see citing works
Toward Robust Anxiety Biomarkers: A Machine Learning Approach in a Large-Scale Sample
Boeke, E. A., Holmes, A. J., & Phelps, E. A. (2020) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2019.05.018 · Cited by 50 · see citing works
Scan Once, Analyse Many: Using Large Open-Access Neuroimaging Datasets to Understand the Brain
Madan, C. (2020) · Neuroinformatics · doi:10.1007/s12021-021-09519-6 · Cited by 38 · see citing works
Neuroimaging-based prediction of mental traits: Road to utopia or Orwell?
Eickhoff, S. B., & Langner, R. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000497 · Cited by 35 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Data leakage inflates prediction performance in connectome-based machine learning models
Rosenblatt, M., Tejavibulya, L., Jiang, R., Noble, S., & Scheinost, D. (2024) · Nature Communications · doi:10.1038/s41467-024-46150-w · Cited by 236 · see citing works
Amiri, M., Raimondo, F., Fisher, P. M., Cacic Hribljan, M., Sidaros, A., Othman, M. H., Zibrandtsen, I., Bergdal, O., Fabritius, M. L., Hansen, A. E., et al. (2024) · Neurocritical Care · doi:10.1007/s12028-023-01816-z · Cited by 44 · see citing works
- Authors
- Yeung, A. W. K., More, S., Wu, J., & Eickhoff, S. B.
- Affiliations
- University of Hong Kong (HK)
- Forschungszentrum Jülich (DE)
- Corresponding author
- Yeung, Andy Wai Kan
- Eickhoff, Simon B.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2022-08
- Volume
- 256
- Pages
- 119275
- ISSN
- 1053-8119 (ISSN-L)
- OpenAlex
- W4225274087
- Cited by
- 36
- References
- 42
- Citations by year
- 2026: 7 · 2025: 10 · 2024: 8 · 2023: 9 · 2022: 2
- FWCI
- 3.74
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- NeuroImage (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (R01 MH074457)
- Horizon 2020 Framework Programme (826421, 945539)
- Deutsche Forschungsgemeinschaft (IRTG 2150, SFB 1451)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced Neuroimaging Techniques and Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
- Keywords
- neuroimaging, phenotype prediction, reporting practices, external validation, leave-one-out cross-validation, connectome-based predictive modeling, sample size, prediction accuracy, Human Connectome Project, confounding adjustment, memory, fluid intelligence, attention
- MeSH terms
- Machine Learning, Algorithms, Attention, Humans, Phenotype, Connectome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
