Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 461

The DOI matches its Crossref record; 154 of the 154 references it cites were found in a registry or an index.

Multimodal Fusion of Brain Imaging Data: A Key to Finding the Missing Link(s) in Complex Mental Illness

Open access
Calhoun, Vince D., Sui, Jing·2016·Biological Psychiatry: Cognitive Neuroscience and Neuroimaging

Abstract

It is becoming increasingly clear that combining multi-modal brain imaging data is able to provide more information for individual subjects by exploiting the rich multimodal information that exists. However, the number of studies that do true multimodal fusion (i.e. capitalizing on joint information among modalities) is still remarkably small given the known benefits. In part, this is because multi-modal studies require broader expertise in collecting, analyzing, and interpreting the results than do unimodal studies. In this paper, we start by introducing the basic reasons why multimodal data fusion is important and what it can do, and importantly how it can help us avoid wrong conclusions and help compensate for imperfect brain imaging studies. We also discuss the challenges that need to be confronted for such approaches to be more widely applied by the community. We then provide a review of the diverse studies that have used multimodal data fusion (primarily focused on psychosis) as well as provide an introduction to some of the existing analytic approaches. Finally, we discuss some up-and-coming approaches to multi-modal fusion including deep learning and multimodal classification which show considerable promise. Our conclusion is that multimodal data fusion is rapidly growing, but it is still underutilized. The complexity of the human brain coupled with the incomplete measurement provided by existing imaging technology makes multimodal fusion essential in order to mitigate against misdirection and hopefully provide a key to finding the missing link(s) in complex mental illness.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.bpsc.2015.12.005

Cites

154 of 154 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 14

  1. Sui, 2009, A new joint blind source separation model for two datasets and its application to second-level FMRI group analysis, Presented at the Organization for Human Brain Mapping Annual Meeting, June 18--23, San Francisco, California.
  2. Liu, 2008, Extracting principle components for discriminant analysis of fMRI images, Proc ICASSP
  3. Miyata, 2007, Interfrontal commissural abnormality in schizophrenia: Tractography-assisted callosal parcellation, Schizophr Res, 97, 236, 10.1016/j.schres.2007.07.032 · doi:10.1016/j.schres.2007.07.032
  4. Correa, 2010, Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI, Neuroimage, 50, 1438, 10.1016/j.neuroimage.2010.01.062 · DOI
  5. Sui, 2011, Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA+ joint ICA model, Neuroimage, 57, 839, 10.1016/j.neuroimage.2011.05.055 · DOI · Paper page
  6. Plis SM, Hjelm RD, Salakhutdinov RR, Calhoun VD (2014): Deep learning models for brain imaging: Model depth enhances discovery power. Proceedings of the Organization of Human Brain Mapping Annual Meeting, June 8--12, Hamburg, Germany.
  7. Castro E, Hjelm D, Plis S, Dinh L, Turner J, Calhoun VD (2015): Independent component estimation of simulated structural magnetic resonance imaging data using deep learning. Presented at the IEEE Machine Learning for Signal Processing Workshop, August 25--29, Boston, Massachusetts.
  8. Srivastava N, Salakhutdinov RR (2012): Multimodal learning with deep Boltzmann machines. In Bartlett P, Pereira FCN, Burges CJC, Bottou L, Weinberger KQ, editors. Advances in Neural Information Processing Systems 25, 2231–2239.
  9. Amin, 2015, A deep-learning approach to translate between brain structure and brain function, Pattern Recognition in NeuroImaging Conference, June 10--12, Palo Alto, California.
  10. Sui, 2012, A selective review of multimodal fusion methods in schizophrenia, Front Hum Neurosci, 6, 27, 10.3389/fnhum.2012.00027 · DOI
  11. Segall, 2012, Correspondence between structure and function in the human brain at rest, Front Neuroinform, 6, 10, 10.3389/fninf.2012.00010 · DOI · Paper page
  12. Michael, 2010, Does function follow form? Methods to fuse structural and functional brain images show decreased linkage in schizophrenia, Hum Brain Mapp, 49, 2626
  13. Smieskova, 2011, Different duration of at-risk mental state associated with neurofunctional abnormalities. A multimodal imaging study, Hum Brain Mapp, 33, 2281, 10.1002/hbm.21360 · DOI
  14. Xu, 2009, Joint source based morphometry identifies linked gray and white matter group differences, Neuroimage, 44, 777, 10.1016/j.neuroimage.2008.09.051 · DOI

Cited by

See all 461

Metadata

Authors
Calhoun, V. D., & Sui, J.
Venue
Biological Psychiatry: Cognitive Neuroscience and Neuroimaging
Publisher
Elsevier BV
Published
2016-05
Volume
1
Issue
3
Pages
230-244
ISSN
2451-9022 (ISSN-L)
OpenAlex
W2252433728
Cited by
461
References
154
FWCI
22.13
Type
journal article
Language
en
Open access
green
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.