Analysis of Multimodal Neuroimaging Data
The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
Each method for imaging brain activity has technical or physiological limits. Thus, combinations of neuroimaging modalities that can alleviate these limitations such as simultaneous recordings of neurophysiological and hemodynamic activity have become increasingly popular. Multimodal imaging setups can take advantage of complementary views on neural activity and enhance our understanding about how neural information processing is reflected in each modality. However, dedicated analysis methods are needed to exploit the potential of multimodal methods. Many solutions to this data integration problem have been proposed, which often renders both comparisons of results and the choice of the right method for the data at hand difficult. In this review we will discuss different multimodal neuroimaging setups, the advances achieved in basic research and clinical application and the methods used. We will provide a comprehensive overview of mathematical tools reoccurring in multimodal neuroimaging studies for artifact removal, data-driven and model-driven analyses, enabling the practitioner to try established or new combinations from these algorithmic building blocks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2005) · doi:10.1002/047174882x · Cited by 13,791
Advances in functional and structural MR image analysis and implementation as FSL
Smith, S. M., Jenkinson, M., Woolrich, M. W., Beckmann, C. F., Behrens, T. E. J., Johansen-Berg, H., Bannister, P. R., De Luca, M., Drobnjak, I., Flitney, D. E., et al. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.07.051 · Cited by 13,630 · cited 1×
“Although the functional relevance of these networks is currently not fully understood there is evidence to suggest that they represent a fundamental and functionally relevant modular organization [258] and that their integrity is compromised in a variety of neurologic and psychiatric conditions [102], [105].”
BackgroundGaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
An Information-Maximization Approach to Blind Separation and Blind Deconvolution
Bell, A. J., & Sejnowski, T. J. (1995) · Neural Computation · doi:10.1162/neco.1995.7.6.1129 · Cited by 8,986
Neurophysiological investigation of the basis of the fMRI signal
Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., & Oeltermann, A. (2001) · Nature · doi:10.1038/35084005 · Cited by 6,551 · cited 8×
“ings using univariate correlation coefficients [99], [172].”
BackgroundMethodResultInfluentialFunctional connectivity in the resting brain: a network analysis of the default mode hypothesis
Greicius, M. D., Krasnow, B., Reiss, A. L., & Menon, V. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0135058100 · Cited by 6,358 · cited 1×
“Although the functional relevance of these networks is currently not fully understood there is evidence to suggest that they represent a fundamental and functionally relevant modular organization [258] and that their integrity is compromised in a variety of neurologic and psychiatric conditions [102], [105].”
BackgroundMR diffusion tensor spectroscopy and imaging
Basser, P. J., Mattiello, J., & LeBihan, D. (1994) · Biophysical Journal · doi:10.1016/s0006-3495(94)80775-1 · Cited by 5,970
Correspondence of the brain's functional architecture during activation and rest
Smith, S. M., Fox, P. T., Miller, K. L., Glahn, D. C., Fox, P. M., Mackay, C. E., Filippini, N., Watkins, K. E., Toro, R., Laird, A. R., et al. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0905267106 · Cited by 5,150 · cited 4×
“In the context of multimodal data analysis event-locked averaging has been used for feature extraction (of evoked-/event-related potentials) to characterize the spatiotemporal dynamics of the hemodynamic response to sensory activation [66], [172], [178], [257] and for artifact removal [4], [100] to name just a few exa…”
BackgroundMethodInfluentialThe basis of anisotropic water diffusion in the nervous system—A technical review.
Beaulieu, C. (2002) · NMR in Biomedicine · doi:10.1002/nbm.782 · Cited by 4,628
Friston, K. J., Harrison, L., & Penny, W. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00202-7 · Cited by 4,583
Jöbsis, F. F. (1977) · Science · doi:10.1126/science.929199 · Cited by 3,899
What we can do and what we cannot do with fMRI
Logothetis, N. K. (2008) · Nature · doi:10.1038/nature06976 · Cited by 3,612 · cited 2×
“clear [169]; hence, the investigation of neurovascular coupling”
BackgroundDiffusion tensor imaging: Concepts and applications
Le Bihan, D., Mangin, J., Poupon, C., Clark, C. A., Pappata, S., Molko, N., & Chabriat, H. (2001) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.1076 · Cited by 3,547 · cited 1×
“diffusion-tensor imaging (DTI) [17], [156].”
An Energy Budget for Signaling in the Grey Matter of the Brain
Attwell, D., & Laughlin, S. B. (2001) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1097/00004647-200110000-00001 · Cited by 3,543 · cited 1×
“The LFP is hypothesized to reflect subthreshold membrane oscillations (the input to or state of a neuron before it is sending out spikes) and have shown to be more correlated with metabolic signals of brain activity than spikes [99], [172]; this finding is consistent with the larger number of mitochondria (the “cellul…”
ResultNunez, P. L., & Srinivasan, R. (2006) · doi:10.1093/acprof:oso/9780195050387.001.0001 · Cited by 2,531
Beyond mind-reading: multi-voxel pattern analysis of fMRI data
Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.07.005 · Cited by 2,271 · cited 1×
“” A robust method for estimating the direction of information flow in noisy multivariate time series is the phase slope index [183], [208], [209].”
MethodNeurovascular regulation in the normal brain and in Alzheimer's disease
Iadecola, C. (2004) · Nature Reviews Neuroscience · doi:10.1038/nrn1387 · Cited by 2,213
Dale, A. M., & Sereno, M. I. (1993) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1993.5.2.162 · Cited by 2,003 · cited 1×
“2) Asymmetric fMRI Biased EEG Analyses: An approach that has proven successful for multimodal data integration is fMRI-constrained current density imaging [59], [164] which has already been proposed in [58].”
BackgroundElectrophysiological signatures of resting state networks in the human brain
Mantini, D., Perrucci, M. G., Del Gratta, C., Romani, G. L., & Corbetta, M. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0700668104 · Cited by 1,938 · cited 1×
“There has been increased interest in this topic as readily available noninvasive multimodal setups enabled researchers to easily study the neurophysiological side of these hemodynamic phenomena [63], [149], [153], [176], [236].”
BackgroundDecoding the visual and subjective contents of the human brain
Kamitani, Y., & Tong, F. (2005) · Nature Neuroscience · doi:10.1038/nn1444 · Cited by 1,829 · cited 1×
“There is extensive literature in the context of fMRI data analysis on decoding mental states using MPA [120], [138], [210] and many software packages are available, such as PyMVPA,5 [115] the Princeton Multi-Voxel Pattern Analysis toolbox6 or SPM plu-”
MethodDynamic Statistical Parametric Mapping
Dale, A. M., Liu, A. K., Fischl, B. R., Buckner, R. L., Belliveau, J. W., Lewine, J. D., & Halgren, E. (2000) · Neuron · doi:10.1016/s0896-6273(00)81138-1 · Cited by 1,787
Dynamics of blood flow and oxygenation changes during brain activation: the Balloon model
Buxton, R. B., Wong, E. C., & Frank, L. R. (1998) · Magnetic Resonance in Medicine · doi:10.1002/mrm.1910390602 · Cited by 1,665
A Method for Removing Imaging Artifact from Continuous EEG Recorded during Functional MRI
Allen, P. J., Josephs, O., & Turner, R. (2000) · NeuroImage · doi:10.1006/nimg.2000.0599 · Cited by 1,301 · cited 1×
“In the context of multimodal data analysis event-locked averaging has been used for feature extraction (of evoked-/event-related potentials) to characterize the spatiotemporal dynamics of the hemodynamic response to sensory activation [66], [172], [178], [257] and for artifact removal [4], [100] to name just a few exa…”
MethodCytochrome oxidase: an endogenous metabolic marker for neuronal activity
Wong-Riley, M. T. T. (1989) · Trends in Neurosciences · doi:10.1016/0166-2236(89)90165-3 · Cited by 1,238
Single-trial analysis and classification of ERP components — A tutorial
Blankertz, B., Lemm, S., Treder, M., Haufe, S., & Müller, K.-R. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.06.048 · Cited by 1,138
Self-citationBOLD correlates of EEG topography reveal rapid resting-state network dynamics
Britz, J., Van De Ville, D., & Michel, C. M. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.02.052 · Cited by 1,070
A new method of the description of the information flow in the brain structures
Kaminski, M. J., & Blinowska, K. J. (1991) · Biological Cybernetics · doi:10.1007/bf00198091 · Cited by 1,047 · cited 1×
“The direction of information flow between dipole time courses is then analyzed using the directed transfer function [137].”
MethodLaufs, H., Krakow, K., Sterzer, P., Eger, E., Beyerle, A., Salek-Haddadi, A., & Kleinschmidt, A. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1831638100 · Cited by 1,028 · cited 4×
“Other oscillations that have been studied more recently are frontal theta, beta and gamma [153], [249].”
BackgroundInfluentialRealistic conductivity geometry model of the human head for interpretation of neuromagnetic data
Hamalainen, M. S., & Sarvas, J. (1989) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.16463 · Cited by 989
The neural basis of functional brain imaging signals
Attwell, D., & Iadecola, C. (2002) · Trends in Neurosciences · doi:10.1016/s0166-2236(02)02264-6 · Cited by 957
Magnetoencephalography: Evidence of Magnetic Fields Produced by Alpha-Rhythm Currents
Cohen, D. (1968) · Science · doi:10.1126/science.161.3843.784 · Cited by 750
EEG microstate sequences in healthy humans at rest reveal scale-free dynamics.
Van De Ville, D., Britz, J., & Michel, C. M. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1007841107 · Cited by 606
Neurophysiology of the BOLD fMRI Signal in Awake Monkeys
Goense, J. B. M., & Logothetis, N. K. (2008) · Current Biology · doi:10.1016/j.cub.2008.03.054 · Cited by 583 · cited 5×
“ings using univariate correlation coefficients [99], [172].”
BackgroundResultInfluentialModalities, Modes, and Models in Functional Neuroimaging
Friston, K. J. (2009) · Science · doi:10.1126/science.1174521 · Cited by 489
Anticipatory haemodynamic signals in sensory cortex not predicted by local neuronal activity
Sirotin, Y. B., & Das, A. (2009) · Nature · doi:10.1038/nature07664 · Cited by 437 · cited 1×
“The point spread function is on the order of 1–2 mm [255], [257].”
BackgroundNeuronal Mechanisms of Cortical Alpha Oscillations in Awake-Behaving Macaques
Bollimunta, A., Chen, Y., Schroeder, C. E., & Ding, M. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.2699-08.2008 · Cited by 434 · cited 1×
“, [31], [269], and [283]), electrophysiological measurements (such as [32], [75], and [139]) and in particular neurovas-”
BackgroundBabiloni, F., Cincotti, F., Babiloni, C., Carducci, F., Mattia, D., Astolfi, L., Basilisco, A., Rossini, P. M., Ding, L., Ni, Y., et al. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.09.036 · Cited by 412
Frontal theta EEG activity correlates negatively with the default mode network in resting state
Scheeringa, R., Bastiaansen, M. C. M., Petersson, K. M., Oostenveld, R., Norris, D. G., & Hagoort, P. (2008) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2007.05.017 · Cited by 400 · cited 1×
“and [248] (for calcium imaging) and [212] and [220] (for VSD”
BackgroundMusso, F., Brinkmeyer, J., Mobascher, A., Warbrick, T., & Winterer, G. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.01.093 · Cited by 387
Devor, A., Dunn, A. K., Andermann, M. L., Ulbert, I., Boas, D. A., & Dale, A. M. (2003) · Neuron · doi:10.1016/s0896-6273(03)00403-3 · Cited by 387
The Berlin brain-computer Interface: accurate performance from first-session in BCI-naive subjects
Blankertz, B., Losch, F., Krauledat, M., Dornhege, G., Curio, G., & Muller, K.-R. (2008) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2008.923152 · Cited by 338
Eichele, T., Specht, K., Moosmann, M., Jongsma, M. L. A., Quiroga, R. Q., Nordby, H., & Hugdahl, K. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0505508102 · Cited by 308
Self-citationSpatio-temporal point-spread function of fMRI signal in human gray matter at 7 Tesla
Shmuel, A., Yacoub, E., Chaimow, D., Logothetis, N. K., & Ugurbil, K. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2006.12.030 · Cited by 294 · cited 1×
“In [254] the authors compared fMRI constrained dipole estimates of unimodal EEG and MEG measurements with a multimodal MEG-EEG estimate.”
BackgroundRitter, P., & Villringer, A. (2006) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2006.06.008 · Cited by 294
Model Studies of the Magnetocardiogram
Grynszpan, F., & Geselowitz, D. B. (1973) · Biophysical Journal · doi:10.1016/s0006-3495(73)86034-5 · Cited by 226 · cited 1×
“The magnetic fields measured with MEG are not affected by volume conduction and can resolve finer structures than EEG [55], [108].”
BackgroundEndogenous brain oscillations and related networks detected by surface EEG‐combined fMRI
Laufs, H. (2008) · Human Brain Mapping · doi:10.1002/hbm.20600 · Cited by 194 · cited 1×
“There has been increased interest in this topic as readily available noninvasive multimodal setups enabled researchers to easily study the neurophysiological side of these hemodynamic phenomena [63], [149], [153], [176], [236].”
BackgroundQuantification of the benefit from integrating MEG and EEG data in minimum ℓ2-norm estimation
Molins, A., Stufflebeam, S. M., Brown, E. N., & Hämäläinen, M. S. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.05.064 · Cited by 192 · cited 1×
“For instance, using a somato-sensory paradigm the authors of [187] found a lower spatial dispersion of dipole estimates for combined MEG-EEG signals compared to MEG alone.”
MethodMultimodal Functional Neuroimaging: Integrating Functional MRI and EEG/MEG
He, B., & Liu, Z. (2008) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2008.2008233 · Cited by 171 · cited 1×
“As we attempt to span a broad field, we will need to be less detailed than more specialized reviews as, for instance for optical imaging data analysis [68], [155], for EEG data analysis [29], [211] , or for special cases of multimodal analyses such as for EEG-fMRI data [121], [195], [245], [272], [273], [280].”
MethodCalhoun, V. D., Adali, T., Stevens, M. C., Kiehl, K. A., & Pekar, J. J. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.12.012 · Cited by 159 · cited 1×
“, [74] or biasing the ICA solution towards an experimentally controlled variable [41], [161].”
BackgroundRauch, A., Rainer, G., & Logothetis, N. K. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0800312105 · Cited by 146
Spatiotemporal precision and hemodynamic mechanism of optical point spreads in alert primates
Sirotin, Y. B., Hillman, E. M. C., Bordier, C., & Das, A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0905509106 · Cited by 121
Estimating the transfer function from neuronal activity to BOLD using simultaneous EEG-fMRI
Rosa, M. J., Kilner, J., Blankenburg, F., Josephs, O., & Penny, W. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.09.011 · Cited by 105
fMRI–EEG integrated cortical source imaging by use of time-variant spatial constraints
Liu, Z., & He, B. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.10.003 · Cited by 103 · cited 2×
“analysis [59], [164], [165] is assumed to be a diagonal ma-”
BackgroundLinear and nonlinear relationships between visual stimuli, EEG and BOLD fMRI signals
Liu, Z., Rios, C., Zhang, N., Yang, L., Chen, W., & He, B. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.01.017 · Cited by 88
Cited by
On the interpretation of weight vectors of linear models in multivariate neuroimaging
Haufe, S., Meinecke, F., Görgen, K., Dähne, S., Haynes, J.-D., Blankertz, B., & Bießmann, F. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.10.067 · Cited by 1,389
Self-citationMultimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296
“EEG and MEG data are a set of timeseries signals reflecting voltage or neuromagnetic field changes recorded at each of the (usually a few dozen of) electrodes attached to the scalp (EEG) or fixed within an MEG scanner helmet.”
BackgroundMethodInfluentialReview of the BCI Competition IV
Tangermann, M., Müller, K.-R., Aertsen, A., Birbaumer, N., Braun, C., Brunner, C., Leeb, R., Mehring, C., Miller, K. J., Müller-Putz, G. R., et al. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00055 · Cited by 1,288
“The challenge in processing signals from multiple sources is to represent, combine and converge information in a way, that is independent of different sampling rates (Bießmann et al., 2009; Biessmann et al., 2011), SNR-levels or varying levels of nonstationarity.”
BackgroundSelf-citationA survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821
“It can provide richer information than a single modality by leveraging modality-specific information (Biessmann, Plis, Meinecke, Eichele, & Muller, 2011; Wagner, Andre, Lingenfelser, & Kim, 2011).”
BackgroundEnhanced performance by a hybrid NIRS–EEG brain computer interface
Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K.-R., & Blankertz, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.07.084 · Cited by 599
Metadata
- Authors
- Biessmann, F., Plis, S., Meinecke, F. C., Eichele, T., & Muller, K.-R.
- Venue
- IEEE Reviews in Biomedical Engineering
- Published
- 2011
- Volume
- 4
- Pages
- 26-58
- ISSN
- 1937-3333 (ISSN-L)
- 1941-1189
- OpenAlex
- W2068356916
- Cited by
- 155
- References
- 55
- FWCI
- 5.88
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.