Analyzing Neuroimaging Data Through Recurrent Deep Learning Models
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
The application of deep learning (DL) models to neuroimaging data poses several challenges, due to the high dimensionality, low sample size and complex temporo-spatial dependency structure of these data. Even further, DL models often act as as black boxes, impeding insight into the association of cognitive state and brain activity. To approach these challenges, we introduce the DeepLight framework, which utilizes long short-term memory (LSTM) based DL models to analyze whole-brain functional Magnetic Resonance Imaging (fMRI) data. To decode a cognitive state (e.g., seeing the image of a house), DeepLight separates an fMRI volume into a sequence of axial brain slices, which is then sequentially processed by an LSTM. To maintain interpretability, DeepLight adapts the layer-wise relevance propagation (LRP) technique. Thereby, decomposing its decoding decision into the contributions of the single input voxels to this decision. Importantly, the decomposition is performed on the level of single fMRI volumes, enabling DeepLight to study the associations between cognitive state and brain activity on several levels of data granularity, from the level of the group down to the level of single time points. To demonstrate the versatility of DeepLight, we apply it to a large fMRI dataset of the Human Connectome Project. We show that DeepLight outperforms conventional approaches of uni- and multivariate fMRI analysis in decoding the cognitive states and in identifying the physiologically appropriate brain regions associated with these states. We further demonstrate DeepLight’s ability to study the fine-grained temporo-spatial variability of brain activity over sequences of single fMRI samples.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · cited 8×
“pattern of a cluster of voxels at time point t and yt the corresponding label, the SVM (Cortes and Vapnik, 1995) is defined as follows:”
InfluentialLearning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Fischl, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.021 · Cited by 9,526
The WU-Minn Human Connectome Project: An overview
Van Essen, D. C., Smith, S. M., Barch, D. M., Behrens, T. E. J., Yacoub, E., & Ugurbil, K. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.05.041 · Cited by 5,651
The minimal preprocessing pipelines for the Human Connectome Project
Glasser, M. F., Sotiropoulos, S. N., Wilson, J. A., Coalson, T. S., Fischl, B., Andersson, J. L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J. R., et al. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.04.127 · Cited by 5,253
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Self-citationOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Distributed and overlapping representations of faces and objects in ventral temporal cortex
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001) · Science · doi:10.1126/science.1063736 · Cited by 3,940
Large-scale automated synthesis of human functional neuroimaging data.
Yarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C., & Wager, T. D. (2011) · Nature Methods · doi:10.1038/nmeth.1635 · Cited by 3,686
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · cited 1×
“We refer the reader to [Samek et al., 2018, Montavon et al., 2018] for more information about explanation methods.”
Self-citationA Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation
Goutte, C., & Gaussier, E. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-31865-1_25 · Cited by 2,285
An fMRI-Based Neurologic Signature of Physical Pain
Wager, T. D., Atlas, L. Y., Lindquist, M. A., Roy, M., Woo, C.-W., & Kross, E. (2013) · New England Journal of Medicine · doi:10.1056/nejmoa1204471 · Cited by 1,597 · cited 4×
“In particular, the General Linear Model (GLM) [Friston et al., 1994], searchlight analysis [Kriegeskorte et al., 2006] and whole-brain Least Absolute Shrinkage Logistic Regression (whole-brain Lasso) [Wager et al., 2013, Grosenick et al., 2013].”
InfluentialLong-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547
Function in the human connectome: task-fMRI and individual differences in behavior
Barch, D. M., Burgess, G. C., Harms, M. P., Petersen, S. E., Schlaggar, B. L., Corbetta, M., Glasser, M. F., Curtiss, S., Dixit, S., Feldt, C., et al. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.05.033 · Cited by 1,529
The Enigmatic temporal pole: a review of findings on social and emotional processing
Olson, I. R., Plotzker, A., & Ezzyat, Y. (2007) · Brain · doi:10.1093/brain/awm052 · Cited by 1,370
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · cited 2×
“Layer-Wise Relevance Propagation in the DeepLight Framework To relate the decoded cognitive state and brain activity, DeepLight utilizes the Layer-Wise Relevance Propagation (LRP; Bach et al., 2015; Montavon et al., 2017; Lapuschkin et al., 2019) method.”
Self-citationSamek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080
Self-citationUnmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K.-R. (2019) · Nature Communications · doi:10.1038/s41467-019-08987-4 · Cited by 1,047
Self-citationHierarchical Feature Representation and Multimodal Fusion with Deep Learning for AD/MCI Diagnosis
Suk, H.-I., Lee, S.-W., & Shen, D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.077 · Cited by 968 · cited 1×
“…the application of DL models to neuroimaging data (e.g., [Plis et al., 2014, Mensch et al., 2018, Sarraf and Tofighi, 2016, Nie et al., 2016, Suk et al., 2014, Petrov et al., 2018, Yousefnezhad and Zhang, 2018]), two major challenges have so far prevented broad DL usage: (1) Neuroimaging…”
Uğurbil, K., Xu, J., Auerbach, E. J., Moeller, S., Vu, A. T., Duarte-Carvajalino, J. M., Lenglet, C., Wu, X., Schmitter, S., Van de Moortele, P. F., et al. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.05.012 · Cited by 812
The power of simulation: Imagining one's own and other's behavior
Decety, J., & Grèzes, J. (2006) · Brain Research · doi:10.1016/j.brainres.2005.12.115 · Cited by 790 · cited 1×
“…self-produced actions and the actions produced Frontiers in Neuroscience | www.frontiersin.org 12 December 2019 | Volume 13 | Article 1321 by others and is generally regarded of as a central hub for the integration of body-related information (for a detailed review, see Decety and Grèzes, 2006).”
BackgroundA general mechanism for perceptual decision-making in the human brain
Heekeren, H. R., Marrett, S., Bandettini, P. A., & Ungerleider, L. G. (2004) · Nature · doi:10.1038/nature02966 · Cited by 741 · cited 3×
“…gyrus, as well as the right fusiform gyrus for the body state, the fusiform gyrus (also known as the fusiform face area FFA; [Haxby et al., 2001, Heekeren et al., 2004]) and amygdala for the face state, the parahippocampal gyrus (or parahippocampal place area PPA; [Haxby et al., 2001, Heekeren…”
InfluentialSelf-citationHuman orbitofrontal cortex represents a cognitive map of state space
Schuck, N. W., Cai, M. B., Wilson, R. C., & Niv, Y. (2016) · Neuron · doi:10.1016/j.neuron.2016.08.019 · Cited by 602
Explaining Recurrent Neural Network Predictions in Sentiment Analysis
Arras, L., Montavon, G., Müller, K.-R., & Samek, W. (2017) · Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.18653/v1/w17-5221 · Cited by 361 · cited 1×
“For these types of connections, we set the relevances of the gate neuron R ( l − 1) g = 0 and the relevances of the source neuron R j denotes the relevances of the upper layer neuron z j (as proposed in [Arras et al., 2017]).”
Self-citation3D Deep Learning for Multi-modal Imaging-Guided Survival Time Prediction of Brain Tumor Patients
Nie, D., Zhang, H., Adeli, E., Liu, L., & Shen, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_25 · Cited by 282 · cited 1×
“While researchers have started exploring the application of DL models to neuroimaging data (e.g., Plis et al., 2014; Suk et al., 2014; Nie et al., 2016; Sarraf and Tofighi, 2016; Mensch et al., 2018; Petrov et al., 2018; Yousefnezhad and Zhang, 2018), two major challenges have so far prevented broad DL usage: (1) Neur…”
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationA Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477
“Another application of DNN explanation techniques in cognitive neuroscience is presented in [186], where a deep model is learned to predict whole-brain fMRI data.”
BackgroundMethodInfluentialSelf-citationTowards Explainable Artificial Intelligence
Samek, W., & Müller, K.-R. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_1 · Cited by 580
Self-citationSalahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508
Metadata
- Authors
- Thomas, A. W., Heekeren, H. R., Müller, K.-R., & Samek, W.
- Venue
- Frontiers in Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2019-12-10
- Volume
- 13
- Pages
- 1321-1321
- ISSN
- 1662-453X (ISSN-L)
- OpenAlex
- W2948702906
- Cited by
- 90
- References
- 28
- FWCI
- 6.49
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.