Efficient neural decoding based on multimodal training
Open accessAbstract
BACKGROUND/OBJECTIVES: Neural decoding methods are often limited by the performance of brain encoders, which map complex brain signals into a latent representation space of perception information. These brain encoders are constrained by the limited amount of paired brain and stimuli data available for training, making it challenging to learn rich neural representations. METHODS: To address this limitation, we present a novel multimodal training approach using paired image and functional magnetic resonance imaging (fMRI) data to establish a brain masked autoencoder that learns the interactions between images and brain activities. Subsequently, we employ a diffusion model conditioned on brain data to decode realistic images. RESULTS: Our method achieves high-quality decoding results in semantic contents and low-level visual attributes, outperforming previous methods both qualitatively and quantitatively, while maintaining computational efficiency. Additionally, our method is applied to decode artificial patterns across region of interests (ROIs) to explore their functional properties. We not only validate existing knowledge concerning ROIs but also unveil new insights, such as the synergy between early visual cortex and higher-level scene ROIs, as well as the competition within the higher-level scene ROIs. CONCLUSIONS: These findings provide valuable insights for future directions in the field of neural decoding.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Attention-based fusion aligns key features and dynamically assigns weights between modalities for deeper semantic capture.
“Attention-based fusion [49,50,51] aligns key features and dynamically assigns weights between modalities for deeper semantic capture.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [51]
Section 3.2 describes a mid-fusion image–fMRI model with attention heads, and Section 5 reports improved reconstruction of semantic content. The paper does not say that this fusion aligns key features or dynamically assigns weights between modalities; Section 3.3 describes cross-attention as conditioning the diffusion model on fMRI input instead. Page numbers are not available in our copy. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · see citing works
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 1× · see citing works
“Our contributions are as follows: • We propose a multimodal masked autoencoder (MAE) [19] jointly trained on paired image–fMRI data to better encourage the fMRI encoder to learn information from image modality, enabling effective and efficient encoding of fMRI data.”
MethodA multi-modal parcellation of human cerebral cortex
Glasser, M. F., Coalson, T. S., Robinson, E. C., Hacker, C. D., Harwell, J., Yacoub, E., Ugurbil, K., Andersson, J., Beckmann, C. F., Jenkinson, M., et al. (2016) · Nature · doi:10.1038/nature18933 · Cited by 5,111 · cited 1× · see citing works
“V1 , V2, V3, and V4 ROIs were selected from the Human Connectome Project (HCP) multimodal parcellation [44], resulting in a total of 4183 vertices.”
MethodDistributed and overlapping representations of faces and objects in ventral temporal cortex
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001) · Science · doi:10.1126/science.1063736 · Cited by 3,940 · see citing works
Diffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362 · see citing works
Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature
Destrieux, C., Fischl, B., Dale, A., & Halgren, E. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.06.010 · Cited by 2,956 · see citing works
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637 · see citing works
Generative Modeling by Estimating Gradients of the Data Distribution
Song, Y., & Ermon, S. (2019) · arXiv · doi:10.48550/arxiv.1907.05600 · Cited by 1,453 · see citing works
The lateral occipital complex and its role in object recognition
Kalanit Grill-spector, A., Kourtzi, Z., & Kanwisher, N. (2001) · Vision Research · doi:10.1016/s0042-6989(01)00073-6 · Cited by 1,410 · see citing works
Vagaries of Visual Perception in Autism
Dakin, S. C., & Frith, U. (2005) · Neuron · doi:10.1016/j.neuron.2005.10.018 · Cited by 763 · see citing works
Parcellations and Hemispheric Asymmetries of Human Cerebral Cortex Analyzed on Surface-Based Atlases
Van Essen, D. C., Glasser, M. F., Dierker, D. L., Harwell, J., & Coalson, T. (2012) · Cerebral Cortex · doi:10.1093/cercor/bhr291 · Cited by 616 · see citing works
A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
Allen, E., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B. A., Pestilli, F., Charest, I., et al. (2021) · Nature Neuroscience · doi:10.1038/s41593-021-00962-x · Cited by 586 · see citing works
Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks
Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01838 · Cited by 561 · see citing works
Generic decoding of seen and imagined objects using hierarchical visual features
Horikawa, T., & Kamitani, Y. (2017) · Nature Communications · doi:10.1038/ncomms15037 · Cited by 514 · see citing works
The impact of atypical sensory processing on social impairments in autism spectrum disorder
Thye, M. D., Bednarz, H. M., Herringshaw, A. J., Sartin, E. B., & Kana, R. K. (2018) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2017.04.010 · Cited by 513 · cited 1× · see citing works
“Furthermore, it holds potential for investigating brain disease states that could impact perception and cognition [2–4].”
BackgroundVisual perception and its impairment in schizophrenia
Butler, P. D., Silverstein, S. M., & Dakin, S. C. (2008) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.03.023 · Cited by 472 · see citing works
Scene Perception in the Human Brain
Epstein, R. A., & Baker, C. I. (2019) · Annual Review of Vision Science · doi:10.1146/annurev-vision-091718-014809 · Cited by 350 · see citing works
VLMo: Unified Vision-Language Pre-Training with Mixture-Of-Modality-Experts
Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., Piao, S., & Wei, F. (2022) · neural information processing systems · doi:10.52202/068431-2384 · Cited by 336 · see citing works
The Occipital Place Area Is Causally and Selectively Involved in Scene Perception
Dilks, D. D., Julian, J. B., Paunov, A. M., & Kanwisher, N. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.4081-12.2013 · Cited by 332 · see citing works
Deep image reconstruction from human brain activity
Shen, G., Horikawa, T., Majima, K., & Kamitani, Y. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006633 · Cited by 319 · see citing works
High-resolution image reconstruction with latent diffusion models from human brain activity
Takagi, Y., & Nishimoto, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01389 · Cited by 314 · see citing works
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-ExpertsPreprint
Bao, H., Wang, W., Li, D., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02358 · Cited by 281 · see citing works
Decoding the neural substrates of reward-related decision making with functional MRI
Hampton, A. N., & O’Doherty, J. P. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0606297104 · Cited by 279 · see citing works
Park, S., Brady, T. F., Greene, M. R., & Oliva, A. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.3885-10.2011 · Cited by 246 · see citing works
Thinking Outside the Box: Rectilinear Shapes Selectively Activate Scene-Selective Cortex
Nasr, S., Echavarria, C. E., & Tootell, R. B. H. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.4802-13.2014 · Cited by 163 · see citing works
End-to-End Deep Image Reconstruction From Human Brain Activity
Shen, G., Dwivedi, K., Majima, K., Horikawa, T., & Kamitani, Y. (2019) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2019.00021 · Cited by 163 · see citing works
BOLD5000, a public fMRI dataset while viewing 5000 visual images
Chang, N., Pyles, J. A., Marcus, A., Gupta, A., Tarr, M. J., & Aminoff, E. (2019) · Scientific Data · doi:10.1038/s41597-019-0052-3 · Cited by 161 · see citing works
Deconstructing Visual Scenes in Cortex: Gradients of Object and Spatial Layout Information
Harel, A., Kravitz, D. J., & Baker, C. I. (2012) · Cerebral Cortex · doi:10.1093/cercor/bhs091 · Cited by 152 · see citing works
Julian, J. B., Ryan, J., Hamilton, R. H., & Epstein, R. A. (2016) · Current Biology · doi:10.1016/j.cub.2016.02.066 · Cited by 152 · see citing works
Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding
Chen, Z., Qing, J., Xiang, T., Yue, W. L., & Zhou, J. H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02175 · Cited by 148 · see citing works
Multivoxel fMRI analysis of color tuning in human primary visual cortex
Parkes, L. M., Marsman, J.-B., Oxley, D., Goulermas, J. Y., & Wuerger, S. M. (2009) · Journal of Vision · doi:10.1167/9.1.1 · Cited by 145 · see citing works
Natural scene reconstruction from fMRI signals using generative latent diffusion
Özçelik, F., & VanRullen, R. (2023) · Scientific Reports · doi:10.1038/s41598-023-42891-8 · Cited by 143 · see citing works
BOLD5000, a public fMRI dataset while viewing 5000 visual images
Chang, N., Pyles, J. A., Marcus, A., et al. (2019) · Figshare · doi:10.1184/r1/8097584.v1 · Cited by 139 · see citing works
Linear reconstruction of perceived images from human brain activity
Schoenmakers, S., Barth, M., Heskes, T., & van Gerven, M. A. J. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.07.043 · Cited by 132 · see citing works
BOLD5000, a public fMRI dataset while viewing 5000 visual images
Chang, N., Pyles, J. A., Marcus, A., Gupta, A., Tarr, M., & Aminoff, E. (2019) · Figshare · doi:10.1184/r1/8097584 · Cited by 132 · see citing works
The occipital place area represents the local elements of scenes
Kamps, F. S., Julian, J. B., Kubilius, J., Kanwisher, N., & Dilks, D. D. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.02.062 · Cited by 122 · see citing works
A Cardinal Orientation Bias in Scene-Selective Visual Cortex
Nasr, S., & Tootell, R. B. H. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.2036-12.2012 · Cited by 114 · see citing works
Gaziv, G., Beliy, R., Granot, N., Hoogi, A., Strappini, F., Golan, T., & Irani, M. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119121 · Cited by 89 · see citing works
Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN
Mozafari, M., Reddy, L., & Rufin Vanrullen, A. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206960 · Cited by 78 · see citing works
Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors
Scotti, P. S., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Cohen, E., Dempster, A. J., Verlinde, N., Yundler, E., Weisberg, D., et al. (2023) · neural information processing systems · doi:10.52202/075280-1073 · Cited by 77 · see citing works
From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRIPreprint
Beliy, R., Gaziv, G., Hoogi, A., Strappini, F., Golan, T., & Irani, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.02431 · Cited by 73 · see citing works
Covert attention allows for continuous control of brain–computer interfaces
Bahramisharif, A., van Gerven, M. A. J., Heskes, T., & Jensen, O. B. (2010) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2010.07174.x · Cited by 68 · see citing works
Özçelik, F., Choksi, B., Mozafari, M., Reddy, L., & VanRullen, R. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892673 · Cited by 64 · see citing works
Visual Size Processing in Early Visual Cortex Follows Lateral Occipital Cortex Involvement
Zeng, H., Fink, G. R., & Weidner, R. (2020) · Journal of Neuroscience · doi:10.1523/jneurosci.2437-19.2020 · Cited by 55 · see citing works
Cortical activation to indoor versus outdoor scenes: an fMRI study
Henderson, J. M., Larson, C. L., & Zhu, D. C. (2006) · Experimental Brain Research · doi:10.1007/s00221-006-0766-2 · Cited by 49 · see citing works
Audiovisual Masked Autoencoders
Georgescu, M.-I., Fonseca, E., Ionescu, R. T., Lučić, M., Schmid, C., & Arnab, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01479 · Cited by 48 · see citing works
Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion PriorsPreprint
Scotti, P. S., Banerjee, A., Goode, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.18274 · Cited by 39 · see citing works
NeuroGen: Activation optimized image synthesis for discovery neuroscience
Gu, Z., Jamison, K. W., Khosla, M., Allen, E., Wu, Y., St-Yves, G., Naselaris, T., Kay, K., Sabuncu, M. R., & Kuceyeski, A. F. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118812 · Cited by 27 · see citing works
Spurious reconstruction from brain activity
Shirakawa, K., Nagano, Y., Tanaka, M., Aoki, S., Muraki, Y., Majima, K., & Kamitani, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107515 · Cited by 19 · see citing works
Neural Decoding and Applications in Bioelectronic Medicine
Bouton, C. E. (2015) · Bioelectronic Medicine · doi:10.15424/bioelectronmed.2014.00012 · Cited by 14 · see citing works
Dynamic decoding of ongoing perception
van Gerven, M. A. J., Kok, P., de Lange, F. P., & Heskes, T. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.05.020 · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- [10]Palatucci M. Pomerleau D. Hinton G.E. Mitchell T.M. Zero-shot learning with semantic output codes Proceedings of the Advances in Neural Information Processing Systems Vancouver, BC, Canada 7–10 December 2009 NeurIPS Proceedings Volume 22
- [34]Dosovitskiy A. Beyer L. Kolesnikov A. Weissenborn D. Zhai X. Unterthiner T. Dehghani M. Minderer M. Heigold G. Gelly S. An image is worth 16x16 words: Transformers for image recognition at scale arXiv 2020 2010.11929
- [35]Vaswani A. Shazeer N. Parmar N. Uszkoreit J. Jones L. Gomez A.N. Kaiser Ł. Polosukhin I. Attention is all you need Proceedings of the Advances in Neural Information Processing Systems Long Beach, CA, USA 4–9 December 2017 NeurIPS Proceeding Volume 30
- [39]Song J. Meng C. Ermon S. Denoising diffusion implicit models arXiv 2020 2010.02502
- [45]Kingma D.P. Ba J. Adam: A method for stochastic optimization arXiv 2014 1412.6980
- [46]Loshchilov I. Hutter F. Sgdr: Stochastic gradient descent with warm restarts arXiv 2016 1608.03983
Cited by
Sorted by citations
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“The encoder-decoder method [52] efficiently represents scene semantics through encoding, interaction, and decoding.”
Lv, X., Tao, Y., & Xue, Y. (2025) · Applied Sciences · doi:10.3390/app15168871 · Cited by 9 · see citing works
Habijan, M., Krpić, Z., Perić, J., & Galić, I. (2025) · Electronics · doi:10.3390/electronics15010076 · Cited by 1 · see citing works
- Authors
- Wang, Y.
- Affiliations
- Fudan University (CN)
- Corresponding author
- Wang, Yun
- ORCID
- Wang, Yun
- Venue
- Brain Sciences
- Publisher
- MDPI AG
- Published
- 2024-09-28
- Volume
- 14
- Issue
- 10
- Pages
- 988
- ISSN
- 2076-3425 (ISSN-L)
- PMC
- PMC11506634
- OpenAlex
- W4402965619
- Cited by
- 3
- References
- 54
- Citations by year
- 2025: 3
- FWCI
- 0.73
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Brain Sciences (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Key R&D Program of China (2019YFA0709502, 2018YFC1312904, 2018SHZDZX01, B18015)
- Shanghai Municipal Science and Technology Major Project (2019YFA0709502, 2018YFC1312904, 2018SHZDZX01, B18015)
- ZJ Lab (2019YFA0709502, 2018YFC1312904, 2018SHZDZX01, B18015)
- Shanghai Center for Brain Science and Brain-Inspired Technology (2019YFA0709502, 2018YFC1312904, 2018SHZDZX01, B18015)
- 111 Project (2019YFA0709502, 2018YFC1312904, 2018SHZDZX01, B18015)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- neural decoding, multimodal training, diffusion models, functional MRI, image decoding, visual cortex, scene perception, latent representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
