The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
High-resolution image reconstruction with latent diffusion models from human brain activity
Open accessAbstract
Reconstructing visual experiences from human brain activity offers a unique way to understand how the brain represents the world, and to interpret the connection between computer vision models and our visual system. While deep generative models have recently been employed for this task, reconstructing realistic images with high semantic fidelity is still a challenging problem. Here, we propose a new method based on a diffusion model (DM) to reconstruct images from human brain activity obtained via functional magnetic resonance imaging (fMRI). More specifically, we rely on a latent diffusion model (LDM) termed Stable Diffusion. This model reduces the computational cost of DMs, while preserving their high generative performance. We also characterize the inner mechanisms of the LDM by studying how its different components (such as the latent vector of image Z, conditioning inputs C, and different elements of the denoising U-Net) relate to distinct brain functions. We show that our proposed method can reconstruct high-resolution images with high fidelity in straight-forward fashion, without the need for any additional training and fine-tuning of complex deep-learning models. We also provide a quantitative interpretation of different LDM components from a neuroscientific perspective. Overall, our study proposes a promising method for reconstructing images from human brain activity, and provides a new framework for understanding DMs. Please check out our webpage at https://sites.google.com/view/stablediffusion-withbrain/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Diffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
Score-Based Generative Modeling through Stochastic Differential Equations
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020) · arXiv · doi:10.48550/arxiv.2011.13456 · Cited by 1,953
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Image Super-Resolution Via Iterative Refinement
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204461 · Cited by 1,829 · cited 1×
“DMs have achieved state-of-the-art performance in several tasks involving conditional image generation [4,39,49], image super resolution [40], image colorization [38], and other related tasks [6, 16, 33, 41].”
BackgroundImage Super-Resolution via Iterative Refinement
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M. (2021) · arXiv · doi:10.48550/arxiv.2104.07636 · Cited by 1,829
Palette: Image-to-Image Diffusion Models
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., & Norouzi, M. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530757 · Cited by 1,717
Generative Modeling by Estimating Gradients of the Data Distribution
Song, Y., & Ermon, S. (2019) · arXiv · doi:10.48550/arxiv.1907.05600 · Cited by 1,453
Identifying natural images from human brain activity
Kay, K. N., Naselaris, T., Prenger, R. J., & Gallant, J. L. (2008) · Nature · doi:10.1038/nature06713 · Cited by 1,286
A Connection Between Score Matching and Denoising Autoencoders
Vincent, P. (2011) · Neural Computation · doi:10.1162/neco_a_00142 · Cited by 1,011
Guclu, U., & van Gerven, M. A. J. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.5023-14.2015 · Cited by 961
Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies
Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011) · Current Biology · doi:10.1016/j.cub.2011.08.031 · Cited by 933
Self-citationGrad-TTS: A Diffusion Probabilistic Model for Text-to-SpeechPreprint
Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., & Kudinov, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.06337 · Cited by 750
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F., Charest, I., et al. (2022) · Nature Neuroscience · doi:10.1038/s41593-021-00962-x · Cited by 577
Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.044 · Cited by 544
Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M.-A., Morito, Y., Tanabe, H. C., Sadato, N., & Kamitani, Y. (2008) · Neuron · doi:10.1016/j.neuron.2008.11.004 · Cited by 543 · cited 3×
“Reconstructing visual images from brain activity, such as that measured by functional Magnetic Resonance Imaging (fMRI), is an intriguing but challenging problem, because the underlying representations in the brain are largely unknown, and the sample size typically associated with brain data is relatively small [17, 2…”
BackgroundThe neural architecture of language: Integrative modeling converges on predictive processing
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2105646118 · Cited by 517
Generic decoding of seen and imagined objects using hierarchical visual features
Horikawa, T., & Kamitani, Y. (2017) · Nature Communications · doi:10.1038/ncomms15037 · Cited by 516
Bayesian Reconstruction of Natural Images from Human Brain Activity
Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M., & Gallant, J. L. (2009) · Neuron · doi:10.1016/j.neuron.2009.09.006 · Cited by 511
Neural decoding of visual imagery during sleep
Horikawa, T., Tamaki, M., Miyawaki, Y., & Kamitani, Y. (2013) · Science · doi:10.1126/science.1234330 · Cited by 498
Shared computational principles for language processing in humans and deep language models
Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., et al. (2022) · Nature Neuroscience · doi:10.1038/s41593-022-01026-4 · Cited by 447
Semantic reconstruction of continuous language from non-invasive brain recordings
Tang, J., LeBel, A., Jain, S., & Huth, A. G. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01304-9 · Cited by 442 · cited 1×
“Examples include explicitly presented visual stimuli [17, 26, 30, 32], semantic content of the presented stimuli [15, 31, 52], imagined content [13, 29], perceived emotions [12, 20, 51], and many other related applications [14, 28].”
BackgroundRecurrence is required to capture the representational dynamics of the human visual system
Kietzmann, T. C., Spoerer, C. J., Sörensen, L. K. A., Cichy, R. M., Hauk, O., & Kriegeskorte, N. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1905544116 · Cited by 406
A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes
Naselaris, T., Olman, C. A., Stansbury, D. E., Ugurbil, K., & Gallant, J. L. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.10.018 · Cited by 356
Brains and algorithms partially converge in natural language processing
Caucheteux, C., & King, J.-R. (2022) · Communications Biology · doi:10.1038/s42003-022-03036-1 · Cited by 318
Deep image reconstruction from human brain activity
Shen, G., Horikawa, T., Majima, K., & Kamitani, Y. (2019) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006633 · Cited by 315
Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision
Wen, H., Shi, J., Zhang, Y., Lu, K.-H., Cao, J., & Liu, Z. (2018) · Cerebral Cortex · doi:10.1093/cercor/bhx268 · Cited by 303
UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic ModelsPreprint
Sasaki, H., Willcocks, C. G., & Breckon, T. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05358 · Cited by 287
Mind Reader: Reconstructing Complex Images from Brain Activities
Lin, S., Sprague, T., & Singh, A. K. (2022) · neural information processing systems · doi:10.52202/068431-2148 · Cited by 176
Groen, I. I., Greene, M. R., Baldassano, C., Fei-Fei, L., Beck, D. M., & Baker, C. I. (2018) · eLife · doi:10.7554/elife.32962 · Cited by 169
Feature-space selection with banded ridge regression
la Tour, T. D., Eickenberg, M., Núñez-Elizalde, A. O., & Gallant, J. L. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119728 · Cited by 165
Decoding the Semantic Content of Natural Movies from Human Brain Activity
Huth, A. G., Lee, T., Nishimoto, S., Bilenko, N. Y., Vu, A. T., & Gallant, J. L. (2016) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2016.00081 · Cited by 163
Self-citationEnd-to-End Deep Image Reconstruction From Human Brain Activity
Shen, G., Dwivedi, K., Majima, K., Horikawa, T., & Kamitani, Y. (2019) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2019.00021 · Cited by 163
Generative adversarial networks for reconstructing natural images from brain activity
Seeliger, K., Güçlü, U., Ambrogioni, L., Güçlütürk, Y., & van Gerven, M. A. J. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.07.043 · Cited by 162
Ren, Z., Li, J., Xue, X., Li, X., Yang, F., Jiao, Z., & Gao, X. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2020.117602 · Cited by 142
Gaziv, G., Beliy, R., Granot, N., Hoogi, A., Strappini, F., Golan, T., & Irani, M. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119121 · Cited by 131
Horikawa, T., Cowen, A. S., Keltner, D., & Kamitani, Y. (2020) · iScience · doi:10.1016/j.isci.2020.101060 · Cited by 101
Takagi, Y., Sakai, Y., Abe, Y., Nishida, S., Harrison, B. J., Martínez‐Zalacaín, I., Soriano‐Mas, C., Narumoto, J., & Tanaka, S. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.01.080 · Cited by 96
Self-citationDecoding naturalistic experiences from human brain activity via distributed representations of words
Nishida, S., & Nishimoto, S. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.08.017 · Cited by 84
Self-citationReconstructing Natural Scenes from fMRI Patterns using BigBiGAN
Mozafari, M., Reddy, L., & VanRullen, R. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206960 · Cited by 82
From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRIPreprint
Beliy, R., Gaziv, G., Hoogi, A., Strappini, F., Golan, T., & Irani, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.02431 · Cited by 76
JPEG Artifact Correction using Denoising Diffusion Restoration ModelsPreprint
Kawar, B., Song, J., Ermon, S., & Elad, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.11888 · Cited by 73
Distinct dimensions of emotion in the human brain and their representation on the cortical surface
Koide‐Majima, N., Nakai, T., & Nishimoto, S. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117258 · Cited by 62
Self-citationPredicting speech from a cortical hierarchy of event-based time scales
Schmitt, L., Erb, J., Tune, S., Rysop, A. U., Hartwigsen, G., & Obleser, J. (2021) · Science Advances · doi:10.1126/sciadv.abi6070 · Cited by 61
Quantitative models reveal the organization of diverse cognitive functions in the brain
Nakai, T., & Nishimoto, S. (2020) · Nature Communications · doi:10.1038/s41467-020-14913-w · Cited by 59
Self-citationNatural Image Reconstruction From fMRI Using Deep Learning: A Survey
Rakhimberdina, Z., Jodelet, Q., Liu, X., & Murata, T. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.795488 · Cited by 49
Cascaded Tuning to Amplitude Modulation for Natural Sound Recognition
Koumura, T., Terashima, H., & Furukawa, S. (2019) · Journal of Neuroscience · doi:10.1523/jneurosci.2914-18.2019 · Cited by 43
Li, X., Tang, L., Deng, J., Qi, X., Zhang, J., Li, M., Liu, Y., Liu, Y., Zhao, W., Gu, Y., et al. (2022) · Communications Biology · doi:10.1038/s42003-022-03135-z · Cited by 37
Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GANPreprint
Fang, T., Qi, Y., & Pan, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.12083 · Cited by 34
Brain2Pix: Fully convolutional naturalistic video frame reconstruction from brain activity
Le, L., Ambrogioni, L., Seeliger, K., Güçlütürk, Y., van Gerven, M. A. J., & Güçlü, U. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.940972 · Cited by 32
Cited by
Catd: unified representation learning for eeg-to-fmri cross-modal generation
Yao, W., Lyu, Z., Mahmud, M., Zhong, N., Lei, B., & Wang, S. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3550206 · Cited by 24
Efficient neural decoding based on multimodal training
Wang, Y. (2024) · Brain Sciences · doi:10.3390/brainsci14100988 · Cited by 3
How Adversarial REM Dreams May Facilitate Creativity, and Why We Become Aware of Them
Deperrois, N., Petrovici, M., Jordan, J., Huber, L., & Senn, W. (2024) · Clinical and Translational Neuroscience · doi:10.3390/ctn8020021 · Cited by 3
Metadata
- Authors
- Takagi, Y., & Nishimoto, S.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 14453-14463
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386076190
- Cited by
- 297
- References
- 57
- FWCI
- 33.54
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.