The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Deep image reconstruction from human brain activity
Open accessAbstract
The mental contents of perception and imagery are thought to be encoded in hierarchical representations in the brain, but previous attempts to visualize perceptual contents have failed to capitalize on multiple levels of the hierarchy, leaving it challenging to reconstruct internal imagery. Recent work showed that visual cortical activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into the hierarchical features of a pre-trained deep neural network (DNN) for the same input image, providing a way to make use of the information from hierarchical visual features. Here, we present a novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers. We found that our method was able to reliably produce reconstructions that resembled the viewed natural images. A natural image prior introduced by a deep generator neural network effectively rendered semantically meaningful details to the reconstructions. Human judgment of the reconstructions supported the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. While our model was solely trained with natural images, it successfully generalized to artificial shapes, indicating that our model was not simply matching to exemplars. The same analysis applied to mental imagery demonstrated rudimentary reconstructions of the subjective content. Our results suggest that our method can effectively combine hierarchical neural representations to reconstruct perceptual and subjective images, providing a new window into the internal contents of the brain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
On the limited memory BFGS method for large scale optimization
Liu, D. C., & Nocedal, J. (1989) · Mathematical Programming · doi:10.1007/bf01589116 · Cited by 8,560
Pattern Recognition and Machine Learning
(2007) · Kybernetes · doi:10.1108/03684920710743466 · Cited by 8,265
The fusiform face area: a module in human extrastriate cortex specialized for face perception
Kanwisher, N., McDermott, J., & Chun, M. M. (1997) · The Journal of Neuroscience · doi:10.1523/jneurosci.17-11-04302.1997 · Cited by 7,738
Image Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025
A cortical representation of the local visual environment
Epstein, R., & Kanwisher, N. (1998) · Nature · doi:10.1038/33402 · Cited by 2,984
Borders of Multiple Visual Areas in Humans Revealed by Functional Magnetic Resonance Imaging
Sereno, M. I., Dale, A. M., Reppas, J. B., Kwong, K. K., Belliveau, J. W., Brady, T. J., Rosen, B. R., & Tootell, R. B. H. (1995) · Science · doi:10.1126/science.7754376 · Cited by 2,658
On the momentum term in gradient descent learning algorithms
Qian, N. (1999) · Neural Networks · doi:10.1016/s0893-6080(98)00116-6 · Cited by 2,312
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Understanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827
Using goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
Generating Images with Perceptual Similarity Metrics based on Deep NetworksPreprint
Dosovitskiy, A., & Brox, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02644 · Cited by 1,276
On optimization methods for deep learning
Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., Le, Q. V., & Ng, A. Y. (2011) · International Conference on Machine Learning · Cited by 1,076
Engel, S. A., Rumelhart, D. E., Wandell, B. A., Lee, A. T., Glover, G. H., Chichilnisky, E.-J., & Shadlen, M. N. (1994) · Nature · doi:10.1038/369525a0 · Cited by 990
Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies
Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011) · Current Biology · doi:10.1016/j.cub.2011.08.031 · Cited by 933
Cortical Regions Involved in Perceiving Object Shape
Kourtzi, Z., & Kanwisher, N. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-09-03310.2000 · Cited by 673
Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.044 · Cited by 544
Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M.-A., Morito, Y., Tanabe, H. C., Sadato, N., & Kamitani, Y. (2008) · Neuron · doi:10.1016/j.neuron.2008.11.004 · Cited by 543
Self-citationGeneric decoding of seen and imagined objects using hierarchical visual features
Horikawa, T., & Kamitani, Y. (2017) · Nature Communications · doi:10.1038/ncomms15037 · Cited by 516
Self-citationBayesian Reconstruction of Natural Images from Human Brain Activity
Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M., & Gallant, J. L. (2009) · Neuron · doi:10.1016/j.neuron.2009.09.006 · Cited by 511
Neural decoding of visual imagery during sleep
Horikawa, T., Tamaki, M., Miyawaki, Y., & Kamitani, Y. (2013) · Science · doi:10.1126/science.1234330 · Cited by 498
Self-citationBrain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, J., Geiger, F., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/407007 · Cited by 458
Inverse retinotopy: Inferring the visual content of images from brain activation patterns
Thirion, B., Duchesnay, E., Hubbard, E., Dubois, J., Poline, J.-B., Lebihan, D., & Dehaene, S. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.06.062 · Cited by 363
A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes
Naselaris, T., Olman, C. A., Stansbury, D. E., Ugurbil, K., & Gallant, J. L. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.10.018 · Cited by 356
Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision
Wen, H., Shi, J., Zhang, Y., Lu, K.-H., Cao, J., & Liu, Z. (2018) · Cerebral Cortex · doi:10.1093/cercor/bhx268 · Cited by 303
Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., & Clune, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09304 · Cited by 230
Han, K., Wen, H., Shi, J., Lu, K.-H., Zhang, Y., Fu, D., & Liu, Z. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.05.039 · Cited by 186
Generative adversarial networks for reconstructing natural images from brain activity
Seeliger, K., Güçlü, U., Ambrogioni, L., Güçlütürk, Y., & van Gerven, M. A. J. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.07.043 · Cited by 162
Linear reconstruction of perceived images from human brain activity
Schoenmakers, S., Barth, M., Heskes, T., & van Gerven, M. A. J. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.07.043 · Cited by 142
Neural portraits of perception: Reconstructing face images from evoked brain activity
Cowen, A., Chun, M. M., & Kuhl, B. A. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.03.018 · Cited by 114
Reconstructing Perceived and Retrieved Faces from Activity Patterns in Lateral Parietal Cortex
Lee, H., & Kuhl, B. A. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.4286-15.2016 · Cited by 106
A Direct Demonstration of Functional Differences between Subdivisions of Human V5/MT+
Strong, S., Silson, E. H., Gouws, A. D., Morland, A. B., & McKeefry, D. J. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhw362 · Cited by 92
Responses of human visual cortex to uniform surfaces
Haynes, J., Lotto, R. B., & Rees, G. E. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0307948101 · Cited by 87
Deep adversarial neural decodingPreprint
Güçlütürk, Y., Güçlü, U., Seeliger, K., Bosch, S., van Lier, R., & van Gerven, M. A. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07109 · Cited by 8
Cited by
Accelerating the discovery of materials for clean energy in the era of smart automation
Tabor, D. P., Roch, L. M., Saikin, S. K., Kreisbeck, C., Sheberla, D., Montoya, J. H., Dwaraknath, S., Aykol, M., Ortiz, C., Tribukait, H., et al. (2018) · Nature Reviews Materials · doi:10.1038/s41578-018-0005-z · Cited by 776
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
Lindsay, G. W. (2021) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01544 · Cited by 557
High-resolution image reconstruction with latent diffusion models from human brain activity
Takagi, Y., & Nishimoto, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01389 · Cited by 297
A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers
Zhang, X., Yao, L., Wang, X., Monaghan, J., McAlpine, D., & Zhang, Y. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc902 · Cited by 296
A systematic collection of medical image datasets for deep learning
Li, J., Zhu, G., Hua, C., Feng, M., BasheerBennamoun, Li, P., Lu, X., Song, J., Shen, P., Xu, X., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.12864 · Cited by 61
Metadata
- Authors
- Shen, G., Horikawa, T., Majima, K., & Kamitani, Y.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2019-01-14
- Volume
- 15
- Issue
- 1
- Pages
- e1006633
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2949290395
- Cited by
- 315
- References
- 39
- FWCI
- 17.44
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.