Reconstruction of natural visual scenes from neural spikes with deep neural networks
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Neural coding is one of the central questions in systems neuroscience for understanding how the brain processes stimulus from the environment, moreover, it is also a cornerstone for designing algorithms of brain-machine interface, where decoding incoming stimulus is highly demanded for better performance of physical devices. Traditionally researchers have focused on functional magnetic resonance imaging (fMRI) data as the neural signals of interest for decoding visual scenes. However, our visual perception operates in a fast time scale of millisecond in terms of an event termed neural spike. There are few studies of decoding by using spikes. Here we fulfill this aim by developing a novel decoding framework based on deep neural networks, named spike-image decoder (SID), for reconstructing natural visual scenes, including static images and dynamic videos, from experimentally recorded spikes of a population of retinal ganglion cells. The SID is an end-to-end decoder with one end as neural spikes and the other end as images, which can be trained directly such that visual scenes are reconstructed from spikes in a highly accurate fashion. Our SID also outperforms on the reconstruction of visual stimulus compared to existing fMRI decoding models. In addition, with the aid of a spike encoder, we show that SID can be generalized to arbitrary visual scenes by using the image datasets of MNIST, CIFAR10, and CIFAR100. Furthermore, with a pre-trained SID, one can decode any dynamic videos to achieve real-time encoding and decoding of visual scenes by spikes. Altogether, our results shed new light on neuromorphic computing for artificial visual systems, such as event-based visual cameras and visual neuroprostheses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 7×
“In this study, simulations were done on three popular image datasets, MNIST, CIFAR10, and CIFAR100 [37, 38].”
MethodInfluentialMultiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253 · cited 1×
“Therefore, these measures are a general guideline rather than the precise characteristic, as it seems that there is no perfect measure [42, 43].”
BackgroundGenerative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477
Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3008413 · Cited by 2,566
A 128$\times$128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor
Lichtsteiner, P., Posch, C., & Delbruck, T. (2008) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2007.914337 · Cited by 2,485 · cited 1×
“Therefore, our method can be applied to neuromorphic retina cameras with spiking or events, such as the dynamic vision sensor [70].”
MethodNatural Image Statistics and Neural Representation
Simoncelli, E. P., & Olshausen, B. A. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.1193 · Cited by 2,460 · cited 2×
“Such observations lead to a central question for systems neuroscience on the sensory system: how neurons represent the input-output relationship between stimuli and their spikes [3, 4, 5].”
BackgroundUsing goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
Towards artificial general intelligence with hybrid Tianjic chip architecture
Pei, J., Deng, L., Song, S., Zhao, M., Zhang, Y., Wu, S., Wang, G., Zou, Z., Wu, Z., He, W., et al. (2019) · Nature · doi:10.1038/s41586-019-1424-8 · Cited by 1,082 · cited 1×
“Advancements of recent artificial intelligence computing align with the development of the next generation of neuromorphic chips and devices, where the new data format is processed as spikes or events [47, 66, 67, 68, 69].”
BackgroundSelf-citationReconstructing Visual Experiences from Brain Activity Evoked by Natural Movies
Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011) · Current Biology · doi:10.1016/j.cub.2011.08.031 · Cited by 933
Eye Smarter than Scientists Believed: Neural Computations in Circuits of the Retina
Gollisch, T., & Meister, M. (2010) · Neuron · doi:10.1016/j.neuron.2009.12.009 · Cited by 795 · cited 2×
“anisms have been identified in understanding the retinal computation of visual scenes by its neurons and neural circuitry [6, 7, 8, 9, 10, 11].”
BackgroundMiyawaki, Y., Uchida, H., Yamashita, O., Sato, M.-A., Morito, Y., Tanabe, H. C., Sadato, N., & Kamitani, Y. (2008) · Neuron · doi:10.1016/j.neuron.2008.11.004 · Cited by 543 · cited 1×
“signal is fMRI data [32], [33], [34], [15], [35], [19].”
BackgroundRetinal stimulation strategies to restore vision: fundamentals and systems
Yue, L., Weiland, J. D., Roska, B., & Humayun, M. S. (2016) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2016.05.002 · Cited by 272 · cited 1×
“important and beyond the traditional way of computing the distance between two or more spike trains in general [61, 62, 63], as recent studies suggest that the performance of neuroprosthesis for restoring vision can be improved by the encoding/decoding algorithms besides of hardware design [60, 64, 65].”
BackgroundRetinal prosthetic strategy with the capacity to restore normal vision
Nirenberg, S., & Pandarinath, C. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1207035109 · Cited by 207
Cited by
Reconstruction of natural images from responses of primate retinal ganglion cells
Brackbill, N., Rhoades, C., Kling, A., Shah, N. P., Sher, A., Litke, A. M., & Chichilnisky, E. (2020) · eLife · doi:10.7554/elife.58516 · Cited by 57
“, 2017), small data sets from salamander retina with unidentified cell types and incomplete spatial coverage (Zhang et al., 2020), or reconstruction of moving dot patterns from rat retina without clear cell types (Botella-Soler et al.”
BackgroundComputational challenges and opportunities for a bi-directional artificial retina
Shah, N. P., & Chichilnisky, E. J. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/aba8b1 · Cited by 39
High-Fidelity Reproduction of Visual Signals by Electrical Stimulation in the Central Primate Retina
Gogliettino, A. R., Madugula, S. S., Grosberg, L. E., Vilkhu, R. S., Brown, J., Nguyen, H., Kling, A., Hottowy, P., Dąbrowski, W., Sher, A., et al. (2023) · The Journal of Neuroscience · doi:10.1523/jneurosci.1091-22.2023 · Cited by 30
High-fidelity reproduction of visual signals by electrical stimulation in the central primate retina
Gogliettino, A. R., Madugula, S. S., Grosberg, L. E., Vilkhu, R. S., Brown, J., Nguyen, H., Kling, A., Hottowy, P., Dąbrowski, W., Sher, A., et al. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.05.24.493162 · Cited by 30
“Future work could apply more sophisticated nonlinear reconstruction methods for optimizing visual perception (Parthasarathy et al., 2017; Zhang et al., 2020; Kim et al., 2021; Wu et al., 2022), such as the convolutional autoencoder used to enhance the naturalistic image reconstructions here…”
Metadata
- Authors
- Zhang, Y., Jia, S., Zheng, Y., Yu, Z., Tian, Y., Ma, S., Huang, T., & Liu, J. K.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2020-05
- Volume
- 125
- Pages
- 19-30
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W3005431840
- Cited by
- 59
- References
- 14
- FWCI
- 3.63
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.