The DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.
Deep convolutional models improve predictions of macaque V1 responses to natural images
Open accessAbstract
Abstract Despite great efforts over several decades, our best models of primary visual cortex (V1) still predict spiking activity quite poorly when probed with natural stimuli, highlighting our limited understanding of the nonlinear computations in V1. Recently, two approaches based on deep learning have been successfully applied to neural data: On the one hand, transfer learning from networks trained on object recognition worked remarkably well for predicting neural responses in higher areas of the primate ventral stream, but has not yet been used to model spiking activity in early stages such as V1. On the other hand, data-driven models have been used to predict neural responses in the early visual system (retina and V1) of mice, but not primates. Here, we test the ability of both approaches to predict spiking activity in response to natural images in V1 of awake monkeys. Even though V1 is rather at an early to intermediate stage of the visual system, we found that the transfer learning approach performed similarly well to the data-driven approach and both outperformed classical linear-nonlinear and wavelet-based feature representations that build on existing theories of V1. Notably, transfer learning using a pre-trained feature space required substantially less experimental time to achieve the same performance. In conclusion, multi-layer convolutional neural networks (CNNs) set the new state of the art for predicting neural responses to natural images in primate V1 and deep features learned for object recognition are better explanations for V1 computation than all previous filter bank theories. This finding strengthens the necessity of V1 models that are multiple nonlinearities away from the image domain and it supports the idea of explaining early visual cortex based on high-level functional goals. Author summary Predicting the responses of sensory neurons to arbitrary natural stimuli is of major importance for understanding their function. Arguably the most studied cortical area is primary visual cortex (V1), where many models have been developed to explain its function. However, the most successful models built on neurophysiologists’ intuitions still fail to account for spiking responses to natural images. Here, we model spiking activity in primary visual cortex (V1) of monkeys using deep convolutional neural networks (CNNs), which have been successful in computer vision. We both trained CNNs directly to fit the data, and used CNNs trained to solve a high-level task (object categorization). With these approaches, we are able to outperform previous models and improve the state of the art in predicting the responses of early visual neurons to natural images. Our results have two important implications. First, since V1 is the result of several nonlinear stages, it should be modeled as such. Second, functional models of entire visual pathways, of which V1 is an early stage, do not only account for higher areas of such pathways, but also provide useful representations for V1 predictions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Matplotlib: a 2D graphics environment
Hunter, J. D. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.55 · Cited by 39,249
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
The NumPy Array: A Structure for Efficient Numerical Computation
van der Walt, S., Colbert, S. C., & Varoquaux, G. (2011) · Computing in Science & Engineering · doi:10.1109/mcse.2011.37 · Cited by 9,381
Receptive fields and functional architecture of monkey striate cortex
Hubel, D. H., & Wiesel, T. N. (1968) · The Journal of Physiology · doi:10.1113/jphysiol.1968.sp008455 · Cited by 6,568
Image Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025
Self-citationEmergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694
Receptive fields of single neurones in the cat's striate cortex
Hubel, D. H., & Wiesel, T. N. (1959) · The Journal of Physiology · doi:10.1113/jphysiol.1959.sp006308 · Cited by 4,653
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Spatiotemporal energy models for the perception of motion
Adelson, E. H., & Bergen, J. R. (1985) · Journal of the Optical Society of America A · doi:10.1364/josaa.2.000284 · Cited by 3,442
Hierarchical models of object recognition in cortex
Riesenhuber, M., & Poggio, T. (1999) · Nature Neuroscience · doi:10.1038/14819 · Cited by 3,309
Docker: lightweight Linux containers for consistent development and deployment
Merkel, D. (2014) · Linux journal · Cited by 3,191
Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks
Oquab, M., Bottou, L., Laptev, I., & Sivic, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.222 · Cited by 3,150 · cited 1×
“In the machine learning community, this concept is known as transfer 33 learning and has been very successful in deep learning [25,26].”
BackgroundJupyter Notebooks – a publishing format for reproducible computational workflows
Thomas, K., Benjamin, R.-K., Fernando, P., Brian, G., Matthias, B., Jonathan, F., Kyle, K., Jessica, H., Jason, G., Sylvain, C., et al. (2016) · Positioning and Power in Academic Publishing: Players, Agents and Agendas · doi:10.3233/978-1-61499-649-1-87 · Cited by 3,063
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Normalization as a canonical neural computation
Carandini, M., & Heeger, D. J. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3136 · Cited by 1,906
Normalization of cell responses in cat striate cortex
Heeger, D. J. (1992) · Visual Neuroscience · doi:10.1017/s0952523800009640 · Cited by 1,835
Zeiler, M. D., Krishnan, D., Taylor, G. W., & Fergus, R. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539957 · Cited by 1,666
Jones, J. P., & Palmer, L. A. (1987) · Journal of Neurophysiology · doi:10.1152/jn.1987.58.6.1233 · Cited by 1,658 · cited 1×
“6 Our current standard model of V1 is based on linear-nonlinear models (LN) [4, 5] and 7 energy models [6] to explain simple and complex cells, respectively.”
MethodUsing goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
Two-dimensional spectral analysis of cortical receptive field profiles
Daugman, J. G. (1980) · Vision Research · doi:10.1016/0042-6989(80)90065-6 · Cited by 1,208
Deep supervised, but not unsupervised, models may explain IT cortical representation
Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003915 · Cited by 1,204
Zipser, D., & Andersen, R. A. (1988) · Nature · doi:10.1038/331679a0 · Cited by 1,187
What Is the Goal of Sensory Coding?
Field, D. J. (1994) · Neural Computation · doi:10.1162/neco.1994.6.4.559 · Cited by 1,183
Waskom, M., Botvinnik, O. B., O'Kane, D., Hobson, P., Ostblom, J., Lukauskas, S., Gemperline, D. C., Augspurger, T., Halchenko, Y., Cole, J. B., et al. (2014) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.592845 · Cited by 1,119
Kriegeskorte, N. (2015) · Annual Review of Vision Science · doi:10.1146/annurev-vision-082114-035447 · Cited by 1,093
Ceramide-orchestrated signalling in cancer cells
Morad, S. A. F., & Cabot, M. C. (2013) · Nature Reviews Cancer · doi:10.1038/nrc3398 · Cited by 1,014
Guclu, U., & van Gerven, M. A. J. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.5023-14.2015 · Cited by 961
Spatial summation in the receptive fields of simple cells in the cat's striate cortex.
Movshon, J. A., Thompson, I. D., & Tolhurst, D. J. (1978) · The Journal of Physiology · doi:10.1113/jphysiol.1978.sp012488 · Cited by 890
Cadieu, C. F., Hong, H., Yamins, D. L. K., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., & DiCarlo, J. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003963 · Cited by 824
Nature and Interaction of Signals From the Receptive Field Center and Surround in Macaque V1 Neurons
Cavanaugh, J. R., Bair, W., & Movshon, J. A. (2002) · Journal of Neurophysiology · doi:10.1152/jn.00692.2001 · Cited by 778
Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., & Oliva, A. (2016) · Scientific Reports · doi:10.1038/srep27755 · Cited by 716
Orientation Selectivity in Macaque V1: Diversity and Laminar Dependence
Ringach, D. L., Shapley, R. M., & Hawken, M. J. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-13-05639.2002 · Cited by 649
Decorrelated neuronal firing in cortical microcircuits
Ecker, A. S., Berens, P., Keliris, G. A., Bethge, M., Logothetis, N. K., & Tolias, A. S. (2010) · Science · doi:10.1126/science.1179867 · Cited by 640
Self-citationReceptive field organization of complex cells in the cat's striate cortex.
Movshon, J. A., Thompson, I. D., & Tolhurst, D. J. (1978) · The Journal of Physiology · doi:10.1113/jphysiol.1978.sp012489 · Cited by 622
Do We Know What the Early Visual System Does?
Carandini, M., Demb, J. B., Mante, V., Tolhurst, D. J., Dan, Y., Olshausen, B. A., Gallant, J. L., & Rust, N. C. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.3726-05.2005 · Cited by 600
Dynamics of orientation tuning in macaque primary visual cortex
Ringach, D. L., Hawken, M. J., & Shapley, R. (1997) · Nature · doi:10.1038/387281a0 · Cited by 537
Spatiotemporal Elements of Macaque V1 Receptive Fields
Rust, N. C., Schwartz, O., Movshon, J. A., & Simoncelli, E. P. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.021 · Cited by 457
Cavanaugh, J. R., Bair, W., & Movshon, J. A. (2002) · Journal of Neurophysiology · doi:10.1152/jn.00693.2001 · Cited by 420
How Close Are We to Understanding V1?
Olshausen, B. A., & Field, D. J. (2005) · Neural Computation · doi:10.1162/0899766054026639 · Cited by 414
The cortex transform: Rapid computation of simulated neural images
Watson, A. B. (1987) · Computer Vision, Graphics, and Image Processing · doi:10.1016/s0734-189x(87)80184-6 · Cited by 394
Texture Synthesis Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.07376 · Cited by 378
Self-citationTime Course and Time-Distance Relationships for Surround Suppression in Macaque V1 Neurons
Bair, W., Cavanaugh, J. R., & Movshon, J. A. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-20-07690.2003 · Cited by 345
State Dependence of Noise Correlations in Macaque Primary Visual Cortex
Ecker, A. S., Berens, P., Cotton, R. J., Subramaniyan, M., Denfield, G. H., Cadwell, C. R., Smirnakis, S. M., Bethge, M., & Tolias, A. S. (2014) · Neuron · doi:10.1016/j.neuron.2014.02.006 · Cited by 323
Self-citationHalf-squaring in responses of cat striate cells
Heeger, D. J. (1992) · Visual Neuroscience · doi:10.1017/s095252380001124x · Cited by 262
Characterization of Neural Responses with Stochastic StimuliPreprint
Simoncelli, E. P., Pillow, J. W., Paninski, L., & Schwartz, O. (2004) · Cited by 236
Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet
Kümmerer, M., Theis, L., & Bethge, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.1045 · Cited by 231
Self-citationDeep Learning Models of the Retinal Response to Natural Scenes
McIntosh, L., Maheswaranathan, N., Nayebi, A., Ganguli, S., & Baccus, S. A. (2017) · PubMed · doi:10.48550/arxiv.1702.01825 · Cited by 230
Spatial Structure of Complex Cell Receptive Fields Measured with Natural Images
Touryan, J., Felsen, G., & Dan, Y. (2005) · Neuron · doi:10.1016/j.neuron.2005.01.029 · Cited by 194
Neural Representation of Natural Images in Visual Area V2
Willmore, B. D. B., Prenger, R., & Gallant, J. L. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.4099-09.2010 · Cited by 111
A Convolutional Subunit Model for Neuronal Responses in Macaque V1
Vintch, B., Movshon, J. A., & Simoncelli, E. P. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.2815-13.2015 · Cited by 105
Beyond the classical receptive field: The effect of contextual stimuli
Spillmann, L., Dresp, B., & Tseng, C. (2015) · Journal of Vision · doi:10.1167/15.9.7 · Cited by 82
DataJoint: managing big scientific data using MATLAB or PythonPreprint
Yatsenko, D., Reimer, J., Ecker, A. S., Walker, E. Y., Sinz, F. H., Berens, P., Hoenselaar, A., Cotton, R. J., Siapas, A. G., & Tolias, A. S. (2015) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/031658 · Cited by 82
Self-citationKalman filter mixture model for spike sorting of non-stationary data
Calabrese, A., & Paninski, L. (2010) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2010.12.002 · Cited by 78
Nonlinear V1 responses to natural scenes revealed by neural network analysis
Prenger, R., Wu, M. C.-K., David, S. V., & Gallant, J. L. (2004) · Neural Networks · doi:10.1016/j.neunet.2004.03.008 · Cited by 77
Using deep learning to probe the neural code for images in primary visual cortex
Kindel, W., Christensen, E., & Zylberberg, J. (2019) · Journal of Vision · doi:10.1167/19.4.29 · Cited by 73
Talebi, V., & Baker, C. L. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.4661-12.2012 · Cited by 68
Natural Image Coding in V1: How Much Use Is Orientation Selectivity?
Eichhorn, J., Sinz, F. H., & Bethge, M. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000336 · Cited by 65
Self-citationAttentional fluctuations induce shared variability in macaque primary visual cortex
Denfield, G. H., Ecker, A. S., Shinn, T., Bethge, M., & Tolias, A. S. (2018) · Nature Communications · doi:10.1038/s41467-018-05123-6 · Cited by 65
Self-citationMultilayer Recurrent Network Models of Primate Retinal Ganglion Cell Responses
Batty, E., Merel, J., Brackbill, N., Heitman, A., Sher, A., Litke, A. M., Chichilnisky, E. J., & Paninski, L. (2017) · International Conference on Learning Representations · Cited by 61
Neural system identification for large populations separating "what" and "where"Preprint
Klindt, D. A., Ecker, A. S., Euler, T., & Bethge, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02653 · Cited by 58
Self-citationTime dilates more with apparent than with physical speed
Goréa, A., & Kim, J. (2015) · Journal of Vision · doi:10.1167/15.1.7 · Cited by 57
Tang, S., Lee, T. S., Li, M., Zhang, Y., Xu, Y., Liu, F., Teo, B., & Jiang, H. (2017) · Current Biology · doi:10.1016/j.cub.2017.11.039 · Cited by 55
Model Constrained by Visual Hierarchy Improves Prediction of Neural Responses to Natural Scenes
Antolík, J., Hofer, S. B., Bednar, J. A., & Mrsic‐Flogel, T. D. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004927 · Cited by 50
Convolutional neural network models of V1 responses to complex patterns
Zhang, Y., Lee, T. S., Li, M., Liu, F., & Tang, S. (2018) · Journal of Computational Neuroscience · doi:10.1007/s10827-018-0687-7 · Cited by 45
The Berkeley Wavelet Transform: A Biologically Inspired Orthogonal Wavelet Transform
Willmore, B. D. B., Prenger, R., Wu, M. C., & Gallant, J. L. (2008) · Neural Computation · doi:10.1162/neco.2007.05-07-513 · Cited by 36
Model-based spike sorting with a mixture of drifting t-distributions
Shan, K. Q., Lubenov, E. V., & Siapas, A. G. (2017) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2017.06.017 · Cited by 35
Hierarchical Modeling of Local Image Features through L_p-Nested Symmetric Distributions
Bethge, M., Simoncelli, E. P., & Sinz, F. H. (2009) · neural information processing systems · Cited by 23
Self-citationA performance-optimized model of neural responses across the ventral visual streamPreprint
Seibert, D., Yamins, D. L. K., Ardila, D., Hong, H., DiCarlo, J. J., & Gardner, J. L. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/036475 · Cited by 14
Using deep learning to reveal the neural code for images in primary visual cortexPreprint
Kindel, W., Christensen, E., & Zylberberg, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06208 · Cited by 14
Cited by
A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F., Charest, I., et al. (2022) · Nature Neuroscience · doi:10.1038/s41593-021-00962-x · Cited by 577
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
Lindsay, G. W. (2021) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01544 · Cited by 557
Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.044 · Cited by 544
The neural architecture of language: Integrative modeling converges on predictive processing
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2105646118 · Cited by 517
Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, J., Geiger, F., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/407007 · Cited by 458
Metadata
- Authors
- Cadena, S. A., Denfield, G. H., Walker, E. Y., Gatys, L. A., Tolias, A. S., Bethge, M., & Ecker, A. S.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2019-04-23
- Volume
- 15
- Issue
- 4
- Pages
- e1006897
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2949449018
- Cited by
- 347
- References
- 77
- FWCI
- 21.23
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.