Performance-optimized hierarchical models predict neural responses in higher visual cortex
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Significance: Humans and monkeys easily recognize objects in scenes. This ability is known to be supported by a network of hierarchically interconnected brain areas. However, understanding neurons in higher levels of this hierarchy has long remained a major challenge in visual systems neuroscience. We use computational techniques to identify a neural network model that matches human performance on challenging object categorization tasks. Although not explicitly constrained to match neural data, this model turns out to be highly predictive of neural responses in both the V4 and inferior temporal cortex, the top two layers of the ventral visual hierarchy. In addition to yielding greatly improved models of visual cortex, these results suggest that a process of biological performance optimization directly shaped neural mechanisms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This is problematic, since CNNs are being used as computational models for human object recognition (e.g. Cadieu et al., 2014; Yamins et al., 2014).”
From ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness (2018) · cites this paper as Yamins et al., 2014
In the citing paper
“Analogous approaches have proven successful in relating the structure of neural activity to computational functions in other domains, including vision 17–19, audition 20, olfaction 21,22, thermosensation 23, perceptual discrimination 24, facial perception 25, and navigation 26,27.”
From A virtual rodent predicts the structure of neural activity across behaviours (2024) · cites this paper as [17]
In the citing paper
“This suggests that more performant models exhibit representations more similar to those of both DLS and MC, consistent with previous reports comparing neural activity with task-optimized neural networks 17,20.”
From A virtual rodent predicts the structure of neural activity across behaviours (2024) · cites this paper as [17]
In the citing paper
“Neural networks also show substantial alignment with biological representations in the brain (Yamins et al., 2014).”
From The Platonic Representation Hypothesis (2024) · cites this paper as Yamins et al., 2014
In the citing paper
“Task-optimized artificial neural network models, for instance of mammalian visual pathways22, have previously demonstrated only a coarse correspondence of the population neural activity between model layers and brain regions.”
From Connectome-constrained networks predict neural activity across the fly visual system (2024) · cites this paper as [22]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 1×
“To control for low-level confounds, we tested the (trivial) pixel model, as well as SIFT, a simple baseline computer vision model (30).”
MethodDistributed hierarchical processing in the primate cerebral cortex
Felleman, D. J., & Van Essen, D. C. (1991) · Cerebral Cortex · doi:10.1093/cercor/1.1.1 · Cited by 7,783 · cited 10×
“Humans effortlessly recognize objects rapidly and accurately despite this enormous variation, an impressive computational feat (1).”
BackgroundMethodInfluentialFukushima, K. (1980) · Biological Cybernetics · doi:10.1007/bf00344251 · Cited by 5,759 · cited 1×
“Proc Natl Acad Sci USA 104(15):6424–6429.”
Marr, D. (1982) · Cumulative Index of Computer Aided Architectural Design · Cited by 5,733
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Preface: Cerebral Cortex Has Come of Age
Goldman-Rakic, P. S., & Rakic, P. (1991) · Cerebral Cortex · doi:10.1093/cercor/1.1.1-a · Cited by 3,393
How does the brain solve visual object recognition?
DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012) · Neuron · doi:10.1016/j.neuron.2012.01.010 · Cited by 1,790 · cited 9×
“HMAX Model (25”
BackgroundMethodInfluentialSelf-citationThe lateral occipital complex and its role in object recognition
Grill-Spector, K., Kourtzi, Z., & Kanwisher, N. (2001) · Vision Research · doi:10.1016/s0042-6989(01)00073-6 · Cited by 1,406 · cited 1×
“This ability is supported by a set of interconnected brain areas collectively called the ventral visual stream (2, 3), with homologous areas in nonhuman primates (4).”
BackgroundInferotemporal cortex and object vision
Tanaka, K. (1996) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.19.030196.000545 · Cited by 1,402 · cited 1×
“The ventral stream is thought to function as a series of hierarchical processing stages (5 – 7) that encode image content (e.g., object identity and category) increasingly explicitly in successive cortical areas (1, 8, 9).”
BackgroundLogothetis, N. K., & Sheinberg, D. L. (1996) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.19.030196.003045 · Cited by 1,351
Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey
Kriegeskorte, N., Mur, M., Ruff, D. A., Kiani, R., Bodurka, J., Esteky, H., Tanaka, K., & Bandettini, P. A. (2008) · Neuron · doi:10.1016/j.neuron.2008.10.043 · Cited by 1,344 · cited 4×
“Although first principles-based models of higher ventral cortex have been proposed (15–20), these models fail to match important features of the higher ventral visual neural representation in both humans and macaques (4, 21).”
BackgroundMethodInfluentialLearning methods for generic object recognition with invariance to pose and lighting
LeCun, Y., Huang, F. J., & Bottou, L. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315150 · Cited by 1,324
Untangling invariant object recognition
DiCarlo, J. J., & Cox, D. D. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.06.010 · Cited by 959
Self-citationA feedforward architecture accounts for rapid categorization
Serre, T., Oliva, A., & Poggio, T. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0700622104 · Cited by 918 · cited 9×
“HMAX Model (25”
BackgroundMethodInfluentialFast Readout of Object Identity from Macaque Inferior Temporal Cortex
Hung, C. P., Kreiman, G., Poggio, T., & DiCarlo, J. J. (2005) · Science · doi:10.1126/science.1117593 · Cited by 854 · cited 3×
“Moreover, we supplemented with serially sampled, single-electrode recording (6,7) and found that neuronal populations from arrays have very similar patterns of image encoding as assembled single-electrode unit populations.”
BackgroundMethodSelf-citationFreiwald, W. A., & Tsao, D. Y. (2010) · Science · doi:10.1126/science.1194908 · Cited by 678
Metamers of the ventral stream
Freeman, J., & Simoncelli, E. P. (2011) · Nature Neuroscience · doi:10.1038/nn.2889 · Cited by 667 · cited 3×
“This parameter space expresses an inclusive version of the hierarhical feedforward network concept and contains models similar to that used in many previous studies for different parameter values (19, 23, 24, 25).”
MethodRiesenhuber, M., & Poggio, T. (2000) · Nature Neuroscience · doi:10.1038/81479 · Cited by 658 · cited 1×
“S11, upper right: (i) local filter statistics, including filter mean and spread and (ii) the pooling exponents trading off between max-pooling and average-pooling (41).”
BackgroundDo We Know What the Early Visual System Does?
Carandini, M., Demb, J. B., Mante, V., Tolhurst, D. J., Dan, Y., Olshausen, B. A., Gallant, J. L., & Rust, N. C. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.3726-05.2005 · Cited by 600 · cited 3×
“As described in the main text, we used a standard methodology for assessing a model’s ability to predict individual sites (12, 13), in which each site is modeled as a linear combination of model outputs.”
BackgroundMethodResultPopulation coding of shape in area V4
Pasupathy, A., & Connor, C. E. (2002) · Nature Neuroscience · doi:10.1038/972 · Cited by 552
Why is Real-World Visual Object Recognition Hard?
Pinto, N., Cox, D. D., & DiCarlo, J. J. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0040027 · Cited by 537 · cited 3×
“This parameter space expresses an inclusive version of the hierarhical feedforward network concept and contains models similar to that used in many previous studies for different parameter values (19, 23, 24, 25).”
BackgroundMethodSelf-citationHow MT cells analyze the motion of visual patterns
Rust, N. C., Mante, V., Simoncelli, E. P., & Movshon, J. A. (2006) · Nature Neuroscience · doi:10.1038/nn1786 · Cited by 516 · cited 3×
“Models were drawn from a large parameter space of convolutional neural networks (CNNs) expressing an inclusive version of the hierarchical processing concept (17, 18, 20, 28).”
BackgroundMethodKiani, R., Esteky, H., Mirpour, K., & Tanaka, K. (2007) · Journal of Neurophysiology · doi:10.1152/jn.00024.2007 · Cited by 511 · cited 1×
“Although first principles-based models of higher ventral cortex have been proposed (15–20), these models fail to match important features of the higher ventral visual neural representation in both humans and macaques (4, 21).”
BackgroundDomain Specificity in Visual Cortex
Downing, P. E., Chan, A. W.-Y., Peelen, M. V., Dodds, C. M., & Kanwisher, N. (2006) · Cerebral Cortex · doi:10.1093/cercor/bhj086 · Cited by 509 · cited 1×
“Taken together, these results point to a computationally rigorous explanation for why heterogeneity is observed in the receptive fields of ventral stream neurons both at the unit and subarea levels (29, 38, 30, 42).”
BackgroundThe topography of high-order human object areas
Malach, R., Levy, I., & Hasson, U. (2002) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(02)01870-3 · Cited by 459 · cited 2×
“DiCarlo JJ, Zoccolan D, Rust NC (2012) How does the brain solve visual object recognition? Neuron 73(3):415–434.”
BackgroundNeural responses to polar, hyperbolic, and Cartesian gratings in area V4 of the macaque monkey
Gallant, J. L., Connor, C. E., Rakshit, S., Lewis, J. W., & Van Essen, D. C. (1996) · Journal of Neurophysiology · doi:10.1152/jn.1996.76.4.2718 · Cited by 443
Rust, N. C., & DiCarlo, J. J. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.0179-10.2010 · Cited by 387
Self-citationUnderlying principles of visual shape selectivity in posterior inferotemporal cortex
Brincat, S. L., & Connor, C. E. (2004) · Nature Neuroscience · doi:10.1038/nn1278 · Cited by 385 · cited 5×
“, it is a rule for producing output on any arbitrary stimulus set and does not explicitly rely on stimuli being in a particular subclass of images (10, 11).”
BackgroundMethodResultInfluentialObject Class Recognition and Localization Using Sparse Features with Limited Receptive Fields
Mutch, J., & Lowe, D. G. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0118-0 · Cited by 373
Pinto, N., Doukhan, D., DiCarlo, J. J., & Cox, D. D. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000579 · Cited by 248 · cited 4×
“(26), with one, two, or three convoluational layers.”
BackgroundMethodInfluentialSelf-citationA Model of V4 Shape Selectivity and Invariance
Cadieu, C., Kouh, M., Pasupathy, A., Connor, C. E., Riesenhuber, M., & Poggio, T. (2007) · Journal of Neurophysiology · doi:10.1152/jn.01265.2006 · Cited by 226
Self-citationTransformation of shape information in the ventral pathway
Connor, C. E., Brincat, S. L., & Pasupathy, A. (2007) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2007.03.002 · Cited by 179 · cited 7×
“For example, neurons in the lowest area, V1, are well described by Gabor-like edge detectors that extract rough object outlines (10), although the V1 population does not show robust tolerance to complex image transformations (9).”
BackgroundMethodResultInfluentialTheoretical Views of Boosting and Applications
Schapire, R. E. (1999) · Lecture Notes in Computer Science · doi:10.1007/3-540-46769-6_2 · Cited by 172 · cited 3×
“S2B), which correspond intuitively to architecturally specialized subregions like those observed in the ventral visual stream (13, 32).”
BackgroundVogels, R., & Orban, G. A. (1994) · Journal of Neurophysiology · doi:10.1152/jn.1994.71.4.1428 · Cited by 154 · cited 3×
“Tables (8) C A HMAX Model V1-like Model”
BackgroundMethodGeisler, W. S. (2003) · The MIT Press eBooks · doi:10.7551/mitpress/7131.003.0061 · Cited by 149
Trade-off between curvature tuning and position invariance in visual area V4
Sharpee, T. O., Kouh, M., & Reynolds, J. H. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1217479110 · Cited by 60 · cited 8×
“HMAX Model (25”
MethodResultInfluentialThe Neural Representation Benchmark and its Evaluation on Brain and MachinePreprint
Cadieu, C. F., Hong, H., Yamins, D., Pinto, N., Majaj, N. J., & DiCarlo, J. J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3530 · Cited by 31 · cited 1×
“We first measured IT neural responses on a benchmark testing image set that exposes key performance characteristics of visual representations (24).”
MethodSelf-citation
Cited by
Image Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025
“In light of the striking similarities between performanceoptimised artificial neural networks and biological vision [11, 31, 3, 19, 16], it is thus tempting to speculate that the human ability to abstract content from style – and therefore our ability to create and enjoy art – might also be primarily a preeminent sign…”
BackgroundA Neural Algorithm of Artistic Style
Gatys, L., Ecker, A., & Bethge, M. (2016) · Journal of Vision · doi:10.1167/16.12.326 · Cited by 2,612
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Building Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
Using goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
“A finer grained mapping of models to neurons is that of linear neural response predictivity of single units 33 .”
BackgroundMethodResultInfluentialSelf-citation
Metadata
- Authors
- Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J.
- Affiliations
- McGovern Institute for Brain Research (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2014-05-08
- Volume
- 111
- Issue
- 23
- Pages
- 8619-8624
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4060707
- OpenAlex
- W2058616551
- Cited by
- 1,924
- References
- 39
- Citations by year
- 2026: 157 · 2025: 215 · 2024: 206 · 2023: 196 · 2022: 199 · 2021: 209 · 2020: 180 · 2019: 174 · 2018: 127 · 2017: 111 · 2016: 92 · 2015: 41 · 2014: 17
- FWCI
- 48.54
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Funders
- National Science Foundation (0964269)
- National Eye Institute (F32EY022845, R01EY014970)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Keywords
- inferior temporal cortex, ventral stream, object recognition, neural response prediction, performance optimization, biologically plausible models, naturalistic images
- MeSH terms
- Algorithms, Animals, Humans, Macaca mulatta, Models, Neurological, Nerve Net, Photic Stimulation, Psychomotor Performance, Visual Cortex, Visual Pathways, Visual Perception, Neural Networks, Computer, Recognition, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.