Simple Learned Weighted Sums of Inferior Temporal Neuronal Firing Rates Accurately Predict Human Core Object Recognition Performance
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
To go beyond qualitative models of the biological substrate of object recognition, we ask: can a single ventral stream neuronal linking hypothesis quantitatively account for core object recognition performance over a broad range of tasks? We measured human performance in 64 object recognition tests using thousands of challenging images that explore shape similarity and identity preserving object variation. We then used multielectrode arrays to measure neuronal population responses to those same images in visual areas V4 and inferior temporal (IT) cortex of monkeys and simulated V1 population responses. We tested leading candidate linking hypotheses and control hypotheses, each postulating how ventral stream neuronal responses underlie object recognition behavior. Specifically, for each hypothesis, we computed the predicted performance on the 64 tests and compared it with the measured pattern of human performance. All tested hypotheses based on low- and mid-level visually evoked activity (pixels, V1, and V4) were very poor predictors of the human behavioral pattern. However, simple learned weighted sums of distributed average IT firing rates exactly predicted the behavioral pattern. More elaborate linking hypotheses relying on IT trial-by-trial correlational structure, finer IT temporal codes, or ones that strictly respect the known spatial substructures of IT (“face patches”) did not improve predictive power. Although these results do not reject those more elaborate hypotheses, they suggest a simple, sufficient quantitative model: each object recognition task is learned from the spatially distributed mean firing rates (100 ms) of ∼60,000 IT neurons and is executed as a simple weighted sum of those firing rates. SIGNIFICANCE STATEMENT We sought to go beyond qualitative models of visual object recognition and determine whether a single neuronal linking hypothesis can quantitatively account for core object recognition behavior. To achieve this, we designed a database of images for evaluating object recognition performance. We used multielectrode arrays to characterize hundreds of neurons in the visual ventral stream of nonhuman primates and measured the object recognition performance of >100 human observers. Remarkably, we found that simple learned weighted sums of firing rates of neurons in monkey inferior temporal (IT) cortex accurately predicted human performance. Although previous work led us to expect that IT would outperform V4, we were surprised by the quantitative precision with which simple IT-based linking hypotheses accounted for human behavior.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Distributed hierarchical processing in the primate cerebral cortex
Felleman, D. J., & Van Essen, D. C. (1991) · Cerebral Cortex · doi:10.1093/cercor/1.1.1 · Cited by 7,783
Dynamic predictions: Oscillations and synchrony in top–down processing
Engel, A. K., Fries, P., & Singer, W. (2001) · Nature Reviews Neuroscience · doi:10.1038/35094565 · Cited by 3,454 · cited 1×
“…tests might force us to turn to more complex underlying neural codes that were not necessary here (e.g., fine-timing or synchrony based codes; Engel et al., 2001) and might open the door for investigating a role for feedback in tasks that require inference (Kersten et al., 2004; Oliva and…”
BackgroundHierarchical models of object recognition in cortex
Riesenhuber, M., & Poggio, T. (1999) · Nature Neuroscience · doi:10.1038/14819 · Cited by 3,309 · cited 1×
“The bio-inspired SLFs are extensions of the C2 features from the HMAX model (Riesenhuber and Poggio, 1999; Serre et al., 2007; Mutch and Lowe, 2008).”
MethodUnsupervised spike detection and sorting with wavelets and superparamagnetic clustering
Quiroga, R. Q., Nadasdy, Z., & Ben-Shaul, Y. (2004) · Neural Computation · doi:10.1162/089976604774201631 · Cited by 2,119
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924 · cited 1×
“Ongoing work is systematically characterizing the nonlinear transformations from retina through V1, V2, and V4 (Pasupathy and Connor, 1999; Hegdé and Van Essen, 2000; Rust and DiCarlo, 2012; Freeman et al., 2013; Yamins et al., 2014).”
BackgroundSelf-citationHow does the brain solve visual object recognition?
DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012) · Neuron · doi:10.1016/j.neuron.2012.01.010 · Cited by 1,790
Self-citationNeural correlations, population coding and computation
Averbeck, B. B., Latham, P. E., & Pouget, A. (2006) · Nature reviews. Neuroscience · doi:10.1038/nrn1888 · Cited by 1,743 · cited 5×
“First, we considered the idea that trial-by-trial correlations in neuronal firing across the IT population might be important to consider when asking if a neuronal linking hypothesis is consistent with behavior (Zohary et al., 1994; Averbeck et al., 2006; Liu et al., 2013).”
BackgroundInfluentialObject perception as Bayesian inference
Kersten, D., Mamassian, P., & Yuille, A. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142005 · Cited by 1,269 · cited 1×
“…force us to turn to more complex underlying neural codes that were not necessary here (e.g., fine-timing or synchrony based codes; Engel et al., 2001) and might open the door for investigating a role for feedback in tasks that require inference (Kersten et al., 2004; Oliva and Torralba, 2006).”
BackgroundCorrelated neuronal discharge rate and its implications for psychophysical performance
Zohary, E., Shadlen, M. N., & Newsome, W. T. (1994) · Nature · doi:10.1038/370140a0 · Cited by 1,237 · cited 3×
“First, we considered the idea that trial-by-trial correlations in neuronal firing across the IT population might be important to consider when asking if a neuronal linking hypothesis is consistent with behavior (Zohary et al., 1994; Averbeck et al., 2006; Liu et al., 2013).”
BackgroundResultNeuronal correlates of a perceptual decision
Newsome, W. T., Britten, K. H., & Movshon, J. A. (1989) · Nature · doi:10.1038/341052a0 · Cited by 1,177
Categorical Representation of Visual Stimuli in the Primate Prefrontal Cortex
Freedman, D. J., Riesenhuber, M., Poggio, T., & Miller, E. K. (2001) · Science · doi:10.1126/science.291.5502.312 · Cited by 1,123
A relationship between behavioral choice and the visual responses of neurons in macaque MT
Britten, K. H., Newsome, W. T., Shadlen, M. N., Celebrini, S., & Movshon, J. A. (1996) · Visual Neuroscience · doi:10.1017/s095252380000715x · Cited by 1,049 · cited 1×
“Our comparison of nonhuman and human primates deviates from approaches that combine neural recording with behavioral testing in the same subjects (Britten et al., 1996; Luna et al., 2005; Cohen and Maunsell, 2011).”
BackgroundA feedforward architecture accounts for rapid categorization
Serre, T., Oliva, A., & Poggio, T. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0700622104 · Cited by 918
Fast Readout of Object Identity from Macaque Inferior Temporal Cortex
Hung, C. P., Kreiman, G., Poggio, T., & DiCarlo, J. J. (2005) · Science · doi:10.1126/science.1117593 · Cited by 854
Self-citationCadieu, C. F., Hong, H., Yamins, D. L. K., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., & DiCarlo, J. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003963 · Cited by 824 · cited 1×
“S2 in Cadieu et al., 2014).”
Self-citationKobatake, E., & Tanaka, K. (1994) · Journal of Neurophysiology · doi:10.1152/jn.1994.71.3.856 · Cited by 807
SENSE AND THE SINGLE NEURON: Probing the Physiology of Perception
Parker, A. J., & Newsome, W. T. (1998) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.21.1.227 · Cited by 801
Spatial attention decorrelates intrinsic activity fluctuations in macaque area V4
Mitchell, J. F., Sundberg, K. A., & Reynolds, J. H. (2009) · Neuron · doi:10.1016/j.neuron.2009.09.013 · Cited by 710
Population coding of shape in area V4
Pasupathy, A., & Connor, C. E. (2002) · Nature Neuroscience · doi:10.1038/972 · Cited by 552 · cited 2×
“Previous ventral stream studies have focused on understanding the nonlinear transformations between the retina and neural responses (Gallant et al., 1996; Hegdé and Van Essen, 2000; Pasupathy and Connor, 2002; Connor et al., 2007; Freeman et al., 2013), including evidence that IT is better at recognition than early re…”
BackgroundMicrostimulation of inferotemporal cortex influences face categorization
Afraz, S.-R., Kiani, R., & Esteky, H. (2006) · Nature · doi:10.1038/nature04982 · Cited by 363
Chestek, C. A., Gilja, V., Nuyujukian, P., Foster, J. D., Fan, J. M., Kaufman, M. T., Churchland, M. M., Rivera-Alvidrez, Z., Cunningham, J. P., Ryu, S. I., et al. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/4/045005 · Cited by 350
Tactile roughness: neural codes that account for psychophysical magnitude estimates
Connor, C., Hsiao, S., Phillips, Jr., & Johnson, K. (1990) · The Journal of Neuroscience · doi:10.1523/jneurosci.10-12-03823.1990 · Cited by 317 · cited 4×
“Quantifying the goodness of a linking hypothesis: consistency Following previous work (Hyvarinen et al., 1968; Newsome et al., 1989; Connor et al., 1990), we reasoned that the neuronal linking hypotheses that are the most likely to correspond to the mechanisms at work in the brain are those that…”
BackgroundInfluential
Cited by
Using goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
Self-citationThe Code for Facial Identity in the Primate Brain
Chang, L., & Tsao, D. Y. (2017) · Cell · doi:10.1016/j.cell.2017.05.011 · Cited by 644
“This process is thought to happen in IT cortex, where neurons carry information about high-level object identity, with invariance to various transformations that do not affect identity (Brincat and Connor, 2004; Ito et al., 1995; Majaj et al., 2015).”
BackgroundThe neural architecture of language: Integrative modeling converges on predictive processing
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2105646118 · Cited by 517
Kar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0392-5 · Cited by 496
Self-citationBrain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, J., Geiger, F., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/407007 · Cited by 458
Self-citation
Metadata
- Authors
- Majaj, N. J., Hong, H., Solomon, E. A., & DiCarlo, J. J.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2015-09-30
- Volume
- 35
- Issue
- 39
- Pages
- 13402-13418
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W2121008432
- Cited by
- 204
- References
- 23
- FWCI
- 5.53
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.