Computational mechanisms underlying cortical responses to the affordance properties of visual scenes
Open accessPublished version of a preprint
Abstract
ABSTRACT Biologically inspired deep convolutional neural networks (CNNs), trained for computer vision tasks, have been found to predict cortical responses with remarkable accuracy. However, the complex internal operations of these models remain poorly understood, and the factors that account for their success are unknown. Here we developed a set of techniques for using CNNs to gain insights into the computational mechanisms underlying cortical responses. We focused on responses in the occipital place area (OPA), a scene-selective region of dorsal occipitoparietal cortex. In a previous study, we showed that fMRI activation patterns in the OPA contain information about the navigational affordances of scenes: that is, information about where one can and cannot move within the immediate environment. We hypothesized that this affordance information could be extracted using a set of purely feedforward computations. To test this idea, we examined a deep CNN with a feedforward architecture that had been previously trained for scene classification. We found that the CNN was highly predictive of OPA representations, and, importantly, that it accounted for the portion of OPA variance that reflected the navigational affordances of scenes. The CNN could thus serve as an image-computable candidate model of affordance-related responses in the OPA. We then ran a series of in silico experiments on this model to gain insights into its internal computations. These analyses showed that the computation of affordance-related features relied heavily on visual information at high-spatial frequencies and cardinal orientations, both of which have previously been identified as low-level stimulus preferences of scene-selective visual cortex. These computations also exhibited a strong preference for information in the lower visual field, which is consistent with known retinotopic biases in the OPA. Visualizations of feature selectivity within the CNN suggested that affordance-based responses encoded features that define the layout of the spatial environment, such as boundary-defining junctions and large extended surfaces. Together, these results map the sensory functions of the OPA onto a fully quantitative model that provides insights into its visual computations. More broadly, they advance integrative techniques for understanding visual cortex across multiple level of analysis: from the identification of cortical sensory functions to the modeling of their underlying algorithmic implementations. AUTHOR SUMMARY How does visual cortex compute behaviorally relevant properties of the local environment from sensory inputs? For decades, computational models have been able to explain only the earliest stages of biological vision, but recent advances in the engineering of deep neural networks have yielded a breakthrough in the modeling of high-level visual cortex. However, these models are not explicitly designed for testing neurobiological theories, and, like the brain itself, their complex internal operations remain poorly understood. Here we examined a deep neural network for insights into the cortical representation of the navigational affordances of visual scenes. In doing so, we developed a set of high-throughput techniques and statistical tools that are broadly useful for relating the internal operations of neural networks with the information processes of the brain. Our findings demonstrate that a deep neural network with purely feedforward computations can account for the processing of navigational layout in high-level visual cortex. We next performed a series of experiments and visualization analyses on this neural network, which characterized a set of stimulus input features that may be critical for computing navigationally related cortical representations and identified a set of high-level, complex scene features that may serve as a basis set for the cortical coding of navigational layout. These findings suggest a computational mechanism through which high-level visual cortex might encode the spatial structure of the local navigational environment, and they demonstrate an experimental approach for leveraging the power of deep neural networks to understand the visual computations of the brain.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Viewshed analysis (which uses 2D polygons known as isovists to represent the visible area from a given vantage point) has been used to characterize local scenes.
“For example, viewshed analysis (which uses 2D polygons known as isovists to represent the visible area from a given vantage point) have been used to characterize local scenes 186 .”
From From cognitive maps to spatial schemas (2023) · cites this paper as [186]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
Marr, D. (1982) · Cumulative Index of Computer Aided Architectural Design · Cited by 5,733 · see citing works
Recognition-by-components: A theory of human image understanding.
Biederman, I. (1987) · Psychological Review · doi:10.1037/0033-295x.94.2.115 · Cited by 5,621 · see citing works
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451 · see citing works
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503 · see citing works
Hierarchical models of object recognition in cortex
Riesenhuber, M., & Poggio, T. (1999) · Nature Neuroscience · doi:10.1038/14819 · Cited by 3,309 · see citing works
A cortical representation of the local visual environment
Epstein, R. A., & Kanwisher, N. (1998) · Nature · doi:10.1038/33402 · Cited by 2,960 · see citing works
Self-citation(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534 · see citing works
The ecodist Package for Dissimilarity-based Analysis of Ecological Data
Goslee, S., & Urban, D. L. (2007) · Journal of Statistical Software · doi:10.18637/jss.v022.i07 · Cited by 2,495 · see citing works
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924 · see citing works
How does the brain solve visual object recognition?
DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012) · Neuron · doi:10.1016/j.neuron.2012.01.010 · Cited by 1,790 · see citing works
Understanding Neural Networks Through Deep VisualizationPreprint
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,687 · see citing works
Using goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621 · see citing works
Deep supervised, but not unsupervised, models may explain IT cortical representation
Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003915 · Cited by 1,204 · see citing works
Kriegeskorte, N. (2015) · Annual Review of Vision Science · doi:10.1146/annurev-vision-082114-035447 · Cited by 1,093 · see citing works
Robust smoothing of gridded data in one and higher dimensions with missing values
Garcia, D. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2009.09.020 · Cited by 1,022 · see citing works
Guclu, U., & van Gerven, M. A. J. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.5023-14.2015 · Cited by 961 · see citing works
A feedforward architecture accounts for rapid categorization
Serre, T., Oliva, A., & Poggio, T. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0700622104 · Cited by 918 · see citing works
A Toolbox for Representational Similarity Analysis
Nili, H., Wingfield, C., Walther, A., Su, L., Marslen‐Wilson, W. D., & Kriegeskorte, N. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003553 · Cited by 914 · see citing works
Representational geometry: integrating cognition, computation, and the brain
Kriegeskorte, N., & Kievit, R. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.06.007 · Cited by 898 · see citing works
Object Detectors Emerge in Deep Scene CNNsPreprint
Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., & Torralba, A. B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6856 · Cited by 823 · see citing works
VISUALLY CONTROLLED LOCOMOTION AND VISUAL ORIENTATION IN ANIMALS*
Gibson, J. J. (1958) · British Journal of Psychology · doi:10.1111/j.2044-8295.1958.tb00656.x · Cited by 819 · see citing works
Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., & Oliva, A. (2016) · Scientific Reports · doi:10.1038/srep27755 · Cited by 716 · see citing works
The retrosplenial contribution to human navigation: a review of lesion and neuroimaging findings
Maguire, E. (2001) · Scandinavian Journal of Psychology · doi:10.1111/1467-9450.00233 · Cited by 584 · see citing works
The neural basis of object perception
Kalanit Grill-spector, A. (2003) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(03)00040-0 · Cited by 562 · see citing works
Greene, M. R., & Oliva, A. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02316.x · Cited by 483 · see citing works
GREENE, M., & OLIVA, A. (2009) · Cognitive Psychology · doi:10.1016/j.cogpsych.2008.06.001 · Cited by 467 · see citing works
Understanding the Results of Multiple Linear Regression
Nimon, K., & Oswald, F. L. (2013) · Organizational Research Methods · doi:10.1177/1094428113493929 · Cited by 352 · see citing works
An algorithmic method for functionally defining regions of interest in the ventral visual pathway
Julian, J. B., Fedorenko, E. G., Webster, J. M., & Kanwisher, N. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.02.055 · Cited by 334 · see citing works
Simple line drawings suffice for functional MRI decoding of natural scene categories
Walther, D. B., Chai, B., Caddigan, E., Beck, D. M., & Fei-Fei, L. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1015666108 · Cited by 235 · see citing works
Groen, I. I. A., Silson, E. H., & Baker, C. I. (2017) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2016.0102 · Cited by 222 · see citing works
Heeger, D. J. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1619788114 · Cited by 220 · see citing works
Neural Correlates for Perception of 3D Surface Orientation from Texture Gradient
Tsutsui, K., Sakata, H., Naganuma, T., & Taira, M. (2002) · Science · doi:10.1126/science.1074128 · Cited by 207 · see citing works
Coding of navigational affordances in the human visual system
Bonner, M., & Epstein, R. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618228114 · Cited by 198 · see citing works
Self-citationUnique semantic space in the brain of each beholder predicts perceived similarity
Charest, I., Kievit, R., Schmitz, T. W., Deca, D., & Kriegeskorte, N. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1402594111 · Cited by 194 · see citing works
Rajimehr, R., Devaney, K. J., Bilenko, N. Y., Young, J. C., & Tootell, R. B. H. (2011) · PLoS Biology · doi:10.1371/journal.pbio.1000608 · Cited by 189 · see citing works
From circuits to behavior: a bridge too far?
Carandini, M. (2012) · Nature Neuroscience · doi:10.1038/nn.3043 · Cited by 187 · see citing works
Cichy, R. M., Khosla, A., Pantazis, D., & Oliva, A. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.03.063 · Cited by 179 · see citing works
Task context impacts visual object processing differentially across the cortex
Harel, A., Kravitz, D. J., & Baker, C. I. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1312567111 · Cited by 177 · see citing works
Groen, I. I. A., Greene, M. R., Baldassano, C. A., Fei-Fei, L., Beck, D. M., & Baker, C. I. (2018) · eLife · doi:10.7554/elife.32962 · Cited by 170 · see citing works
Neural pattern similarity reveals the inherent intersection of social categories
Stolier, R. M., & Freeman, J. B. (2016) · Nature Neuroscience · doi:10.1038/nn.4296 · Cited by 166 · see citing works
Thinking Outside the Box: Rectilinear Shapes Selectively Activate Scene-Selective Cortex
Nasr, S., Echavarria, C. E., & Tootell, R. B. H. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.4802-13.2014 · Cited by 163 · see citing works
Silson, E. H., Chan, A. W.-Y., Reynolds, R. C., Kravitz, D. J., & Baker, C. I. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.0137-15.2015 · Cited by 156 · see citing works
Visual Scene Processing in Familiar and Unfamiliar Environments
Epstein, R. A., Higgins, J. S., Jablonski, K., & Feiler, A. M. (2007) · Journal of Neurophysiology · doi:10.1152/jn.00003.2007 · Cited by 153 · see citing works
Self-citationObject Ensemble Processing in Human Anterior-Medial Ventral Visual Cortex
Cant, J. S., & Yaoda Xu, A. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.3325-11.2012 · Cited by 133 · see citing works
A Network for Scene Processing in the Macaque Temporal Lobe
Kornblith, S., Cheng, X., Ohayon, S., & Tsao, D. Y. (2013) · Neuron · doi:10.1016/j.neuron.2013.06.015 · Cited by 131 · see citing works
Outside Looking In: Landmark Generalization in the Human Navigational System
Marchette, S. A., Vass, L. K., Ryan, J., & Epstein, R. A. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.2270-15.2015 · Cited by 123 · see citing works
Self-citationA Cardinal Orientation Bias in Scene-Selective Visual Cortex
Nasr, S., & Tootell, R. B. H. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.2036-12.2012 · Cited by 114 · see citing works
Pixels to Voxels: Modeling Visual Representation in the Human BrainPreprint
Agrawal, P., Stansbury, D. E., Malik, J., & Gallant, J. L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.5104 · Cited by 91 · see citing works
Principles for models of neural information processing
Kay, K., & Weiner, K. S. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.08.016 · Cited by 84 · see citing works
Nonaccidental Properties Underlie Human Categorization of Complex Natural Scenes
Walther, D. B., & Shen, D. (2014) · Psychological Science · doi:10.1177/0956797613512662 · Cited by 82 · see citing works
Understanding mid-level representations in visual processing
Peirce, J. W. (2015) · Journal of Vision · doi:10.1167/15.7.5 · Cited by 71 · see citing works
Silson, E. H., Groen, I. I. A., Kravitz, D. J., & Baker, C. I. (2016) · Journal of Vision · doi:10.1167/16.6.14 · Cited by 68 · see citing works
Separating the chaff from the wheat: possible origins of the oblique effect
Keil, M. S., & Cristóbal, G. (2000) · Journal of the Optical Society of America A · doi:10.1364/josaa.17.000697 · Cited by 64 · see citing works
Read, J. C. A., Georgiou, R., Brash, C., Yazdani, P., Whittaker, R. G., Trevelyan, A. J., & Serrano‐Pedraza, I. (2015) · Journal of Vision · doi:10.1167/15.1.5 · Cited by 55 · see citing works
Choo, H., & Walther, D. B. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.04.021 · Cited by 50 · see citing works
Measurement of crosstalk in stereoscopic display systems used for vision research
Baker, D. H., Kaestner, M., & Gouws, A. D. (2016) · Journal of Vision · doi:10.1167/16.15.14 · Cited by 45 · see citing works
Neural Systems for Visual Scene Recognition
Epstein, R. A. (2014) · The MIT Press eBooks · doi:10.7551/mitpress/9780262027854.003.0006 · Cited by 43 · see citing works
Self-citationScene content is predominantly conveyed by high spatial frequencies in scene-selective visual cortex
Berman, D., Golomb, J. D., & Walther, D. B. (2017) · PLoS ONE · doi:10.1371/journal.pone.0189828 · Cited by 42 · see citing works
Computational and Cognitive Neuroscience of Vision
Zhao, Q. (2016) · Cognitive science and technology · doi:10.1007/978-981-10-0213-7 · Cited by 39 · see citing works
Bryan, P. B., Julian, J. B., & Epstein, R. A. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00137 · Cited by 35 · see citing works
Self-citationTowards a Theory of Computation in the Visual Cortex
Mély, D. A., & Serre, T. (2016) · Cognitive science and technology · doi:10.1007/978-981-10-0213-7_4 · Cited by 22 · see citing works
A model of surface depth and orientation predicts BOLD responses in human scene-selective cortex
Lescroart, M. D., & Gallant, J. L. (2015) · Journal of Vision · doi:10.1167/15.12.573 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Performance vs. competence in human–machine comparisons
Firestone, C. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1905334117 · Cited by 150 · see citing works
From cognitive maps to spatial schemas
Farzanfar, D., Spiers, H. J., Moscovitch, M., & Rosenbaum, R. S. (2023) · Nature Reviews Neuroscience · doi:10.1038/s41583-022-00655-9 · Cited by 115 · see citing works
Unveiling functions of the visual cortex using task-specific deep neural networks
Dwivedi, K., Bonner, M. F., Cichy, R. M., & Roig, G. (2021) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009267 · Cited by 52 · see citing works
Self-citationEmergence of Visual Center-Periphery Spatial Organization in Deep Convolutional Neural Networks
Mohsenzadeh, Y., Mullin, C., Lahner, B., & Oliva, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-61409-0 · Cited by 31 · see citing works
- Authors
- Bonner, M. F., & Epstein, R. A.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Bonner, Michael F.
- Epstein, Russell A.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-04-23
- Volume
- 14
- Issue
- 4
- Pages
- e1006111
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC5933806
- OpenAlex
- W2951209071
- Cited by
- 94
- References
- 67
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 6 · 2023: 13 · 2022: 6 · 2021: 15 · 2020: 12 · 2019: 21 · 2018: 8
- FWCI
- 8.34
- Type
- journal article
- Language
- en
- Update notices
- new_version (2018-05-03)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- National Eye Institute (EY022350, EY027047)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- occipital place area, deep convolutional neural networks, high spatial frequency, cardinal orientation, visual field
- MeSH terms
- Algorithms, Artificial Intelligence, Brain Mapping, Humans, Magnetic Resonance Imaging, Models, Neurological, Occipital Lobe, Photic Stimulation, Visual Cortex, Visual Fields, Visual Perception, Neural Networks, Computer, Computational Biology, Spatial Navigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
