Unveiling functions of the visual cortex using task-specific deep neural networks
Open accessPublished version of a preprint
The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Abstract
The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Using this approach, known as neuroconnectionism, numerous studies have identified shared representational structure across brains and ANN models, as well as factors influencing this alignment [5–11]”
From The Umwelt Representation Hypothesis: rethinking Universality (2026) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Marr, D. (1982) · Cumulative Index of Computer Aided Architectural Design · Cited by 5,733
Representational similarity analysis-connecting the branches of systems neuroscience
Kriegeskorte, N. (2008) · Frontiers in Systems Neuroscience · doi:10.3389/neuro.06.004.2008 · Cited by 3,355
Information-based functional brain mapping
Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0600244103 · Cited by 2,113 · cited 1×
“To generate a functional map across the whole visual cortex we performed a searchlight analysis [11,12].”
MethodPerformance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Using goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
The lateral occipital complex and its role in object recognition
Grill-Spector, K., Kourtzi, Z., & Kanwisher, N. (2001) · Vision Research · doi:10.1016/s0042-6989(01)00073-6 · Cited by 1,406
Domain-Specific Knowledge Systems in the Brain: The Animate-Inanimate Distinction
Caramazza, A., & Shelton, J. R. (1998) · Journal of Cognitive Neuroscience · doi:10.1162/089892998563752 · Cited by 1,348
Deep supervised, but not unsupervised, models may explain IT cortical representation
Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003915 · Cited by 1,204
Grill-Spector, K., & Malach, R. (2004) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.27.070203.144220 · Cited by 1,184 · cited 1×
“Each of these regions has been stipulated to fulfill a distinct sub-function in enabling perception [2].”
BackgroundTaskonomy: Disentangling Task Transfer Learning
Zamir, A. R., Sax, A., Shen, W., Guibas, L., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00391 · Cited by 1,128
A deep learning framework for neuroscience
Richards, B. A., Lillicrap, T. P., Beaudoin, P., Bengio, Y., Bogacz, R., Christensen, A., Clopath, C., Costa, R. P., de Berker, A., Ganguli, S., et al. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0520-2 · Cited by 1,027
Guclu, U., & van Gerven, M. A. J. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.5023-14.2015 · Cited by 961
Lillicrap, T. P., Santoro, A., Marris, L., Akerman, C. J., & Hinton, G. (2020) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-0277-3 · Cited by 908
The functional architecture of the ventral temporal cortex and its role in categorization
Grill-Spector, K., & Weiner, K. S. (2014) · Nature Reviews Neuroscience · doi:10.1038/nrn3747 · Cited by 837
Cadieu, C. F., Hong, H., Yamins, D. L. K., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., & DiCarlo, J. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003963 · Cited by 824
Mishkin, M., & Ungerleider, L. G. (1982) · Behavioural Brain Research · doi:10.1016/0166-4328(82)90081-x · Cited by 810
Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., & Oliva, A. (2016) · Scientific Reports · doi:10.1038/srep27755 · Cited by 716
Self-citationProbabilistic Maps of Visual Topography in Human Cortex
Wang, L., Mruczek, R. E. B., Arcaro, M. J., & Kastner, S. (2015) · Cerebral Cortex · doi:10.1093/cercor/bhu277 · Cited by 712
How does nature program neuron types?
Borst, A. (2008) · Frontiers in Neuroscience · doi:10.3389/neuro.01.016.2008 · Cited by 705
Reading Hidden Intentions in the Human Brain
Haynes, J.-D., Sakai, K., Rees, G., Gilbert, S., Frith, C., & Passingham, R. E. (2007) · Current Biology · doi:10.1016/j.cub.2006.11.072 · Cited by 702
Kokkinos, I. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.579 · Cited by 657
A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F., Charest, I., et al. (2022) · Nature Neuroscience · doi:10.1038/s41593-021-00962-x · Cited by 577
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
Lindsay, G. W. (2021) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01544 · Cited by 557
Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.044 · Cited by 544
Legendre, P. (2008) · Journal of Plant Ecology · doi:10.1093/jpe/rtm001 · Cited by 501
Kar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0392-5 · Cited by 496
Deep Neural Networks as Scientific Models
Cichy, R. M., & Kaiser, D. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.01.009 · Cited by 454
Self-citationNeural population control via deep image synthesis
Bashivan, P., Kar, K., & DiCarlo, J. J. (2019) · Science · doi:10.1126/science.aav9436 · Cited by 434
Making big data open: data sharing in neuroimaging
Poldrack, R. A., & Gorgolewski, K. J. (2014) · Nature Neuroscience · doi:10.1038/nn.3818 · Cited by 425
Theories of Error Back-Propagation in the Brain
Whittington, J. C. R., & Bogacz, R. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.12.005 · Cited by 420
Recurrence is required to capture the representational dynamics of the human visual system
Kietzmann, T. C., Spoerer, C. J., Sörensen, L. K. A., Cichy, R. M., Hauk, O., & Kriegeskorte, N. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1905544116 · Cited by 406
Self-citationNeuronal basis of contrast discrimination
Boynton, G. M., Demb, J. B., Glover, G. H., & Heeger, D. J. (1999) · Vision Research · doi:10.1016/s0042-6989(98)00113-8 · Cited by 391
Neuronal correlates of perception in early visual cortex
Ress, D., & Heeger, D. J. (2003) · Nature Neuroscience · doi:10.1038/nn1024 · Cited by 353
Retinotopic Organization of Human Ventral Visual Cortex
Arcaro, M. J., McMains, S. A., Singer, B. D., & Kastner, S. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2807-09.2009 · Cited by 348
Scene Perception in the Human Brain
Epstein, R. A., & Baker, C. I. (2019) · Annual Review of Vision Science · doi:10.1146/annurev-vision-091718-014809 · Cited by 347
Human Cortical Activity Correlates With Stereoscopic Depth Perception
Backus, B. T., Fleet, D. J., Parker, A. J., & Heeger, D. J. (2001) · Journal of Neurophysiology · doi:10.1152/jn.2001.86.4.2054 · Cited by 294
Control of synaptic plasticity in deep cortical networks
Roelfsema, P. R., & Holtmaat, A. (2018) · Nature Reviews Neuroscience · doi:10.1038/nrn.2018.6 · Cited by 246
Ponce, C. R., Xiao, W., Schade, P. F., Hartmann, T. S., Kreiman, G., & Livingstone, M. S. (2019) · Cell · doi:10.1016/j.cell.2019.04.005 · Cited by 245
Contrast Sensitivity in Human Visual Areas and Its Relationship to Object Recognition
Avidan, G., Harel, M., Hendler, T., Ben-Bashat, D., Zohary, E., & Malach, R. (2002) · Journal of Neurophysiology · doi:10.1152/jn.2002.87.6.3102 · Cited by 242
Searchlight analysis: Promise, pitfalls, and potential
Etzel, J. A., Zacks, J. M., & Braver, T. S. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.03.041 · Cited by 236
Integrative benchmarking to advance neurally mechanistic models of human intelligence
Schrimpf, M., Kubilius, J., Lee, M. J., Ratan Murty, N. A., Ajemian, R., & DiCarlo, J. J. (2020) · Neuron · doi:10.1016/j.neuron.2020.07.040 · Cited by 233
Representation of the Perceived 3-D Object Shape in the Human Lateral Occipital Complex
Kourtzi, Z. (2003) · Cerebral Cortex · doi:10.1093/cercor/13.9.911 · Cited by 219
Crichton, M. L., Muller, D. A., Depelsenaire, A. C. I., Pearson, F. E., Wei, J., Coffey, J., Zhang, J., Fernando, G. J. P., & Kendall, M. A. F. (2016) · Scientific Reports · doi:10.1038/srep28722 · Cited by 209
Coding of navigational affordances in the human visual system
Bonner, M. F., & Epstein, R. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618228114 · Cited by 204
Self-citationStanley, D. A., & Rubin, N. (2003) · Neuron · doi:10.1016/s0896-6273(02)01148-0 · Cited by 201
Behrmann, M., & Moscovitch, M. (1994) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1994.6.1.1 · Cited by 185
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
Batra, D., Gokaslan, A., Kembhavi, A., Maksymets, O., Mottaghi, R., Savva, M., Toshev, A., & Wijmans, E. (2020) · arXiv · doi:10.48550/arxiv.2006.13171 · Cited by 177
Groen, I. I., Greene, M. R., Baldassano, C., Fei-Fei, L., Beck, D. M., & Baker, C. I. (2018) · eLife · doi:10.7554/elife.32962 · Cited by 169
Güçlü, U., & van Gerven, M. A. J. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.12.036 · Cited by 148
Task-Driven Convolutional Recurrent Models of the Visual System
Nayebi, A., Bear, D. M., Kubilius, J., Kar, K., Ganguli, S., Sussillo, D., DiCarlo, J. J., & Yamins, D. L. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00053 · Cited by 120
Human Scene-Selective Areas Represent 3D Configurations of Surfaces
Lescroart, M. D., & Gallant, J. L. (2019) · Neuron · doi:10.1016/j.neuron.2018.11.004 · Cited by 116
Representation Similarity Analysis for Efficient Task Taxonomy & Transfer Learning
Dwivedi, K., & Roig, G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01267 · Cited by 101 · cited 2×
“We have observed that DNNs in the same task group show a higher correlation with each other as compared to DNNs in other task groups [14].”
MethodResultSelf-citationComputational mechanisms underlying cortical responses to the affordance properties of visual scenes
Bonner, M. F., & Epstein, R. A. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006111 · Cited by 94
Self-citationCognitive Ontology and Region- versus Network-Oriented Analyses
Klein, C. (2012) · Philosophy of Science · doi:10.1086/667843 · Cited by 86 · cited 1×
“This indicates a many-to-one mapping [48] between functions and brain regions.”
Background(2021) · doi:10.1162/jocn_a_01755 · Cited by 85
Neural Response to Perception of Volume in the Lateral Occipital Complex
Moore, C., & Engel, S. A. (2001) · Neuron · doi:10.1016/s0896-6273(01)00197-0 · Cited by 84
Legendre, P. (2007) · Chinese Journal of Plant Ecology · doi:10.17521/cjpe.2007.0124 · Cited by 77
Khaligh-Razavi, S.-M., Henriksson, L., Kay, K., & Kriegeskorte, N. (2017) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2016.10.007 · Cited by 70
BrainIAK tutorials: User-friendly learning materials for advanced fMRI analysis
Kumar, M., Ellis, C. T., Lu, Q., Zhang, H., Capotă, M., Willke, T. L., Ramadge, P. J., Turk‐Browne, N. B., & Norman, K. A. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007549 · Cited by 70
AllenAct: A Framework for Embodied AI Research
Weihs, L., Salvador, J., Kotar, K., Jain, U., Zeng, K.-H., Mottaghi, R., & Kembhavi, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12760 · Cited by 39
End-to-end neural system identification with neural information flow
Seeliger, K., Ambrogioni, L., Güçlütürk, Y., van den Bulk, L. M., Güçlü, U., & van Gerven, M. A. J. (2021) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1008558 · Cited by 38
A massive 7T fMRI dataset to bridge cognitive and computational neurosciencePreprint
Allen, E., St-Yves, G., Wu, Y., et al. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.02.22.432340 · Cited by 37
NeuroGen: Activation optimized image synthesis for discovery neuroscience
Gu, Z., Jamison, K. W., Khosla, M., Allen, E., Wu, Y., St-Yves, G., Naselaris, T., Kay, K., Sabuncu, M. R., & Kuceyeski, A. F. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118812 · Cited by 36
Visual pathways from the perspective of cost functions and multi-task deep neural networks
Scholte, H. S., Losch, M., Ramakrishnan, K., de Haan, E. H. F., & Bohté, S. M. (2017) · Cortex · doi:10.1016/j.cortex.2017.09.019 · Cited by 31
Neural Taskonomy: Inferring the Similarity of Task-Derived Representations from Brain Activity
Wang, A. Y., Wehbe, L., & Tarr, M. J. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/708016 · Cited by 27
Diverse deep neural networks all predict human IT well, after training and fitting
Storrs, K. R., Kietzmann, T. C., Walther, A., Mehrer, J., & Kriegeskorte, N. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.05.07.082743 · Cited by 25
Unraveling Representations in Scene-selective Brain Regions Using Scene-Parsing Deep Neural Networks
Dwivedi, K., Cichy, R. M., & Roig, G. (2020) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01624 · Cited by 23
Self-citationLearning Generalizable Visual Representations via Interactive Gameplay
Weihs, L., Kembhavi, A., Ehsani, K., Pratt, S. M., Han, W., Herrasti, A., Kolve, E., Schwenk, D., Mottaghi, R., & Farhadi, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.08195 · Cited by 11
Cited by
The neuroconnectionist research programme
Doerig, A., Sommers, R. P., Seeliger, K., Richards, B., Ismael, J., Lindsay, G. W., Kording, K. P., Konkle, T., van Gerven, M. A. J., Kriegeskorte, N., et al. (2023) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00705-w · Cited by 233
Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., & Konkle, T. (2022) · doi:10.1101/2022.03.28.485868 · Cited by 160
“…has been significant interest in the Taskonomy models for controlled model comparisons (including by us; Conwell et al., 2021; Wang et al., 2019; Dwivedi et al., 2021; Cadena et al., 2022), a direct implication of these results is that these models should not be used in future research to make…”
BackgroundResultConwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., & Konkle, T. (2024) · Nature Communications · doi:10.1038/s41467-024-53147-y · Cited by 100
Understanding Human Object Vision: A Picture Is Worth a Thousand Representations
Bracci, S., & de Beeck, H. O. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-032720-041031 · Cited by 125
High-performing neural network models of visual cortex benefit from high latent dimensionality
Elmoznino, E., & Bonner, M. F. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011792 · Cited by 49
Self-citation
Metadata
- Authors
- Dwivedi, K., Bonner, M. F., Cichy, R. M., & Roig, G.
- Affiliations
- Goethe University Frankfurt (DE)
- Johns Hopkins University (US)
- Freie Universität Berlin (DE)
- Corresponding author
- Dwivedi, Kshitij
- Roig, Gemma
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-08-13
- Volume
- 17
- Issue
- 8
- Pages
- e1009267
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC8407579
- OpenAlex
- W3192643265
- Cited by
- 52
- References
- 70
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 16 · 2023: 11 · 2022: 8 · 2021: 2
- FWCI
- 4.86
- Type
- journal article
- Language
- en
- Update notices
- new_version (2021-08-31)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) (submittedVersion, cc-by)
- Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) (submittedVersion, cc-by-nc)
- Refubium (Universitätsbibliothek der Freien Universität Berlin) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Universitätsbibliothek der FU Berlin Hochschulschriftenstelle u. Dokumentenserver (cc-by)
- Funders
- Alfons und Gertrud Kassel-Stiftung
- dfg (CI241/1-1, CI241/3-1)
- European Research Council (ERC-2018-StG 803370)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Keywords
- visual cortex, functional mapping, deep neural network, ventral stream, dorsal stream, scene perception, 3D scene perception, semantic processing, hierarchical visual processing, functional MRI, computational psychiatry
- MeSH terms
- Deep Learning, Adult, Brain Mapping, Female, Humans, Magnetic Resonance Imaging, Male, Models, Neurological, Photic Stimulation, Semantics, Task Performance and Analysis, Visual Cortex, Visual Perception, Neural Networks, Computer, Computational Biology, Imaging, Three-Dimensional, Functional Neuroimaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.