Unveiling functions of the visual cortex using task-specific deep neural networks
Open accessPublished version of a preprint
The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Abstract
The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Using this approach, known as neuroconnectionism, numerous studies have identified shared representational structure across brains and ANN models, as well as factors influencing this alignment [5–11]”
From The Umwelt Representation Hypothesis: rethinking Universality (2026) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Marr, D. (1982) · Cumulative Index of Computer Aided Architectural Design · Cited by 5,733
Representational similarity analysis-connecting the branches of systems neuroscience
Kriegeskorte, N. (2008) · Frontiers in Systems Neuroscience · doi:10.3389/neuro.06.004.2008 · Cited by 3,355
Information-based functional brain mapping
Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0600244103 · Cited by 2,113 · cited 1×
“To generate a functional map across the whole visual cortex we performed a searchlight analysis [11,12].”
MethodPerformance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Using goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
The lateral occipital complex and its role in object recognition
Grill-Spector, K., Kourtzi, Z., & Kanwisher, N. (2001) · Vision Research · doi:10.1016/s0042-6989(01)00073-6 · Cited by 1,406
Domain-Specific Knowledge Systems in the Brain: The Animate-Inanimate Distinction
Caramazza, A., & Shelton, J. R. (1998) · Journal of Cognitive Neuroscience · doi:10.1162/089892998563752 · Cited by 1,348
Cited by
The neuroconnectionist research programme
Doerig, A., Sommers, R. P., Seeliger, K., Richards, B., Ismael, J., Lindsay, G. W., Kording, K. P., Konkle, T., van Gerven, M. A. J., Kriegeskorte, N., et al. (2023) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00705-w · Cited by 233
Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., & Konkle, T. (2022) · doi:10.1101/2022.03.28.485868 · Cited by 160
“…has been significant interest in the Taskonomy models for controlled model comparisons (including by us; Conwell et al., 2021; Wang et al., 2019; Dwivedi et al., 2021; Cadena et al., 2022), a direct implication of these results is that these models should not be used in future research to make…”
BackgroundResultConwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., & Konkle, T. (2024) · Nature Communications · doi:10.1038/s41467-024-53147-y · Cited by 100
Understanding Human Object Vision: A Picture Is Worth a Thousand Representations
Bracci, S., & de Beeck, H. O. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-032720-041031 · Cited by 125
High-performing neural network models of visual cortex benefit from high latent dimensionality
Elmoznino, E., & Bonner, M. F. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011792 · Cited by 49
Self-citationPulvermüller, F. (2023) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2023.102511 · Cited by 55
Skeletal representations of shape in the human visual cortex
Ayzenberg, V., Kamps, F. S., Dilks, D. D., & Lourenco, S. F. (2021) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2021.108092 · Cited by 53
Degrees of algorithmic equivalence between the brain and its DNN models
Schyns, P. G., Snoek, L., & Daube, C. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.09.003 · Cited by 44
“…we rarely know the causal origin of the F being processed and represented in the brain (a second-degree question with more constrained benchmark 〈 S, R 〉 F ; cf [83]), let alone the speci fi c computations that cause them [a third-degree question, with even more constrained 〈 S, R 〉 A ( F ) ].”
BackgroundDiverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks
Cadena, S. A., Willeke, K. F., Restivo, K., Denfield, G. H., Sinz, F. H., Bethge, M., Tolias, A. S., & Ecker, A. S. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012056 · Cited by 39
“fMRI data) from multiple visual areas [12,49], we evaluated the intermediate representations of taskonomy networks – not just their final layers – to evaluate their affinities to single-cell responses.”
MethodResultDeep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
Oota, S. R., Chen, Z., Gupta, M., Bapi, R. S., Jobard, G., Alexandre, F., & Hinaut, X. (2025) · Transactions on machine learning research · doi:10.48550/arxiv.2307.10246 · Cited by 43
Universal dimensions of visual representation
Chen, Z., & Bonner, M. F. (2025) · Science Advances · doi:10.1126/sciadv.adw7697 · Cited by 16
Self-citationForecasting COVID-19 activity in Australia to support pandemic response: May to October 2020
Moss, R. G., Price, D. J., Golding, N., Dawson, P., McVernon, J., Hyndman, R. J., Shearer, F. M., & McCaw, J. M. (2023) · Scientific Reports · doi:10.1038/s41598-023-35668-6 · Cited by 16
Representational maps in the brain: concepts, approaches, and applications
Noda, T., Aschauer, D. F., Chambers, A. R., Seiler, J. P.-H., & Rumpel, S. (2024) · Frontiers in Cellular Neuroscience · doi:10.3389/fncel.2024.1366200 · Cited by 20
Versatile multiple object tracking in sparse 2D/3D videos via deformable image registration
Ryu, J., Nejatbakhsh, A., Torkashvand, M., Gangadharan, S., Seyedolmohadesin, M., Kim, J., Paninski, L., & Venkatachalam, V. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012075 · Cited by 21
The topology and geometry of neural representations
Lin, B., & Kriegeskorte, N. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2317881121 · Cited by 42
“Deep neural networks (DNNs) have emerged as important models of vision and brain-information processing in recent years [70, 71, 72, 73, 74].”
BackgroundGolan, T., Taylor, J., Schütt, H., Peters, B., Sommers, R. P., Seeliger, K., Doerig, A., Linton, P., Konkle, T., van Gerven, M., et al. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x23001553 · Cited by 12
Visual features are processed before navigational affordances in the human brain
Dwivedi, K., Saba-Sadiya, S., Balode, M. P., Roig, G., & Cichy, R. M. (2024) · Scientific Reports · doi:10.1038/s41598-024-55652-y · Cited by 13
“To ensure comparability across models, for each model, we selected the block4 and the output layer from the encoder as the representative taskspecific layers following (Dwivedi et al., 2021).”
BackgroundMethodSelf-citationRepresentation of locomotive action affordances in human behavior, brains, and deep neural networks
Bartnik, C. G., Sartzetaki, C., Sanchez, A. P., Molenkamp, E., Bommer, S., Vukšić, N., & Groen, I. I. A. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414005122 · Cited by 15
Decoding face recognition abilities in the human brain
Faghel-Soubeyrand, S., Ramon, M., Bamps, E., Zoia, M., Woodhams, J., Richoz, A.-R., Caldara, R., Gosselin, F., & Charest, I. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae095 · Cited by 13
“Recent studies using computational techniques have shown that word representations derived from models of natural language processing explain significant variance in the visual ventral stream (18, 75 – 77).”
BackgroundSummarization of Videos with the Signature Transform
de Curtò, J., de Zarzà, I., Roig, G., & Calafate, C. T. (2023) · Electronics · doi:10.3390/electronics12071735 · Cited by 9
Self-citationTomasello, R., Carriere, M., & Pulvermüller, F. (2024) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2024.108816 · Cited by 13
“…neural levels to offer an explanation of higher cognitive functions that are consistent with neuroscience knowledge and accepted by the neuroscience community (Breakspear, 2017; Dwivedi et al., 2021; Pezzulo et al., 2013; Pulvermüller, 2023; Pulvermüller et al., 2021; van Albada et al., 2021).”
BackgroundExploring the categorical nature of colour perception: Insights from artificial networks
Akbarinia, A. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106758 · Cited by 16
Brainformer: Mimic human visual brain functions to machine vision models via fMRI
Nguyen, X.-B., Li, X., Sinha, P., Khan, S. U., & Luu, K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129213 · Cited by 17
“According to recent studies [70, 33, 17], the brain can be divided into several Regions of Interest (ROI), where each region holds a different function.”
BackgroundPinzuti, E., Wollstadt, P., Tüscher, O., & Wibral, M. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010380 · Cited by 10
Human Visual Cortex and Deep Convolutional Neural Network Care Deeply about Object Background
Loke, J., Seijdel, N., Snoek, L., Sörensen, L. K. A., van de Klundert, R., van der Meer, M., Quispel, E., Cappaert, N. L. M., & Scholte, H. S. (2024) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02098 · Cited by 10
“…dynamics are not limited to low-level image statistics but also include high-level features such as animacy, object category, and semantics (Doerig et al., 2022; Takagi & Nishimoto, 2022; Dwivedi, Bonner, Cichy, & Roig, 2021; Ritchie et al., 2021; Eickenberg, Gramfort, Varoquaux, & Thirion, 2017).”
BackgroundTool Representations in Human Visual Cortex
Cortinovis, D., Peelen, M. V., & Bracci, S. (2024) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02281 · Cited by 14
Neural computations in prosopagnosia
Faghel-Soubeyrand, S., Richoz, A.-R., Waeber, D., Woodhams, J., Caldara, R., Gosselin, F., & Charest, I. (2024) · Cerebral Cortex · doi:10.1093/cercor/bhae211 · Cited by 7
“, 2021), and promoting a broad description of underlying brain mechanisms by testing diverse models on a whole-brain basis (Dwivedi et al., 2021; Kriegeskorte & Diedrichsen, 2019; Popham et al., 2021) might provide a more comprehensive understanding of these processes.”
BackgroundHierarchically Compositional Tasks and Deep Convolutional NetworksPreprint
Deza, A., Liao, Q., Banburski, A., & Poggio, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.13915 · Cited by 10
VisionTwinNet: Gated Clarity Enhancement Paired With Light-Robust CD Transformers
Chen, T., & Chen, A. (2024) · IEEE Access · doi:10.1109/access.2024.3350173 · Cited by 4
One Hundred Neural Networks and Brains Watching Videos: Lessons from AlignmentPreprint
Sartzetaki, C., Roig, G., Snoek, C. G. M., & Groen, I. I. A. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.12.05.626975 · Cited by 9
“…encoding (Eickenberg et al., 2017; St-Yves & Naselaris, 2018; La Tour et al., 2022) and RSA (Cichy et al., 2016; Dobs et al., 2019; King et al., 2019; Bartnik et al., 2024) as well as methods combining the two approaches (Dwivedi et al., 2021; Konkle & Alvarez, 2022; Conwell et al., 2022; 2024).”
Self-citationHuang, C.-I., Crump, R. E., Crowley, E. H., Hope, A., Bessell, P. R., Shampa, C., Miaka, E. M., & Rock, K. S. (2023) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0011299 · Cited by 3
Decoding face recognition abilities in the human brainPreprint
Faghel-Soubeyrand, S., Ramon, M., Bamps, E., Zoia, M., Woodhams, J., Richoz, A.-R., Caldara, R., Gosselin, F., & Charest, I. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.03.19.484245 · Cited by 4
Neural Correlates of High-Level Visual Saliency ModelsPreprint
Kroner, A., Senden, M., & Goebel, R. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.07.29.551075 · Cited by 4
“In practice, we averaged the representations within each class [54] and thus obtained two sets that represent a dichotomy of methods towards saliency prediction.”
MethodThe Umwelt Representation Hypothesis: rethinking Universality
Bosch, V., Sommers, R. P., Doerig, A., & Kietzmann, T. C. (2026) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2026.07.004 · Cited by 2
Neuro-cortical signatures of attention in uni- and multi-sensory audio-visual processing in children
Pervin, Z., Solis, I., Wang, Y.-P., Calhoun, V. D., Wilson, T. W., & Stephen, J. M. (2025) · Neuroscience · doi:10.1016/j.neuroscience.2025.11.021 · Cited by 2
NeuroFusionNet: cross-modal modeling from brain activity to visual understanding
Lang, K., Fang, J., & Su, G. (2025) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2025.1545971 · Cited by 3
Grosbard, I. D., & Yovel, G. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.09.632216 · Cited by 4
Dima, D. C., Culham, J. C., & Mohsenzadeh, Y. (2026) · Communications Biology · doi:10.1038/s42003-026-09834-1 · Cited by 2
Visual Object Agnosia in Acute Ischemic Stroke: A First Neuroimaging Biomarker
Тихомиров, Г. В., Nizgny Novgorod, R. F., Grigorieva, V. N., & Surkova, A. S. (2021) · Doctor Ru · doi:10.31550/1727-2378-2021-20-9-6-10 · Cited by 1
Models of vision need some action
Rothkopf, C. A., Bremmer, F., Fiehler, K., Dobs, K., & Triesch, J. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x23001577 · Cited by 1
A 2D Gabor-wavelet baseline model out-performs a 3D surface model in scene-responsive cortex
Shafer-Skelton, A., Brady, T. F., & Serences, J. T. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013888 · Cited by 2
“Dwivedi et al. [29] and Wang et al. [30] tested how well human brain responses could be predicted by the internal representations of several DNNs [15] that were trained to predict 3D properties of scene images (e.g., surface normals) vs. 2D or semantic properties of images (e.g., 2D edges or scene…”
A New Pooling Method for Cnn-based Deep Learning ModelsPreprint
Turhan, K., Kalaycı, E., & Özdemir, S. (2025) · Research Square · doi:10.21203/rs.3.rs-5871802/v1 · Cited by 1
Self-supervised learning yields representational signatures of category-selective cortexPreprint
Janini, D., & Cichy, R. M. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.09.704031 · Cited by 1
Self-citationRetina Gap Junction Networks Facilitate Blind Denoising in the Visual HierarchyPreprint
Yue, Y., Lun, K., He, L., He, G., Zhang, S., Ma, L., Liu, J. K., Tian, Y., Du, K., & Huang, T. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.05.16.491952 · Cited by 1
“…that the early visual areas (V1 and V2) may have a functional role related to low-level 2D visual tasks such as denoising Pratte et al. ( 2013 ); Dwivedi et al. ( 2021 ), whereas later visual areas (V3, V4, and IT) may involve 3D and semantic tasks such as classification Eickenberg et al. ( 2017…”
BackgroundUnderstanding how visual information is represented in humans and machines
Dwivedi, K. (2022) · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · doi:10.21248/gups.72036
Self-citationExploring neural architectures for simultaneously recognizing multiple visual attributes
Han, Z., & Sereno, A. B. (2024) · Scientific Reports · doi:10.1038/s41598-024-80679-6
Farahat, A., & Vinck, M. A. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.02.05.636721 · Cited by 2
“…various deep neural network models, trained under specific constraints regarding their architecture, objective function, and training data, can serve as tests for specific hypotheses about brain function [Zhuang et al., 2021, Dwivedi et al., 2021, Doerig et al., 2023, Cichy and Kaiser, 2019].”
Aligning machines with human values: opacity and ethical sustainability of AI
Cianciaruso, F. M. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01111-8
Contrastive learning to fine-tune feature extraction models for the visual cortex
Mulrooney, A., Li, Z., & Brockmeier, A. J. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014656
“[28] used different deep neural networks with the same architecture trained for different vision tasks and found correspondence between the task and ROI, for example, edge detection for V1 and semantic tasks for later layers.”
Metadata
- Authors
- Dwivedi, K., Bonner, M. F., Cichy, R. M., & Roig, G.
- Affiliations
- Goethe University Frankfurt (DE)
- Johns Hopkins University (US)
- Freie Universität Berlin (DE)
- Corresponding author
- Dwivedi, Kshitij
- Roig, Gemma
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-08-13
- Volume
- 17
- Issue
- 8
- Pages
- e1009267
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC8407579
- OpenAlex
- W3192643265
- Cited by
- 52
- References
- 70
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 16 · 2023: 11 · 2022: 8 · 2021: 2
- FWCI
- 4.86
- Type
- journal article
- Language
- en
- Update notices
- new_version (2021-08-31)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) (submittedVersion, cc-by)
- Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) (submittedVersion, cc-by-nc)
- Refubium (Universitätsbibliothek der Freien Universität Berlin) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Universitätsbibliothek der FU Berlin Hochschulschriftenstelle u. Dokumentenserver (cc-by)
- Funders
- Alfons und Gertrud Kassel-Stiftung
- dfg (CI241/1-1, CI241/3-1)
- European Research Council (ERC-2018-StG 803370)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Keywords
- visual cortex, functional mapping, deep neural network, ventral stream, dorsal stream, scene perception, 3D scene perception, semantic processing, hierarchical visual processing, functional MRI, computational psychiatry
- MeSH terms
- Deep Learning, Adult, Brain Mapping, Female, Humans, Magnetic Resonance Imaging, Male, Models, Neurological, Photic Stimulation, Semantics, Task Performance and Analysis, Visual Cortex, Visual Perception, Neural Networks, Computer, Computational Biology, Imaging, Three-Dimensional, Functional Neuroimaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.