CoChat

Brain-like functional specialization emerges spontaneously in deep neural networks

Open access

Published version of a preprint

The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.

Abstract

The human brain contains multiple regions with distinct, often highly specialized functions, from recognizing faces to understanding language to thinking about what others are thinking. However, it remains unclear why the cortex exhibits this high degree of functional specialization in the first place. Here, we consider the case of face perception using artificial neural networks to test the hypothesis that functional segregation of face recognition in the brain reflects a computational optimization for the broader problem of visual recognition of faces and other visual categories. We find that networks trained on object recognition perform poorly on face recognition and vice versa and that networks optimized for both tasks spontaneously segregate themselves into separate systems for faces and objects. We then show functional segregation to varying degrees for other visual categories, revealing a widespread tendency for optimization (without built-in task-specific inductive biases) to lead to functional specialization in machines and, we conjecture, also brains.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1126/sciadv.abl8913

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Distinct face and object representations in singly trained CNNs while a dual-task CNN performs well. (A) Three networks with VGG16 architecture (left) were optimized, one on face identity categorization (Face CNN in red), one on object categorization (Object CNN in orange), and one on both tasks simultaneously (dual-task CNN in gray). (B) Decoding accuracy of held-out face identities and held-out object categories using activation patterns extracted from the penultimate layer [i.e., FC2 in (A)] of the Face CNN and the Object CNN. The Face CNN outperforms the Object CNN in face decoding and vice versa for object decoding. Thus, the representations optimized for each task do not naturally support the other. The dashed gray line indicates chance level (1%). Error bars indicate SEM across classification folds. (C) A dual-task CNN optimized on both tasks performed and the separate networks (% top 1 accuracy on the test set). Error bars denote 95% confidence interval (CI) bootstrapped across classes and stimuli.
Fig. 1. Distinct face and object representations in singly trained CNNs while a dual-task CNN performs well. (A) Three networks with VGG16 architecture (left) were optimized, one on face identity categorization (Face CNN in red), one on object categorization (Object CNN in orange), and one on both tasks simultaneously (dual-task CNN in gray). (B) Decoding accuracy of held-out face identities and held-out object categories using activation patterns extracted from the penultimate layer [i.e., FC2 in (A)] of the Face CNN and the Object CNN. The Face CNN outperforms the Object CNN in face decoding and vice versa for object decoding. Thus, the representations optimized for each task do not naturally support the other. The dashed gray line indicates chance level (1%). Error bars indicate SEM across classification folds. (C) A dual-task CNN optimized on both tasks performed and the separate networks (% top 1 accuracy on the test set). Error bars denote 95% confidence interval (CI) bootstrapped across classes and stimuli.CC BY-NC · PubMed Central
Fig. 2. Lesion experiments in the last convolutional layer reveal spontaneous task segregation. (A) Schematic of lesion experiments for the last convolutional layer (see “Conv13” in Fig. 1A) in VGG16. Each filter in the layer was ablated while measuring the loss to batches of face (top) and object (bottom) images. The filters were rank-ordered by their corresponding losses to determine those that contribute most to face (red) or object recognition (orange). (B) Normalized performance of face and object tasks after lesioning the 20% highest-ranking filters for the face task (top) and the object task (bottom) in the last convolutional layer. Error bars denote 95% CIs bootstrapped across classes and stimuli.
Fig. 2. Lesion experiments in the last convolutional layer reveal spontaneous task segregation. (A) Schematic of lesion experiments for the last convolutional layer (see “Conv13” in Fig. 1A) in VGG16. Each filter in the layer was ablated while measuring the loss to batches of face (top) and object (bottom) images. The filters were rank-ordered by their corresponding losses to determine those that contribute most to face (red) or object recognition (orange). (B) Normalized performance of face and object tasks after lesioning the 20% highest-ranking filters for the face task (top) and the object task (bottom) in the last convolutional layer. Error bars denote 95% CIs bootstrapped across classes and stimuli.CC BY-NC · PubMed Central
Fig. 3. Spontaneous segregation of face and object tasks in mid-level processing stages. (A) Task segregation, measured as combined index of the differences in proportional drops in performance on the face and object task, when the 20% highest-contributing filters are dropped in each convolutional layer. Task segregation increased after the first convolutional layers to a maximum index of 0.75. Shaded area represents 95% CIs bootstrapped across classes and stimuli. (B) Images optimized to drive responses in three example filters among the top 10 selected filters for the face (left) and the object (right) task in convolutional layers 5, 9, and 13 (rows). The size of the receptive fields increases, and features become more task specific in later layers.
Fig. 3. Spontaneous segregation of face and object tasks in mid-level processing stages. (A) Task segregation, measured as combined index of the differences in proportional drops in performance on the face and object task, when the 20% highest-contributing filters are dropped in each convolutional layer. Task segregation increased after the first convolutional layers to a maximum index of 0.75. Shaded area represents 95% CIs bootstrapped across classes and stimuli. (B) Images optimized to drive responses in three example filters among the top 10 selected filters for the face (left) and the object (right) task in convolutional layers 5, 9, and 13 (rows). The size of the receptive fields increases, and features become more task specific in later layers.CC BY-NC · PubMed Central
Fig. 4. No segregation for random tasks. (A) Schematic of randomly assigning 50% of the face and 50% of the object classes to new tasks A and B. Each filter in the last convolutional layer was ablated while measuring the loss to batches of images belonging to task A or task B. Using a greedy procedure, the filters were rank-ordered by their corresponding losses to determine those that contribute most to task A or task B. (B) Normalized performance of tasks A (dark gray) and B (light gray) after lesioning the 20% highest-contributing filters for tasks A (left) and B (right) in the last convolutional layer. Performance decrement through lesioning was smaller than for the original tasks (Fig. 2B) and affected both tasks equally. Error bars denote 95% CIs bootstrapped across classes and stimuli.
Fig. 4. No segregation for random tasks. (A) Schematic of randomly assigning 50% of the face and 50% of the object classes to new tasks A and B. Each filter in the last convolutional layer was ablated while measuring the loss to batches of images belonging to task A or task B. Using a greedy procedure, the filters were rank-ordered by their corresponding losses to determine those that contribute most to task A or task B. (B) Normalized performance of tasks A (dark gray) and B (light gray) after lesioning the 20% highest-contributing filters for tasks A (left) and B (right) in the last convolutional layer. Performance decrement through lesioning was smaller than for the original tasks (Fig. 2B) and affected both tasks equally. Error bars denote 95% CIs bootstrapped across classes and stimuli.CC BY-NC · PubMed Central
Fig. 5. Dual-trained CNN is most correlated with behavior. Correlations between behavioral RDMs for either face (left, n = 14) or object (right, n = 15) stimuli and layer-specific RDMs obtained from activation patterns in the Face CNN (red), the Object CNN (in yellow), and the dual-task CNN (in gray) to the corresponding stimuli. Color-shaded areas denote bootstrapped SEM across participants. Gray-shaded horizontal bars indicate estimated noise ceiling based on the variability across participants.
Fig. 5. Dual-trained CNN is most correlated with behavior. Correlations between behavioral RDMs for either face (left, n = 14) or object (right, n = 15) stimuli and layer-specific RDMs obtained from activation patterns in the Face CNN (red), the Object CNN (in yellow), and the dual-task CNN (in gray) to the corresponding stimuli. Color-shaded areas denote bootstrapped SEM across participants. Gray-shaded horizontal bars indicate estimated noise ceiling based on the variability across participants.CC BY-NC · PubMed Central
Fig. 6. Spontaneous segregation to varying degrees for food or car recognition. (A) In addition to the dual-task model for face and object tasks (red), we trained one dual-task model on food (green) and object categorization and another one on car (blue) and object categorization. (B) Task segregation was measured by lesioning the most-contributing filters for faces, food, and cars (respectively) and objects in each convolutional layer. Task segregation was found for all tasks to varying degrees. Task segregation for cars and objects increased later, to a lesser degree, than for food or faces and objects. Color-shaded areas denote 95% CIs bootstrapped across classes and stimuli.
Fig. 6. Spontaneous segregation to varying degrees for food or car recognition. (A) In addition to the dual-task model for face and object tasks (red), we trained one dual-task model on food (green) and object categorization and another one on car (blue) and object categorization. (B) Task segregation was measured by lesioning the most-contributing filters for faces, food, and cars (respectively) and objects in each convolutional layer. Task segregation was found for all tasks to varying degrees. Task segregation for cars and objects increased later, to a lesser degree, than for food or faces and objects. Color-shaded areas denote 95% CIs bootstrapped across classes and stimuli.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 58 references

Research works printed but not linked: 6

  1. S. Finger Origins Of Neuroscience: A History Of Explorations Into Brain Function (Oxford Univ. Press 2001).
  2. C. R. Gallistel The replacement of general-purpose learning models with adaptively specialized learning modules in The Cognitive Neurosciences M. S. Gazzaniga Ed. (MIT Press ed. 2 2000) pp. 1179–1191.
  3. B. Long G. Kachergis K. Agrawal M. C. Frank Detecting social information in a dense dataset of infants’ natural visual experience (PsyArXiv 2020); https://psyarxiv.com/z7tdg/.
  4. 10.1037/0033-295x.98.2.164 · doi:10.1037/0033-295x.98.2.164 · Paper page
  5. A. Nayebi N. C. L. Kong C. Zhuang J. L. Gardner A. M. Norcia D. L. K. Yamins Unsupervised models of mouse visual cortex. bioRxiv 448730 [Preprint] (2021); www.biorxiv.org/content/10.1101/2021.06.16.448730v1. · doi:10.1101/2021.06.16.448730v1
  6. A. Krizhevsky I. Sutskever G. E. Hinton in Advances in Neural Information Processing Systems (MIT Press 2012) pp. 1097–1105.

Cited by

See all 137

Metadata

Authors
Dobs, K., Martinez, J., Kell, A. J. E., & Kanwisher, N.
Affiliations
  • McGovern Institute for Brain Research (US)
  • Mortimer B. Zuckerman Mind Brain Behavior Institute (US)
Corresponding author
Dobs, Katharina
Venue
Science Advances
Published
2022-03-18
Volume
8
Issue
11
Pages
eabl8913-eabl8913
ISSN
2375-2548 (ISSN-L)
OpenAlex
W4225899846
Cited by
129
References
52
Citations by year
2026: 25 · 2025: 24 · 2024: 33 · 2023: 30 · 2022: 16
FWCI
13.28
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • Science Advances (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Funders
Eunice Kennedy Shriver National Institute of Child Health and Human Development (DP1HD091947)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
functional specialization, functional segregation, face recognition (cognition), object recognition, deep neural network, computational optimization, inductive bias, brain organization, visual cortex

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.