CoChat
Sign Up
Cited by 406Visual perception and processing mechanismsFace Recognition and PerceptionVisual Attention and Saliency Detection

The DOI matches its Crossref record; 66 of the 68 references it cites were found in a registry or an index.

Recurrence is required to capture the representational dynamics of the human visual system

Open access
Kietzmann, Tim C., Spoerer, Courtney J., Sörensen, Lynn K. A., Cichy, Radoslaw M., Hauk, Olaf, Kriegeskorte, Nikolaus·2019·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.1905544116PMC PMC6815174
Open this paper in CoChat — free accountOpen PDF

Abstract

Significance: Understanding the computational principles that underlie human vision is a key challenge for neuroscience and could help improve machine vision. Feedforward neural network models process their input through a deep cascade of computations. These models can recognize objects in images and explain aspects of human rapid recognition. However, the human brain contains recurrent connections within and between stages of the cascade, which are missing from the models that dominate both engineering and neuroscience. Here, we measure and model the dynamics of human brain activity during visual perception. We compare feedforward and recurrent neural network models and find that only recurrent models can account for the dynamic transformations of representations among multiple regions of visual cortex.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.1905544116

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Representational dynamics analysis (RDA) reveals how feature selectivity emerges over time along distinct ventral-stream regions. (A) RDA pipeline to extract source-space RDM movies. (B) Participant-averaged RDMs (ranked) for regions V1–V3, V4t/LO, and IT/PHC at selected time points. All ROIs exhibit distinctive multistage representational trajectories. (C) Linear modeling of the temporal evolution of feature selectivity reveals a staggered emergence of representational distinctions within and across ventral-stream ROIs (black curve, V1–V3; blue, V4t/LO; red, IT/PHC). The horizontal bars indicate time points with effects significantly exceeding prestimulus baseline (nonparametric cluster-correction; cluster inclusion and significance level, P < 0.05). SE across participants shown as shaded area. Representational geometries at selected time points and ROIs are visualized in 2D using multidimensional scaling (MDS) to visualize effect magnitude.
Fig. 1. Representational dynamics analysis (RDA) reveals how feature selectivity emerges over time along distinct ventral-stream regions. (A) RDA pipeline to extract source-space RDM movies. (B) Participant-averaged RDMs (ranked) for regions V1–V3, V4t/LO, and IT/PHC at selected time points. All ROIs exhibit distinctive multistage representational trajectories. (C) Linear modeling of the temporal evolution of feature selectivity reveals a staggered emergence of representational distinctions within and across ventral-stream ROIs (black curve, V1–V3; blue, V4t/LO; red, IT/PHC). The horizontal bars indicate time points with effects significantly exceeding prestimulus baseline (nonparametric cluster-correction; cluster inclusion and significance level, P < 0.05). SE across participants shown as shaded area. Representational geometries at selected time points and ROIs are visualized in 2D using multidimensional scaling (MDS) to visualize effect magnitude.CC BY-NC-ND · PubMed Central
Fig. 2. RSA Granger causality analysis was performed to estimate information flow between ventral-stream areas. (A) Feedforward (purple) and feedback (orange) direction of Granger causal influence between early and intermediate ROIs, and (B) effects between intermediate- and high-level ROIs. The horizontal bars indicate time points with causal interactions exceeding effects during prestimulus baseline (FDR corrected at q < 0.05). Data are shown baseline corrected.
Fig. 2. RSA Granger causality analysis was performed to estimate information flow between ventral-stream areas. (A) Feedforward (purple) and feedback (orange) direction of Granger causal influence between early and intermediate ROIs, and (B) effects between intermediate- and high-level ROIs. The horizontal bars indicate time points with causal interactions exceeding effects during prestimulus baseline (FDR corrected at q < 0.05). Data are shown baseline corrected.CC BY-NC-ND · PubMed Central
Fig. 3. DNN modeling of ventral-stream representational dynamics. (A) The RDM movies of all 3 ventral-stream regions were used as time-varying deep learning objectives targeting separate DNN layers, together with a time-decaying category objective at the readout. Each artificial network thereby attempts to simultaneously capture the representational dynamics of all ventral-stream areas. Stimulus display (Top Left) adapted with permission from ref. 20. (B) Development of the average pattern distance across time. MEG data are shown together with the results of ramping feedforward, and recurrent DNNs. (C) Average frame-by-frame RDM correlation between model and brain. Correlations estimated on separate data from individual participants, shown as gray dots. Data are normalized by the predictive performance of the MEG RDM movies used for training (normalization factor shown for each region at the level of 1.0). For all ROIs (black, V1–V3; blue, V4t/LO; red, IT/PHC), recurrent networks significantly outperform ramping feedforward architectures (significance indicated by gray horizontal lines above). (D) Cross-validated predictive performance of different DNN architectures trained on the MEG data when tested against fMRI RDMs, acquired from the same participants and ROIs. Correlations were noise-normalized using the respective lower bound of the noise ceiling. For all regions, recurrent networks significantly outperform ramping feedforward architectures.
Fig. 3. DNN modeling of ventral-stream representational dynamics. (A) The RDM movies of all 3 ventral-stream regions were used as time-varying deep learning objectives targeting separate DNN layers, together with a time-decaying category objective at the readout. Each artificial network thereby attempts to simultaneously capture the representational dynamics of all ventral-stream areas. Stimulus display (Top Left) adapted with permission from ref. 20. (B) Development of the average pattern distance across time. MEG data are shown together with the results of ramping feedforward, and recurrent DNNs. (C) Average frame-by-frame RDM correlation between model and brain. Correlations estimated on separate data from individual participants, shown as gray dots. Data are normalized by the predictive performance of the MEG RDM movies used for training (normalization factor shown for each region at the level of 1.0). For all ROIs (black, V1–V3; blue, V4t/LO; red, IT/PHC), recurrent networks significantly outperform ramping feedforward architectures (significance indicated by gray horizontal lines above). (D) Cross-validated predictive performance of different DNN architectures trained on the MEG data when tested against fMRI RDMs, acquired from the same participants and ROIs. Correlations were noise-normalized using the respective lower bound of the noise ceiling. For all regions, recurrent networks significantly outperform ramping feedforward architectures.CC BY-NC-ND · PubMed Central
Fig. 4. DNN cooling studies. (A) Virtual cooling experiments allow for the specific targeting and deactivation of input connections into distinct network layers. The effects of lateral (Left) and top-down (Middle) computations on object categorization performance vary across network depth, with stronger effects observed while deactivating recurrence in earlier layers. Applied to the whole network (Right), the cooling of lateral and top down connections have comparable effects, with perhaps stronger reliance on lateral connectivity. (B) Targeting specific connection types throughout the network reveals the importance of lateral and top-down network connections for modeling human ventral-stream dynamics. Later network layers are increasingly robust to the cooling of top-down input.
Fig. 4. DNN cooling studies. (A) Virtual cooling experiments allow for the specific targeting and deactivation of input connections into distinct network layers. The effects of lateral (Left) and top-down (Middle) computations on object categorization performance vary across network depth, with stronger effects observed while deactivating recurrence in earlier layers. Applied to the whole network (Right), the cooling of lateral and top down connections have comparable effects, with perhaps stronger reliance on lateral connectivity. (B) Targeting specific connection types throughout the network reveals the importance of lateral and top-down network connections for modeling human ventral-stream dynamics. Later network layers are increasingly robust to the cooling of top-down input.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 68 references

Research works printed but not linked: 2

  1. J. Mehrer, T. C. Kietzmann, N. Kriegeskorte, “Deep neural networks trained on ecologically relevant categories better explain human IT” in Cognitive Computational Neuroscience Meeting (Cognitive Computational Neuroscience, 2017), pp. 1–2.
  2. D. P. Kingma J. L. Ba Adam: A method for stochastic optimization. arXiv:1412.6980 (30 January 2017). · arXiv

Cited by

Sorted by citations

See all 406

Related

Metadata

Authors
Kietzmann, T. C., Spoerer, C. J., Sörensen, L. K. A., Cichy, R. M., Hauk, O., & Kriegeskorte, N.
Affiliations
  • Radboud University Nijmegen (NL)
  • MRC Cognition and Brain Sciences Unit (GB)
  • University of Amsterdam (NL)
  • Freie Universität Berlin (DE)
  • Columbia University (US)
Corresponding author
Kietzmann, Tim C.
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2019-10-07
Volume
116
Issue
43
Pages
21854-21863
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.1905544116
PMC
PMC6815174
OpenAlex
W2979357328
Cited by
406
References
68
Citations by year
2026: 44 · 2025: 60 · 2024: 66 · 2023: 57 · 2022: 58 · 2021: 66 · 2020: 50 · 2019: 5
FWCI
26.02
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Refubium (Universitätsbibliothek der Freien Universität Berlin) (submittedVersion, cc-by-nc-nd)
  • UvA-DARE (University of Amsterdam) (submittedVersion, cc-by-nc-nd)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Apollo (University of Cambridge) (submittedVersion, cc-by-nc-nd)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • arXiv (Cornell University)
  • Universitätsbibliothek der FU Berlin Hochschulschriftenstelle u. Dokumentenserver
  • Apollo
Funders
  • European Commission (604102, 261352)
  • Deutsche Forschungsgemeinschaft
  • Medical Research Council (MC-A060- 5PR20)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords
recurrent neural networks, ventral stream, representational dynamics, object recognition, Granger causality, bidirectional information flow, top-down connections
MeSH terms
Deep Learning, Adult, Female, Humans, Models, Neurological, Nerve Net, Visual Pathways, Visual Perception, Magnetoencephalography, Feedback, Sensory

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review