Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks
Abstract
Recurrent neural networks (RNNs) are useful tools for learning nonlinear relationships between time-varying inputs and outputs with complex temporal dependencies. Recently developed algorithms have been successful at training RNNs to perform a wide variety of tasks, but the resulting networks have been treated as black boxes: their mechanism of operation remains unknown. Here we explore the hypothesis that fixed points, both stable and unstable, and the linearized dynamics around them, can reveal crucial aspects of how RNNs implement their computations. Further, we explore the utility of linearization in areas of phase space that are not true fixed points but merely points of very slow movement. We present a simple optimization technique that is applied to trained RNNs to find the fixed and slow points of their dynamics. Linearization around these slow regions can be used to explore, or reverse-engineer, the behavior of the RNN. We describe the technique, illustrate it using simple examples, and finally showcase it on three high-dimensional RNN examples: a 3-bit flip-flop device, an input-dependent sine wave generator, and a two-point moving average. In all cases, the mechanisms of trained networks could be inferred from the sets of fixed and slow points and the linearized dynamics around them.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These intermediate fixed points are unstable along a small number of dimensions (saddle fixed points) which 'funnel' neural activity to the appropriate location during a remapping event.
“In line with observations by Sussillo and Barak, 2013 on a discrete flip-flop task, these intermediate fixed points are unstable along a small number of dimensions (i.e. saddle fixed points; Figure 3D, gold points) which ‘funnel’ neural activity to the appropriate location during a remapping event.”
From Remapping in a recurrent neural network model of navigation and context inference (2023) · cites this paper as Sussillo and Barak, 2013
These intermediate fixed points are unstable along a small number of dimensions (saddle fixed points) which funnel neural activity to the appropriate location during a remapping event.
“In line with observations by Sussillo and Barak, 2013 on a discrete flip-flop task, these intermediate fixed points are unstable along a small number of dimensions (i.e. saddle fixed points; Figure 3D, gold points) which ‘funnel’ neural activity to the appropriate location during a remapping event.”
From Remapping in a recurrent neural network model of navigation and context inference (2023) · cites this paper as Sussillo and Barak, 2013
The eigenvalues and eigenvectors of the Jacobian matrix provide a local linear dynamical approximation to the full system.
“The eigenvalues and eigenvectors of this matrix then provide a local linear dynamical approximation to the full system as explained in Sussillo and Barak, 2013 and in the main text.”
From Remapping in a recurrent neural network model of navigation and context inference (2023) · cites this paper as Sussillo and Barak, 2013
Cites
Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication
Jaeger, H., & Haas, H. (2004) · Science · doi:10.1126/science.1091277 · Cited by 3,811 · see citing works
Neuronal correlates of parametric working memory in the prefrontal cortex
Romo, R., Brody, C. D., Hernández, A., & Lemus, L. (1999) · Nature · doi:10.1038/20939 · Cited by 822 · see citing works
Brain-wide neuronal dynamics during motor adaptation in zebrafish
Ahrens, M. B., Li, J. M., Orger, M. B., Robson, D. N., Schier, A. F., Engert, F., & Portugues, R. (2012) · Nature · doi:10.1038/nature11057 · Cited by 729 · see citing works
Decision Making in Recurrent Neuronal Circuits
Wang, X.-J. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.034 · Cited by 679 · cited 2× · see citing works
“This is in contrast to network models that are explicitly constructed to implement a specific known mechanism (see Wang, 2008; Hopfield, 1982).”
Result
Flagged first, then by citations
Cited by
Sorted by citations
The practical implementation of artificial intelligence technologies in medicine
He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., & Zhang, K. (2019) · Nature Medicine · doi:10.1038/s41591-018-0307-0 · Cited by 2,246 · see citing works
Artificial intelligence in surgery: promises and perils
Hashimoto, D. A., Rosman, G., Rus, D., & Meireles, O. R. (2018) · Annals of Surgery · doi:10.1097/sla.0000000000002693 · Cited by 1,468 · see citing works
Cocreation of Value in a Platform Ecosystem: The Case of Enterprise Software1
Ceccagnoli, M., Forman, C., Huang, P., & Wu, D. J. (2012) · MIS Quarterly · doi:10.2307/41410417 · Cited by 794 · see citing works
Computation Through Neural Population Dynamics
Vyas, S., Golub, M. D., Sussillo, D., & Shenoy, K. V. (2020) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-092619-094115 · Cited by 671 · see citing works
Self-citationTask representations in neural networks trained to perform many cognitive tasks
Yang, G. R., Joglekar, M. R., Song, H. F., Newsome, W. T., & Wang, X.-J. (2019) · Nature Neuroscience · doi:10.1038/s41593-018-0310-2 · Cited by 549 · see citing works
A neural network that finds a naturalistic solution for the production of muscle activity
Sussillo, D., Churchland, M. M., Kaufman, M. T., & Shenoy, K. V. (2015) · Nature Neuroscience · doi:10.1038/nn.4042 · Cited by 598 · see citing works
Self-citationToroidal topology of population activity in grid cells
Gardner, R. J., Hermansen, E., Pachitariu, M., Burak, Y., Baas, N. A., Dunn, B. A., Moser, M.-B., & Moser, E. I. (2022) · Nature · doi:10.1038/s41586-021-04268-7 · Cited by 428 · see citing works
Drawbacks of artificial intelligence and their potential solutions in the healthcare sector.
khan, B., Fatima, H., Qureshi, A., Kumar, S., Hanan, A., Hussain, J., & Abdullah, S. (2023) · Biomedical Materials & Devices · doi:10.1007/s44174-023-00063-2 · Cited by 461 · see citing works
Song, H. F., Yang, G. R., & Wang, X.-J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004792 · Cited by 281 · see citing works
Neural circuits as computational dynamical systems
Sussillo, D. (2014) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2014.01.008 · Cited by 247 · see citing works
Self-citationThe role of population structure in computations through neural dynamics
Dubreuil, A., Valente, A., Beiran, M., Mastrogiuseppe, F., & Ostojic, S. (2022) · Nature Neuroscience · doi:10.1038/s41593-022-01088-4 · Cited by 237 · see citing works
Flexible multitask computation in recurrent networks utilizes shared dynamical motifs
Driscoll, L., Shenoy, K., & Sussillo, D. (2022) · doi:10.1101/2022.08.15.503870 · Cited by 217 · see citing works
Self-citationMotor primitives in space and time via targeted gain modulation in cortical networks
Stroud, J. P., Porter, M. A., Hennequin, G., & Vogels, T. P. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0276-0 · Cited by 119 · see citing works
Machine Learning in Pharmacometrics: Opportunities and Challenges
McComb, M., Bies, R., & Ramanathan, M. (2022) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.14801 · Cited by 128 · see citing works
Auxiliary Tasks and Exploration Enable ObjectGoal Navigation
Ye, J., Batra, D., Das, A., & Wijmans, E. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01581 · Cited by 120 · see citing works
A goal-driven modular neural network predicts parietofrontal neural dynamics during grasping
Michaels, J. A., Schaffelhofer, S., Agudelo-Toro, A., & Scherberger, H. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2005087117 · Cited by 103 · see citing works
Different population dynamics in the supplementary motor area and motor cortex during reaching
Lara, A. H., Cunningham, J. P., & Churchland, M. M. (2018) · Nature Communications · doi:10.1038/s41467-018-05146-z · Cited by 101 · see citing works
Metastable dynamics of neural circuits and networks
Brinkman, B. A. W., Yan, H., Maffei, A., Park, I. M., Fontanini, A., Wang, J., & La Camera, G. (2022) · Applied Physics Reviews · doi:10.1063/5.0062603 · Cited by 64 · see citing works
Emergent behaviour and neural dynamics in artificial agents tracking odour plumes
Singh, S. H., van Breugel, F., Rao, R. P. N., & Brunton, B. W. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00599-w · Cited by 61 · see citing works
“We analyse emergent behaviours and neural dynamics at the network level, which provides us with an abstract understanding of task-relevant neural computations that is robust to small changes in network architecture and training hyperparameters 39,41,42 .”
BackgroundMethodInfluentialEmergence of belief-like representations through reinforcement learning
Hennig, J. A., Romero Pinto, S. A., Yamaguchi, T., Linderman, S. W., Uchida, N., & Gershman, S. J. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011067 · Cited by 41 · see citing works
Do Agile Managed Information Systems Projects Fail Due to a Lack of Emotional Intelligence?
Luong, T. T., Sivarajah, U., & Weerakkody, V. (2021) · Information Systems Frontiers · doi:10.1007/s10796-019-09962-6 · Cited by 45 · see citing works
Barriers to Implementing an Artificial Intelligence Model for Unplanned Readmissions
Baxter, S. L., Bass, J. S., & Sitapati, A. M. (2020) · ACI Open · doi:10.1055/s-0040-1716748 · Cited by 30 · see citing works
A coherent perceptron for all-optical learning
Tezak, N., & Mabuchi, H. (2015) · EPJ Quantum Technology · doi:10.1140/epjqt/s40507-015-0023-3 · Cited by 22 · see citing works
Remapping in a recurrent neural network model of navigation and context inference
Low, I. I., Giocomo, L. M., & Williams, A. H. (2023) · eLife · doi:10.7554/elife.86943 · Cited by 15 · see citing works
“We numerically identified fixed points according to the methods described in Sussillo & Barak.”
BackgroundMethodResultInfluentialEvolution of Brains and Computers: The Roads Not Taken
Solé, R., & Seoane, L. F. (2022) · Entropy · doi:10.3390/e24050665 · Cited by 14 · see citing works
Remapping in a recurrent neural network model of navigation and context inference
Low, I. I. C., Giocomo, L. M., & Williams, A. H. (2023) · doi:10.1101/2023.01.25.525596 · Cited by 3 · see citing works
Remapping in a recurrent neural network model of navigation and context inference
Low, I. I., Giocomo, L. M., & Williams, A. H. (2023) · eLife · doi:10.7554/elife.86943.3 · Cited by 1 · see citing works
- Authors
- Sussillo, D., & Barak, O.
- Affiliations
- Stanford University (US)
- Columbia University Irving Medical Center (US)
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2013-03
- Volume
- 25
- Issue
- 3
- Pages
- 626-649
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W1980324747
- Cited by
- 575
- References
- 4
- Citations by year
- 2026: 56 · 2025: 66 · 2024: 55 · 2023: 57 · 2022: 33 · 2021: 46 · 2020: 45 · 2019: 46 · 2018: 35 · 2017: 33 · 2016: 22 · 2015: 15 · 2014: 20 · 2013: 21 · 2012: 9
- FWCI
- 20.31
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Gatsby Charitable Foundation
- National Institutes of Health
- National Institute of Mental Health (MH093338, R01MH093338)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Reservoir Computing
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
- Keywords
- recurrent neural networks, fixed points, slow dynamics, linearization, low-dimensional dynamics, high-dimensional dynamical systems, phase space, sine wave generator, moving average, nonlinear dynamics
- MeSH terms
- Artificial Intelligence, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
