Learning, Memory, and the Role of Neural Network Architecture
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
The performance of information processing systems, from artificial neural networks to natural neuronal ensembles, depends heavily on the underlying system architecture. In this study, we compare the performance of parallel and layered network architectures during sequential tasks that require both acquisition and retention of information, thereby identifying tradeoffs between learning and memory processes. During the task of supervised, sequential function approximation, networks produce and adapt representations of external information. Performance is evaluated by statistically analyzing the error in these representations while varying the initial network state, the structure of the external information, and the time given to learn the information. We link performance to complexity in network architecture by characterizing local error landscape curvature. We find that variations in error landscape structure give rise to tradeoffs in performance; these include the ability of the network to maximize accuracy versus minimize inaccuracy and produce specific versus generalizable representations of information. Parallel networks generate smooth error landscapes with deep, narrow minima, enabling them to find highly specific representations given sufficient time. While accurate, however, these representations are difficult to generalize. In contrast, layered networks generate rough error landscapes with a variety of local minima, allowing them to quickly find coarse representations. Although less accurate, these representations are easily adaptable. The presence of measurable performance tradeoffs in both layered and parallel networks has implications for understanding the behavior of a wide variety of natural and artificial learning systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Forecasting with artificial neural networks:
Zhang, G., Eddy Patuwo, B., & Y. Hu, M. (1998) · International Journal of Forecasting · doi:10.1016/s0169-2070(97)00044-7 · Cited by 4,229
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770
Homeostatic plasticity in the developing nervous system
Turrigiano, G. G., & Nelson, S. B. (2004) · Nature Reviews Neuroscience · doi:10.1038/nrn1327 · Cited by 2,485
Activity-dependent scaling of quantal amplitude in neocortical neurons
Turrigiano, G. G., Leslie, K. R., Desai, N. S., Rutherford, L. C., & Nelson, S. B. (1998) · Nature · doi:10.1038/36103 · Cited by 2,480
Synaptic plasticity: taming the beast
Abbott, L. F., & Nelson, S. B. (2000) · Nature Neuroscience · doi:10.1038/81453 · Cited by 2,263 · cited 1×
“These competing processes are also maintained in natural neuronal systems and neural circuit models through homeostatic plasticity mechanisms such as synaptic scaling [54,55] and redistribution [56,57], in addition to the rehearsal methods employed here [19–23].”
Tononi, G., Sporns, O., & Edelman, G. M. (1994) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.91.11.5033 · Cited by 1,626 · cited 1×
“This performance supports the hypothesis that short path lengths (similar to the serialization h =‘ [52]) and low connection densities may facilitate simultaneous performance of information segregation (memory retention) and integration (generalization) within natural neuronal systems [53].”
BackgroundCellular and network mechanisms of rhythmic recurrent activity in neocortex
Sanchez-Vives, M. V., & McCormick, D. A. (2000) · Nature Neuroscience · doi:10.1038/79848 · Cited by 1,539 · cited 1×
“. Lastly, we assumed feedforward connectivity. The addition of recurrent connections could be used to study the relationship between recurrent structure and oscillatory functions such as cortical sleep rhythms”
MethodTraining induces changes in white-matter architecture
Scholz, J., Klein, M. C., Behrens, T. E. J., & Johansen-Berg, H. (2009) · Nature Neuroscience · doi:10.1038/nn.2412 · Cited by 1,508
Rapid formation and selective stabilization of synapses for enduring motor memories
Xu, T., Yu, X., Perlik, A. J., Tobin, W. F., Zweig, J. A., Tennant, K., Jones, T., & Zuo, Y. (2009) · Nature · doi:10.1038/nature08389 · Cited by 1,193
Modular and hierarchically modular organization of brain networks
Meunier, D., Lambiotte, R., & Bullmore, E. T. (2010) · Frontiers in Neuroscience · doi:10.3389/fnins.2010.00200 · Cited by 1,156
Visuomotor integration is associated with zero time-lag synchronization among cortical areas
Roelfsema, P. R., Engel, A. K., König, P., & Singer, W. (1997) · Nature · doi:10.1038/385157a0 · Cited by 1,136
Ratcliff, R. (1990) · Psychological Review · doi:10.1037//0033-295x.97.2.285 · Cited by 941
Spontaneously emerging cortical representations of visual attributes
Kenet, T., Bibitchkov, D., Tsodyks, M., Grinvald, A., & Arieli, A. (2003) · Nature · doi:10.1038/nature02078 · Cited by 924 · cited 1×
“serial and parallel [5] structural motifs which enable the performance of disparate functions [6–9].”
BackgroundCatastrophic forgetting, rehearsal and pseudorehearsal
ROBINS, A. (1995) · Connection Science · doi:10.1080/09540099550039318 · Cited by 833 · cited 2×
“Pseudorehearsal has been shown to prevent catastrophic forgetting in both feedforward and recurrent networks and does not require extensive storage of examples [22,23].”
BackgroundMethodRojas, R. (1996) · doi:10.1007/978-3-642-61068-4 · Cited by 810
Age-related changes in modular organization of human brain functional networks
Meunier, D., Achard, S., Morcom, A., & Bullmore, E. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.09.062 · Cited by 756
Chittka, L., & Niven, J. (2009) · Current Biology · doi:10.1016/j.cub.2009.08.023 · Cited by 598 · cited 1×
“Natural neuronal systems display a complex combination of serial and parallel”
BackgroundModeling the impact of lesions in the human brain
Alstott, J., Breakspear, M., Hagmann, P., Cammoun, L., & Sporns, O. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000408 · Cited by 561 · cited 1×
“The converse case of network degradation, or disruptions to structural connectivity, is also known to have widespread consequences in functional properties of the brain [61–63].”
BackgroundDynamical consequences of lesions in cortical networks
Honey, C. J., & Sporns, O. (2008) · Human Brain Mapping · doi:10.1002/hbm.20579 · Cited by 353 · cited 1×
“The converse case of network degradation, or disruptions to structural connectivity, is also known to have widespread consequences in functional properties of the brain [61–63].”
BackgroundThe statistical mechanics of complex signaling networks: Nerve growth factor signaling.
Brown, K. S., Hill, C. C., Calero, G. A., Myers, C. R., Lee, K. H., Sethna, J. P., & Cerione, R. A. (2004) · Physical Biology · doi:10.1088/1478-3967/1/3/006 · Cited by 265 · cited 2×
“We adopt the terminology used in previous studies and refer to directions with high and low curvature as stiff and sloppy, respectively [31,32].”
BackgroundMethodSelf-citationFunctional Connectivity of the Medial Temporal Lobe Relates to Learning and Awareness
McIntosh, A. R., Rajah, M. N., & Lobaugh, N. J. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-16-06520.2003 · Cited by 89
Cited by
Small-World Brain Networks Revisited
Bassett, D. S., & Bullmore, E. T. (2017) · The Neuroscientist · doi:10.1177/1073858416667720 · Cited by 842
Self-citationProjections from neocortex mediate top-down control of memory retrieval
Rajasethupathy, P., Sankaran, S., Marshel, J. H., Kim, C. K., Ferenczi, E., Lee, S. Y., Berndt, A., Ramakrishnan, C., Jaffe, A., Lo, M., et al. (2015) · Nature · doi:10.1038/nature15389 · Cited by 509
The physics of brain network structure, function and control
Lynn, C. W., & Bassett, D. S. (2019) · Nature Reviews Physics · doi:10.1038/s42254-019-0040-8 · Cited by 400
Self-citationCliques and cavities in the human connectome
Sizemore, A. E., Giusti, C., Kahn, A., Vettel, J. M., Betzel, R. F., & Bassett, D. S. (2018) · Journal of Computational Neuroscience · doi:10.1007/s10827-017-0672-6 · Cited by 372
Self-citationOptimal trajectories of brain state transitions
Gu, S., Betzel, R. F., Mattar, M. G., Cieslak, M., Delio, P. R., Grafton, S. T., Pasqualetti, F., & Bassett, D. S. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.01.003 · Cited by 183
“One key challenge hampering progress is the complexity of these trajectories, which stems in part from the architectural complexity of the underlying anatomy (Hermundstad et al., 2011, 2013, 2014).”
Self-citation
Metadata
- Authors
- Hermundstad, A. M., Brown, K. S., Bassett, D. S., & Carlson, J. M.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2011-06-30
- Volume
- 7
- Issue
- 6
- Pages
- e1002063
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2170345111
- Cited by
- 50
- References
- 21
- FWCI
- 1.77
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.