The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Continual learning of context-dependent processing in neural networks
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It has been shown that input created from a generative model can indeed assist with learning, such as preventing catastrophic forgetting.149”
From The overfitted brain: Dreams evolved to assist generalization (2021) · cites this paper as [149]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Golub, G. (2012) · doi:10.56021/9781421407944 · Cited by 34,505
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
An Integrative Theory of Prefrontal Cortex Function
Miller, E. K., & Cohen, J. D. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.167 · Cited by 12,803
Neural Mechanisms of Selective Visual Attention
Desimone, R., & Duncan, J. (1995) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.18.030195.001205 · Cited by 8,388
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
International Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995) · Psychological Review · doi:10.1037/0033-295x.102.3.419 · Cited by 5,256
Li, Z., & Hoiem, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2773081 · Cited by 4,202
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117
The prefontral cortex and cognitive control
Miller, E. K. (2000) · Nature Reviews Neuroscience · doi:10.1038/35036228 · Cited by 2,016
Context-dependent computation by recurrent dynamics in prefrontal cortex
Mante, V., Sussillo, D., Shenoy, K. V., & Newsome, W. T. (2013) · Nature · doi:10.1038/nature12742 · Cited by 1,910
Neuronal gamma-band synchronization as a fundamental process in cortical computation
Fries, P. (2009) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.051508.135603 · Cited by 1,713
The importance of mixed selectivity in complex cognitive tasks
Rigotti, M., Barak, O., Warden, M. R., Wang, X.-J., Daw, N. D., Miller, E. K., & Fusi, S. (2013) · Nature · doi:10.1038/nature12160 · Cited by 1,639
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307 · cited 1×
“In contrast to regularization approaches, other types of continual learning methods involve dynamically introducing extra neurons or layers along the learning process [42], which may help mitigate the tradeoff described above [15].”
BackgroundThe ventral visual pathway: an expanded neural framework for the processing of object quality
Kravitz, D. J., Saleem, K. S., Baker, C. I., Ungerleider, L. G., & Mishkin, M. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2012.10.011 · Cited by 1,204
Categorical Representation of Visual Stimuli in the Primate Prefrontal Cortex
Freedman, D. J., Riesenhuber, M., Poggio, T., & Miller, E. K. (2001) · Science · doi:10.1126/science.291.5502.312 · Cited by 1,123
Generating Coherent Patterns of Activity from Chaotic Neural Networks
Sussillo, D., & Abbott, L. F. (2009) · Neuron · doi:10.1016/j.neuron.2009.07.018 · Cited by 1,092
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089 · cited 1×
“Shuffled MNIST Experiment 3 tasks Accuracy (%) 10 tasks Accuracy (%) 100 tasks Accuracy (%) SGD#[14] 71.”
MethodWord learning as Bayesian inference.
Xu, F., & Tenenbaum, J. B. (2007) · Psychological Review · doi:10.1037/0033-295x.114.2.245 · Cited by 1,061
The Frontal Lobes and Voluntary Action
Passingham, R. (1993) · doi:10.1093/oso/9780198521853.001.0001 · Cited by 970
Ratcliff, R. (1990) · Psychological Review · doi:10.1037/0033-295x.97.2.285 · Cited by 939
Fast Readout of Object Identity from Macaque Inferior Temporal Cortex
Hung, C. P., Kreiman, G., Poggio, T., & DiCarlo, J. J. (2005) · Science · doi:10.1126/science.1117593 · Cited by 854
The intelligibility of speech as a function of the context of the test materials.
Miller, G. A., Heise, G. A., & Lichten, W. (1951) · Journal of Experimental Psychology · doi:10.1037/h0062491 · Cited by 770
What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated
Kumaran, D., Hassabis, D., & McClelland, J. L. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.05.004 · Cited by 759
Branch-specific dendritic Ca2+ spikes cause persistent synaptic plasticity
Cichon, J., & Gan, W. (2015) · Nature · doi:10.1038/nature14251 · Cited by 527
Cortical information flow during flexible sensorimotor decisions
Siegel, M., Buschman, T. J., & Miller, E. K. (2015) · Science · doi:10.1126/science.aab0551 · Cited by 448
The Frontal Cortex-Basal Ganglia System in Primates
Wise, S. P., Murray, E. A., & Gerfen, C. R. (1996) · Critical Reviews in Neurobiology · doi:10.1615/critrevneurobiol.v10.i3-4.30 · Cited by 437
Progress & Compress: A scalable framework for continual learningPreprint
Schwarz, J., Luketina, J., Czarnecki, W. M., Grabska‐Barwińska, A., Teh, Y. W., Pascanu, R., & Hadsell, R. T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.06370 · Cited by 422 · cited 4×
“P If the capacity limit of the entire network is finally approached, two solutions can be considered: 1) introduction of a larger α or the forgetting factor used in RLS [16] and online EWC [50]; and 2) addition of more layer(s), e.g., CDP module (see below for details), to provide more space to preserve previously lear…”
InfluentialICDAR 2013 Chinese Handwriting Recognition Competition
Yin, F., Wang, Q., Zhang, X.-Y., & Liu, C. (2013) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2013.218 · Cited by 324 · cited 1×
“We found that a classifier trained with the OWM could learn to recognize all 3,755 characters sequentially, with a final accuracy ∼ 92% closely approaching the results obtained in human performance when recognizing handwritten Chinese characters (∼ 96%) [28].”
ResultMiller, E. K. (1999) · Neuron · doi:10.1016/s0896-6273(00)80673-x · Cited by 291
Anatomy of the Prefrontal Cortex
Fuster, J. M. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-407815-4.00002-7 · Cited by 282
Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
Masse, N. Y., Grant, G. D., & Freedman, D. J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1803839115 · Cited by 278
What helps where – and why? Semantic relatedness for knowledge transfer
Rohrbach, M., Stark, M. M., Szarvas, G., Gurevych, I., & Schiele, B. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540121 · Cited by 273
Microstructural proliferation in human cortex is coupled with the development of face processing
Gomez, J., Barnett, M., Natu, V. S., Mezer, A., Palomero‐Gallagher, N., Weiner, K. S., Amunts, K., Zilles, K., & Kalanit Grill-spector, A. (2017) · Science · doi:10.1126/science.aag0311 · Cited by 176
Optimal filtering algorithms for fast learning in feedforward neural networks
Shah, S., Palmieri, F., & Datum, M. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80139-x · Cited by 172
Chinese Handwriting Recognition Contest 2010
Liu, C., Yin, F., Wang, D.-H., & Wang, Q. (2010) · Chinese Conference on Pattern Recognition · doi:10.1109/ccpr.2010.5659229 · Cited by 47
Dimensionality of Object Representations in Monkey Inferotemporal Cortex
Lehky, S. R., Kiani, R., Esteky, H., & Tanaka, K. (2014) · Neural Computation · doi:10.1162/neco_a_00648 · Cited by 44
The Role of Prefrontal Mixed Selectivity in Cognitive Control
Cardenas, A. R., & Viswanathan, P. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.1816-16.2016 · Cited by 8
Beijixiong3510 (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3346079 · Cited by 1
Research works printed but not linked: 15
- Newell, A. Unified Theories of Cognition (Harvard Univ. Press, 1994).
- Haykin, S. S. Adaptive Filter theory (Pearson Education India, 2008).
- Singhal, S. & Wu, L. Training feed-forward networks with the extended kalman algorithm. In International Conference on Acoustics, Speech, and Signal Processing 1187–1190 (IEEE, 1989).
- Jaeger, H. Controlling recurrent neural networks by conceptors. Preprint at https://arxiv.org/abs/1403.3369 (2014). · arXiv
- Nair, V. & Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In International Conference on Machine Learning 807–814 (PMLR, 2010).
- Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W. & Zhang, B.-T. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems 4652–4662 (Curran Associates, 2017).
- Zenke, F., Poole, B. & Ganguli, S. Continual learning through synaptic intelligence. In International Conference on Machine Learning 6072–6082 (PMLR, 2017).
- Řehůřek, R. & Sojka, P. Software framework for topic modelling with large corpora. Proc. LREC 2010 Workshop on New Challenges for NLP Frameworks 45–50 (ELRA, 2010).
- Shin, H., Lee, J. K., Kim, J. & Kim, J. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems 2990–2999 (Curran Associates, 2017).
- Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 3320–3328 (Curran Associates, 2014).
- Hinton, G., Vinyals, O. & Dean, J. Distilling the knowledge in a neural network. Preprint at https://arxiv.org/abs/1503.02531 (2015). · arXiv
- Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. Thirteenth International Conference on Artificial Intelligence and Statistics 249–256 (Microtome, 2010).
- Nair, V. & Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proc. 27th International Conference on Machine Learning (ICML-10) 807–814 (PMLR, 2010).
- Srivastava, R. K., Masci, J., Kazerounian, S., Gomez, F. & Schmidhuber, J. Compete to compute. In Advances in Neural Information Processing Systems 2310–2318 (Curran Associates, 2013).
- Hu, W. et al. Overcoming catastrophic forgetting via model adaptation. In International Conference on Learning Representations (ICLR, 2019).
Cited by
Sorted by citations
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113
“Alternatively, gradient projection can also be performed without storing old training samples [51], [65], [66], [67].”
BackgroundThree types of incremental learning
van de Ven, G. M., Tuytelaars, T., & Tolias, A. S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00568-3 · Cited by 665
“2 shows how Split MNIST, which is a popular toy problem for continual learning 27,28,31,32 , can be performed according to each of the three scenarios.”
If deep learning is the answer, what is the question?
Saxe, A., Nelli, S., & Summerfield, C. (2021) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-00395-8 · Cited by 397
“Hebbian learning might encourage the formation of orthogonal neural codes for different temporal contexts 133 , which in turn enables tasks to be learned in different neural subspaces 132 .”
BackgroundClass-Incremental Learning: A Survey
Zhou, D.-W., Wang, Q.-W., Qi, Z.-H., Ye, H.-J., Zhan, D.-C., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429383 · Cited by 376
“OWM [56] only allows weight modification in the direction orthogonal to the subspace spanned by all previously learned inputs.”
BackgroundInfluentialOrthogonal Gradient Descent for Continual LearningPreprint
Farajtabar, M., Azizan, N., Mott, A., & Li, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07104 · Cited by 184
Metadata
- Authors
- Zeng, G., Chen, Y., Cui, B., & Yu, S.
- Affiliations
- Chinese Academy of Sciences (CN)
- ORCID
- Yu, Shan
- Venue
- Nature Machine Intelligence
- Published
- 2019-08-09
- Volume
- 1
- Issue
- 8
- Pages
- 364-372
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W2895699061
- Cited by
- 309
- References
- 41
- Citations by year
- 2026: 51 · 2025: 63 · 2024: 54 · 2023: 52 · 2022: 23 · 2021: 39 · 2020: 21 · 2019: 6
- FWCI
- 10.27
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Nature Machine Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- continual learning, catastrophic forgetting, context-dependent processing, neural network modules, multi-task learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
