Lifelong learning of human actions with deep neural network self-organization
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Lifelong learning is fundamental in autonomous robotics for the acquisition and fine-tuning of knowledge through experience. However, conventional deep neural models for action recognition from videos do not account for lifelong learning but rather learn a batch of training data with a predefined number of action classes and samples. Thus, there is the need to develop learning systems with the ability to incrementally process available perceptual cues and to adapt their responses over time. We propose a self-organizing neural architecture for incrementally learning to classify human actions from video sequences. The architecture comprises growing self-organizing networks equipped with recurrent neurons for processing time-varying patterns. We use a set of hierarchically arranged recurrent networks for the unsupervised learning of action representations with increasingly large spatiotemporal receptive fields. Lifelong learning is achieved in terms of prediction-driven neural dynamics in which the growth and the adaptation of the recurrent networks are driven by their capability to reconstruct temporally ordered input sequences. Experimental results on a classification task using two action benchmark datasets show that our model is competitive with state-of-the-art methods for batch learning also when a significant number of sample labels are missing or corrupted during training sessions. Additional experiments show the ability of our model to adapt to non-stationary input avoiding catastrophic interference.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Kohonen, T. (1990) · Proceedings of the IEEE · doi:10.1109/5.58325 · Cited by 8,414
Neurogenesis in the adult human hippocampus
Eriksson, P. S., Perfilieva, E., Björk-Eriksson, T., Alborn, A.-M., Nordborg, C., Peterson, D. A., & Gage, F. H. (1998) · Nature Medicine · doi:10.1038/3305 · Cited by 6,369
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 2×
“This principle roughly resembles schemes of predictive coding (Huang & Rao, 2011; Rao & Ballard, 1999) proposing that the visual cortex can be modeled as a hierarchical network with reciprocal connections where top-down feedback connections from higher-order cortical areas convey predictions of…”
BackgroundMcClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995) · Psychological Review · doi:10.1037/0033-295x.102.3.419 · Cited by 5,256
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Hierarchical models of object recognition in cortex
Riesenhuber, M., & Poggio, T. (1999) · Nature Neuroscience · doi:10.1038/14819 · Cited by 3,309 · cited 1×
“Mechanisms of MAX-pooling in hierarchical models are also compatible with neurophysiological data (Riesenhuber & Poggio, 1999), e.g. showing that the response of neurons in the inferotemporal (IT) cortex is dominated by the stimulus producing the highest firing rate (Sato, 1989).”
ResultUnderstanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Adult Neurogenesis in the Mammalian Brain: Significant Answers and Significant Questions
Ming, G.-L., & Song, H. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.001 · Cited by 2,651 · cited 1×
“However, its process of neural growth does not resemble biologically plausible mechanisms of neurogenesis (e.g. Eriksson, Perfilieva, BjorkEriksson, Alborn, Nordborg, Peterson, & Gage, 1998; Ming & Song, 2011).”
BackgroundDeep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504 · cited 1×
“…obtain invariant responses by alternating layers of feature detectors and nonlinear pooling neurons using the maximum (MAX) operation, which has been shown to achieve higher feature specificity and more robust invariance with respect to linear summation (see Guo et al., 2016 for a review).”
MethodUsing goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
Competitive learning: From interactive activation to adaptive resonance
Grossberg, S. (1987) · Cognitive Science · doi:10.1111/j.1551-6708.1987.tb00862.x · Cited by 1,471 · cited 1×
“When dealing with non-stationary data, processing systems should account for the integration of new information while preserving existing knowledge (Grossberg, 1987).”
BackgroundEncyclopedia of the Sciences of Learning
(2012) · doi:10.1007/978-1-4419-1428-6 · Cited by 1,091
Development of the brain depends on the visual environment
BLAKEMORE, C., & COOPER, G. F. (1970) · Nature · doi:10.1038/228477a0 · Cited by 1,005 · cited 1×
“A number of studies suggested that visual input is crucial for normal cortical organization, since the cortex tunes itself to the input distribution (Blakemore & Cooper, 1970; Blakemore & Van Sluyters, 1975; Hirsch & Spinelli, 1970; Sengpiel & Stawinski, 1999).”
BackgroundNeural mechanisms for the recognition of biological movements
Giese, M. A., & Poggio, T. (2003) · Nature Reviews Neuroscience · doi:10.1038/nrn1057 · Cited by 955 · cited 5×
“…of body action features in two distinct neural pathways and their subsequent integration in the STS are in line with neurophysiological evidence (Giese & Poggio, 2003; Giese & Rizzolatti, 2015), our model is a strong simplification with respect to biological mechanisms where the two streams…”
BackgroundResultInfluentialA hierarchy of temporal receptive windows in human cortex
Hasson, U., Yang, E., Vallines, I., Heeger, D. J., & Rubin, N. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.5487-07.2008 · Cited by 874
Topographic mapping of a hierarchy of temporal receptive windows using a narrated story
Lerner, Y., Honey, C. J., Silbert, L. J., & Hasson, U. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.3684-10.2011 · Cited by 834
Huang, Y., & Rao, R. P. N. (2011) · WIREs Cognitive Science · doi:10.1002/wcs.142 · Cited by 470
Johnson, J. S., & Newport, E. L. (1991) · Cognition · doi:10.1016/0010-0277(91)90054-8 · Cited by 416
Tani, J., & Nolfi, S. (1999) · Neural Networks · doi:10.1016/s0893-6080(99)00060-x · Cited by 259
Self-citationCompetitive Hebbian Learning Rule Forms Perfectly Topology Preserving Maps
Martinetz, T. (1993) · ICANN ’93 · doi:10.1007/978-1-4471-2063-6_104 · Cited by 256 · cited 1×
“In this case, the learning parameters are kept fixed and map topology develops through competitive Hebbian learning (Martinetz, 1993).”
BackgroundThe temporal paradox of Hebbian learning and homeostatic plasticity
Zenke, F., Gerstner, W., & Ganguli, S. (2017) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.03.015 · Cited by 201
MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation
Jain, A., Tompson, J., LeCun, Y., & Bregler, C. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16808-1_21 · Cited by 142
Cited by
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117
Self-citationDeep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2904897 · Cited by 1,768
Ahmad, S., Shakeel, I., Mehfuz, S., & Ahmad, J. (2023) · Computer Science Review · doi:10.1016/j.cosrev.2023.100568 · Cited by 197
Semantic Communication: A Survey on Research Landscape, Challenges, and Future Directions
Getu, T. M., Kaddoum, G., & Bennis, M. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3520707 · Cited by 110
Metadata
- Authors
- Parisi, G. I., Tani, J., Weber, C., & Wermter, S.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2017-12
- Volume
- 96
- Pages
- 137-149
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2754791538
- Cited by
- 129
- References
- 24
- FWCI
- 5.95
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.