The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Learning Complex, Extended Sequences Using the Principle of History Compression
Abstract
Previous neural network learning algorithms for sequence processing are computationally expensive and perform poorly when it comes to long time lags. This paper first introduces a simple principle for reducing the descriptions of event sequences without loss of information. A consequence of this principle is that only unexpected inputs can be relevant. This insight leads to the construction of neural architectures that learn to “divide and conquer” by recursively decomposing sequences. I describe two architectures. The first functions as a self-organizing multilevel hierarchy of recurrent networks. The second, involving only two recurrent networks, tries to collapse a multilevel predictor hierarchy into a single recurrent net. Experiments show that the system can require less computation per time step and many fewer training sequences than conventional training algorithms for recurrent nets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories
Williams, R. J., & Peng, J. (1990) · Neural Computation · doi:10.1162/neco.1990.2.4.490 · Cited by 577
Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.131 · Cited by 357
Self-citationExperimental Analysis of the Real-time Recurrent Learning Algorithm
WILLIAMS, R. J., & ZIPSER, D. (1989) · Connection Science · doi:10.1080/09540098908915631 · Cited by 292
Learning Factorial Codes by Predictability Minimization
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.6.863 · Cited by 264
Self-citationSchmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.2.243 · Cited by 130
Self-citationRing, M. B. (1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-200-7.50071-4 · Cited by 26
Cited by
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Self-citationDeep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Self-citationSupervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836
Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06228 · Cited by 1,246
Self-citationLearning long-term dependencies in NARX recurrent neural networks
Lin, T., Horne, B. G., Tino, P., & Giles, C. L. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.548162 · Cited by 784
Formal theory of creativity, fun, and intrinsic motivation (1990–2010)
Schmidhuber, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2056368 · Cited by 747
Self-citationLSTM recurrent networks learn simple context-free and context-sensitive languages
Gers, F. A., & Schmidhuber, E. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.963769 · Cited by 722
Human-level performance in 3D multiplayer games with population-based reinforcement learning
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. (2019) · Science · doi:10.1126/science.aau6249 · Cited by 662
(2014) · International Conference on Learning Representations · Cited by 634
An Introductory Review of Deep Learning for Prediction Models With Big Data
Emmert-Streib, F., Yang, Z., Feng, H., Tripathi, S., & Dehmer, M. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00004 · Cited by 604
Gated Feedback Recurrent Neural Networks
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.02367 · Cited by 583
“…vector h t which is a function of all the previous symbols x 1 , · · · , x t − 1 and current symbol x t : This approach of using a neural network to model a probability distribution over sequences is widely used, for instance, in language modeling (see, e.g., Bengio et al., 2001; Mikolov, 2012).”
MethodGradient calculations for dynamic recurrent neural networks: a survey
Pearlmutter, B. A. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.410363 · Cited by 582
An application of recurrent nets to phone probability estimation
Robinson, A. J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279192 · Cited by 441
Hierarchical Multiscale Recurrent Neural Networks
Chung, J., Ahn, S., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.01704 · Cited by 386
Opinion Mining with Deep Recurrent Neural Networks
Irsoy, O., & Cardie, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1080 · Cited by 384
“Stacked recurrent neural networks have been proposed as a way of constructing deep RNNs (Schmidhuber, 1992; El Hihi and Bengio, 1995).”
MethodSagheer, A., & Kotb, M. (2019) · Scientific Reports · doi:10.1038/s41598-019-55320-6 · Cited by 351
Sequential User-based Recurrent Neural Network Recommendations
Donkers, T., Loepp, B., & Ziegler, J. (2017) · Proceedings of the Eleventh ACM Conference on Recommender Systems · doi:10.1145/3109859.3109877 · Cited by 359
Deep Learning Advances in Computer Vision with 3D Data: A Survey
Ioannidou, A., Chatzilari, E., Nikolopoulos, S., & Kompatsiaris, I. (2018) · ACM Computing Surveys · doi:10.1145/3042064 · Cited by 327
A Review on Deep Learning Techniques for Video Prediction
Oprea, S., Martinez-Gonzalez, P., Garcia-Garcia, A., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., & Argyros, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3045007 · Cited by 301
Deep learning for time series forecasting: The electric load case
Gasparin, A., Lukovic, S., & Alippi, C. (2022) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12060 · Cited by 293
Nematus: a Toolkit for Neural Machine Translation
Sennrich, R., Firat, O., Cho, K., Birch, A., Haddow, B., Hitschler, J., Junczys-Dowmunt, M., Läubli, S., Miceli Barone, A. V., Mokry, J., et al. (2017) · Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-3017 · Cited by 291
Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts
Schmidhuber, J. (2006) · Connection Science · doi:10.1080/09540090600768658 · Cited by 268
Self-citationLearning Factorial Codes by Predictability Minimization
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.6.863 · Cited by 264
Self-citationTani, J., & Nolfi, S. (1999) · Neural Networks · doi:10.1016/s0893-6080(99)00060-x · Cited by 259
Ha, D., & Schmidhuber, J. (2018) · Zenodo · doi:10.5281/zenodo.1207631 · Cited by 187
Self-citationOn the Binding Problem in Artificial Neural Networks
Greff, K., van Steenkiste, S., & Schmidhuber, J. (2020) · arXiv · doi:10.48550/arxiv.2012.05208 · Cited by 102
“Alternatively, Bengio (2019) argues for the importance of a low-dimensional ‘conscious state’ (working memory) composed of largely independent units of abstraction that can be selected via attention (perhaps reminiscient of Schmidhuber, 1992b).”
BackgroundInfluentialSelf-citationNeck sensor-supported hyoid bone movement tracking during swallowing
Mao, S., Zhang, Z., Khalifa, Y., Donohue, C., Coyle, J. L., & Sejdic, E. (2019) · Royal Society Open Science · doi:10.1098/rsos.181982 · Cited by 47
Human-Timescale Adaptation in an Open-Ended Task Space
Adaptive Agent Team, Bauer, J., Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.07608 · Cited by 19
“For the first four billion steps of training, we use an additional distillation loss (Schmidhuber, 1992; Schmitt et al., 2018; Czarnecki et al., 2019) to guide AdA’s learning with the policy of a pre-trained teacher, in a process known as kickstarting; iterating this process leads to a generational…”
MethodTandem decoding of children's speech for keyword detection in a child-robot interaction scenario
Wöllmer, M., Schuller, B., Batliner, A., Steidl, S., & Seppi, D. (2011) · ACM Transactions on Speech and Language Processing · doi:10.1145/1998384.1998386 · Cited by 18
Sparse recovery of time-frequency representations via recurrent neural networks
Khalifa, Y., Zhang, Z., & Sejdic, E. (2017) · 2017 22nd International Conference on Digital Signal Processing (DSP) · doi:10.1109/icdsp.2017.8096132 · Cited by 5
Modern Machine Learning Methods for Time Series Analysis
Huang, C., & Petukhina, A. (2022) · Statistics and Computing · doi:10.1007/978-3-031-13584-2_10 · Cited by 2
A survey on neuro-mimetic deep learning via predictive coding
Salvatori, T., Mali, A., Buckley, C. L., Lukasiewicz, T., Rao, R. P. N., Friston, K., & Ororbia, A. (2026) · Neural Networks · doi:10.1016/j.neunet.2025.108161 · Cited by 2
Metadata
- Authors
- Schmidhuber, J.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 1992-03
- Volume
- 4
- Issue
- 2
- Pages
- 234-242
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2036317923
- Cited by
- 398
- References
- 6
- FWCI
- 2.68
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.