Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
We present a method for learning nonlinear systems, echo state networks (ESNs). ESNs employ artificial recurrent neural networks in a way that has recently been proposed independently as a learning mechanism in biological brains. The learning method is computationally efficient and easy to use. On a benchmark task of predicting a chaotic time series, accuracy is improved by a factor of 2400 over previous techniques. The potential for engineering applications is illustrated by equalizing a communication channel, where the signal error rate is improved by two orders of magnitude.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Although work in artificial intelligence and machine-learning research has shown that similar or even more complex outcomes can be achieved with a variety of techniques, including deep neural networks with trained weights (Gibney, 2015), reservoir networks with a trained output mapping (Maass et al., 2002;Jaeger & Haas, 2004), networks that are evolved using a genetic algorithm (Iizuka & Di Paolo,”
From A potential mechanism for Gibsonian resonance: behavioral entrainment emerges from local homeostasis in an unsupervised reservoir network (2024) · cites this paper as Jaeger & Haas, 2004
In the citing paper
“Therefore, a sequence production solely by a recurrent neural network can be unstable against small fluctuations in activities and unwanted noises (Jaeger and Haas, 2004).”
From The Cerebro-Cerebellum as a Locus of Forward Model: A Review (2020) · cites this paper as Jaeger and Haas, 2004
Citations
Cites
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369
Oscillation and Chaos in Physiological Control Systems
Mackey, M. C., & Glass, L. (1977) · Science · doi:10.1126/science.267326 · Cited by 4,135
Maass, W., Natschläger, T., & Markram, H. (2002) · Neural Computation · doi:10.1162/089976602760407955 · Cited by 4,025
Real-time prediction of hand trajectory by ensembles of cortical neurons in primates
Wessberg, J., Stambaugh, C. R., Kralik, J. D., Beck, P. D., Laubach, M., Chapin, J. K., Kim, J., Biggs, S. J., Srinivasan, M. A., & Nicolelis, M. A. L. (2000) · Nature · doi:10.1038/35042582 · Cited by 1,487
'Neural-gas' network for vector quantization and its application to time-series prediction
Martinetz, T. M., Berkovich, S. G., & Schulten, K. J. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.238311 · Cited by 1,431
Approximation of dynamical systems by continuous time recurrent neural networks
Funahashi, K.-I., & Nakamura, Y. (1993) · Neural Networks · doi:10.1016/s0893-6080(05)80125-x · Cited by 980
A new evolutionary system for evolving artificial neural networks
Yao, X., & Liu, Y. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.572107 · Cited by 874
Applying LSTM to Time Series Predictable through Time-Window Approaches
Gers, F. A., Eck, D., & Schmidhuber, J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44668-0_93 · Cited by 334
Computational neuroscience: a comprehensive approach
(2004) · Choice Reviews Online · doi:10.5860/choice.41-5284 · Cited by 279
Research works printed but not linked: 10
- K. Doya, in The Handbook of Brain Theory and Neural Networks, M. A. Arbib, Ed. (MIT Press, Cambridge, MA, 1995), pp. 796–800.
- J. Vesanto in Proc. WSOM'97 (1997); www.cis.hut.fi/projects/monitor/publications/papers/wsom97.ps.
- L. Chudy, I. Farkas, Neural Network World8, 481 (1998).
- H. Bersini M. Birattari G. Bontempi in Proc. IEEE World Congr. on Computational Intelligence (IJCNN'98) (1997) pp. 2102–2106; ftp://iridia.ulb.ac.be/pub/lazy/papers/IridiaTr1997-13_2.ps.gz.
- V. J. Mathews, J. Lee, in Advanced Signal Processing: Algorithms, Architectures, and Implementations V (Proc. SPIE Vol. 2296), (SPIE, San Diego, CA, 1994), pp. 317–327.
- J. Hertzberg H. Jaeger F. Schönherr in Proc. 15th Europ. Conf. on Art. Int. (ECAI 02) F. van Harmelen Ed. (IOS Press Amsterdam 2002) pp. 708–712; www.ais.fhg.de/∼schoenhe/papers/ECAI02.pdf.
- H. Jaeger, in Advances in Neural Information Processing Systems 15, S. Becker, S. Thrun, K. Obermayer, Eds. (MIT Press, Cambridge, MA, 2003) pp. 593–600.
- G. E. Hinton, in Parallel Models of Associative Memory, G. E. Hinton, J. A. Anderson, Eds. (Erlbaum, Hillsdale, NJ, 1981), pp. 161–187.
- M. Kawato, in The Handbook of Brain Theory and Neural Networks, M. Arbib, Ed. (MIT Press, Cambridge, MA, 1995), pp. 172–178.
- G. B. Stanley, Neurocomputing38–40, 1703 (2001).
Cited by
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
“In fact, some popular RNN algorithms restricted credit assignment to a single step backwards (Elman, 1990; Jordan, 1986, 1997), also inmore recent studies (Jaeger, 2001, 2004; Maass et al., 2002).”
BackgroundMethodResultInfluentialOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
“Echo State Networks (Jaeger and Haas, 2004) avoid the exploding and vanishing gradients problem by not learning Wrec and Win.”
BackgroundResult(2013) · Cited by 3,740
Deep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342
Reservoir computing approaches to recurrent neural network training
Lukoševičius, M., & Jaeger, H. (2009) · Computer Science Review · doi:10.1016/j.cosrev.2009.03.005 · Cited by 2,956
“Demonstrations of RLS are presented in [17, 43].”
BackgroundInfluentialSelf-citationMachine learning and the physical sciences
Carleo, G., Cirac, I., Cranmer, K., Daudet, L., Schuld, M., Tishby, N., Vogt-Maranto, L., & Zdeborová, L. (2019) · Reviews of Modern Physics · doi:10.1103/revmodphys.91.045002 · Cited by 2,417
Recent advances in physical reservoir computing: A review
Tanaka, G., Yamane, T., Héroux, J. B., Nakane, R., Kanazawa, N., Takeda, S., Numata, H., Nakano, D., & Hirose, A. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.03.005 · Cited by 2,146
“Reservoir computing (RC) is originally an RNN-based framework and is therefore suitable for temporal/sequential information processing (Jaeger & Haas, 2004).”
BackgroundMethodInfluentialInformation processing using a single dynamical node as complex system
Appeltant, L., Soriano, M. C., Van der Sande, G., Danckaert, J., Massar, S., Dambre, J., Schrauwen, B., Mirasso, C. R., & Fischer, I. (2011) · Nature Communications · doi:10.1038/ncomms1476 · Cited by 1,716
“Th e training of the read-out follows the standard procedure for RC 5,7 .”
BackgroundMethodMitarai, K., Negoro, M., Kitagawa, M., & Fujii, K. (2018) · Physical Review A · doi:10.1103/physreva.98.032309 · Cited by 1,639
The importance of mixed selectivity in complex cognitive tasks
Rigotti, M., Barak, O., Warden, M. R., Wang, X.-J., Daw, N. D., Miller, E. K., & Fusi, S. (2013) · Nature · doi:10.1038/nature12160 · Cited by 1,639
Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors
Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., & Schaal, S. (2013) · Neural Computation · doi:10.1162/neco_a_00393 · Cited by 1,598
An Empirical Exploration of Recurrent Network Architectures
Józefowicz, R., Zaremba, W., & Sutskever, I. (2015) · International Conference on Machine Learning · Cited by 2,051
A Survey on Deep Learning: Algorithms, Techniques, and Applications
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., & Iyengar, S. S. (2019) · ACM Computing Surveys · doi:10.1145/3234150 · Cited by 1,403
“One method attempts to overcome this challenge by using reservoir computing [71].”
MethodNeuromorphic computing with nanoscale spintronic oscillators
Torrejon, J., Riou, M., Araujo, F. A., Tsunegi, S., Khalsa, G., Querlioz, D., Bortolotti, P., Cros, V., Yakushiji, K., Fukushima, A., et al. (2017) · Nature · doi:10.1038/nature23011 · Cited by 1,374
Artificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial
Chen, M., Challita, U., Saad, W., Yin, C., & Debbah, M. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2926625 · Cited by 1,117
“In fact, ESNs can be also trained by using online methods such as the least mean squares (LMS) algorithm [75], or the recursive least squares (RLS) algorithm [76].”
MethodDeep learning for cyber security intrusion detection: Approaches, datasets, and comparative study
Ferrag, M. A., Maglaras, L., Moschoyiannis, S., & Janicke, H. (2020) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2019.102419 · Cited by 1,130
“There are many types of RNNs such as Elman networks proposed by [140], Jordan networks proposed by [141] and Echo State networks proposed by [142].”
MethodPathak, J., Hunt, B., Girvan, M., Lu, Z., & Ott, E. (2018) · Physical Review Letters · doi:10.1103/physrevlett.120.024102 · Cited by 1,303
Generating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · International Conference on Machine Learning · Cited by 1,783
Physics for neuromorphic computing
Marković, D., Mizrahi, A., Querlioz, D., & Grollier, J. (2020) · Nature Reviews Physics · doi:10.1038/s42254-020-0208-2 · Cited by 1,176
Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control
Brunton, S. L., & Kutz, J. N. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108380690 · Cited by 1,154
An experimental unification of reservoir computing methods
Verstraeten, D., Schrauwen, B., D’Haene, M., & Stroobandt, D. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.003 · Cited by 1,135
Generating Coherent Patterns of Activity from Chaotic Neural Networks
Sussillo, D., & Abbott, L. F. (2009) · Neuron · doi:10.1016/j.neuron.2009.07.018 · Cited by 1,092
“As stated in the Introduction, Jaeger and Haas (2004) eliminated the problem of delayed effects by clamping feedback during learning.”
InfluentialReservoir computing using dynamic memristors for temporal information processing
Du, C., Cai, F., Zidan, M. A., Ma, W., Lee, S. H., & Lu, W. D. (2017) · Nature Communications · doi:10.1038/s41467-017-02337-y · Cited by 1,015
“Following a similar approach in a prior study 35 , we ignore the fi rst 50 initial data points in the transient period and train the readout function weights w i ( i = 1, … 90) using the last 250 points in the training sequence using simple linear regression.”
MethodRobotics: Science and Systems VIII
(2013) · The MIT Press eBooks · doi:10.7551/mitpress/9816.001.0001 · Cited by 1,028
Optimization and applications of echo state networks with leaky- integrator neurons
Jaeger, H., Lukoševičius, M., Popovici, D., & Siewert, U. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.016 · Cited by 948
Self-citationParallel photonic information processing at gigabyte per second data rates using transient states
Brunner, D., Soriano, M. C., Mirasso, C. R., & Fischer, I. (2013) · Nature Communications · doi:10.1038/ncomms2368 · Cited by 938
“This requirement corresponds to the approximation property in machine learning 4,7 .”
BackgroundMethodState-dependent computations: spatiotemporal processing in cortical networks
Buonomano, D. V., & Maass, W. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2558 · Cited by 926
“In some of the models, the encoding of past stimuli was based solely on ongoing changes in the active state produced by the recurrent architectur”
BackgroundExperimental demonstration of reservoir computing on a silicon photonics chip
Vandoorne, K., Mechet, P., Van Vaerenbergh, T., Fiers, M., Morthier, G., Verstraeten, D., Schrauwen, B., Dambre, J., & Bienstman, P. (2014) · Nature Communications · doi:10.1038/ncomms4541 · Cited by 892
“Internally, the time scale of the network is set by the optical delay, which is easily scalable to perform Boolean operations in the range of several hundreds of Gbit s (cid:2) 1 .”
BackgroundMethodInfluentialA Practical Guide to Applying Echo State Networks
Lukoševičius, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_36 · Cited by 867
“Demonstrations of RLS for ESNs are presented in [11,43].”
BackgroundMethodOptoelectronic Reservoir Computing
Paquot, Y., Duport, F., Smerieri, A., Dambre, J., Schrauwen, B., Haelterman, M., & Massar, S. (2012) · Scientific Reports · doi:10.1038/srep00287 · Cited by 783
“(4) and (5), are derived in the supplementary material; the various stages of processing of the reservoir nodes and inputs are shown in Fig.”
BackgroundLarger, L., Soriano, M. C., Brunner, D., Appeltant, L., Gutierrez, J. M., Pesquera, L., Mirasso, C. R., & Fischer, I. (2012) · Optics Express · doi:10.1364/oe.20.003241 · Cited by 864
Remaining useful life estimation of engineered systems using vanilla LSTM neural networks
Wu, Y., Yuan, M., Dong, S., Lin, L., & Liu, Y. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.05.063 · Cited by 789
Why neurons mix: high dimensionality for higher cognition
Fusi, S., Miller, E. K., & Rigotti, M. (2016) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2016.01.010 · Cited by 766
“Moreover, high-dimensional neural representations are also a crucial component of dynamical models of recurrent neural networks like echo state machines or liquid state machines [35–40].”
BackgroundFrom the neuron doctrine to neural networks
Yuste, R. (2015) · Nature Reviews Neuroscience · doi:10.1038/nrn3962 · Cited by 754
“Neural circuits could be constantly changing, as if they were a ‘liquid state’ machine 109,119 .”
BackgroundChen, M., Mozaffari, M., Saad, W., Yin, C., Debbah, M., & Hong, C. S. (2017) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2017.2680898 · Cited by 742
“Echo state networks are a special type of recurrent neural networks designed for performing non-linear systems forecasting [30].”
BackgroundMinimum Complexity Echo State Network
Rodan, A., & Tino, P. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2089641 · Cited by 710
Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning
Feng, G., Huang, G.-B., Lin, Q., & Gay, R. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2009.2024147 · Cited by 654
“Different from recurrent neural networks, the hidden layers in feedforword neural networks have no lateral connections and/or cyclic connections [1]–[3].”
BackgroundTemporal data classification and forecasting using a memristor-based reservoir computing system
Moon, J., Ma, W., Shin, J. H., Cai, F., Du, C., Lee, S. H., & Lu, W. D. (2019) · Nature Electronics · doi:10.1038/s41928-019-0313-3 · Cited by 645
Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data
Pathak, J., Lu, Z., Hunt, B. R., Girvan, M., & Ott, E. (2017) · Chaos: An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5010300 · Cited by 620
Next generation reservoir computing
Gauthier, D. J., Bollt, E., Griffith, A., & Barbosa, W. A. S. (2021) · Nature Communications · doi:10.1038/s41467-021-25801-2 · Cited by 582
“Here, we demonstrate that it is easy to design a well-performing NVAR for three challenging RC benchmark problems: (1) forecasting the shortterm dynamics; (2) reproducing the long-term climate of a”
MethodIntegration of nanoscale memristor synapses in neuromorphic computing architectures
Indiveri, G., Linares-Barranco, B., Legenstein, R., Deligeorgis, G., & Prodromakis, T. (2013) · Nanotechnology · doi:10.1088/0957-4484/24/38/384010 · Cited by 613
“For example, Reservoir Computing (RC, also known under the terms Liquid State Machines and Echo State Networks) is a powerful general principle for computation and learning with complex dynamical systems such as recurrent networks of analog and spiking neurons [85, 86] or optoelectronic devices”
BackgroundA neural network that finds a naturalistic solution for the production of muscle activity
Sussillo, D., Churchland, M. M., Kaufman, M. T., & Shenoy, K. V. (2015) · Nature Neuroscience · doi:10.1038/nn.4042 · Cited by 598
Emotion, Cognition, and Mental State Representation in Amygdala and Prefrontal Cortex
Salzman, C. D., & Fusi, S. (2010) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.051508.135256 · Cited by 588
Dynamic memristor-based reservoir computing for high-efficiency temporal signal processing
(2021) · doi:10.1038/s41467-020-20692-1 · Cited by 597
Advances in photonic reservoir computing
Van der Sande, G., Brunner, D., & Soriano, M. C. (2017) · Nanophotonics · doi:10.1515/nanoph-2016-0132 · Cited by 591
“Introduced independently by Jaeger and Haas [13] and Maass et al. [14], their most striking innovation was the usage of random distributions for input connections W in and internal connections W int .”
BackgroundMethodComplex photonics: Dynamics and applications of delay-coupled semiconductors lasers
Soriano, M. C., García-Ojalvo, J., Mirasso, C. R., & Fischer, I. (2013) · Reviews of Modern Physics · doi:10.1103/revmodphys.85.421 · Cited by 582
“Computationally hard tasks, such as chaotic time series prediction (Jaeger and Haas, 2004) or speech recognition (Verstraeten et al.”
BackgroundOpening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks
Sussillo, D., & Barak, O. (2013) · Neural Computation · doi:10.1162/neco_a_00409 · Cited by 575
“Recently, there has been progress in training RNNs to perform desired behaviors (Maass, Natschläger, & Markram, 2002; Jaeger & Haas, 2004; Maass, Joshi, & Sontag, 2007; Sussillo & Abbott, 2009; Martens & Sutskever, 2011).”
BackgroundVowel recognition with four coupled spin-torque nano-oscillators
Romera, M., Talatchian, P., Tsunegi, S., Abreu Araujo, F., Cros, V., Bortolotti, P., Trastoy, J., Yakushiji, K., Fukushima, A., Kubota, H., et al. (2018) · Nature · doi:10.1038/s41586-018-0632-y · Cited by 572
An Experimental Review on Deep Learning Architectures for Time Series Forecasting
Lara-Benítez, P., Carranza-García, M., & Riquelme, J. C. (2021) · International Journal of Neural Systems · doi:10.1142/s0129065721300011 · Cited by 532
Quanvolutional neural networks: powering image recognition with quantum circuits
Henderson, M., Shakya, S., Pradhan, S., & Cook, T. (2020) · Quantum Machine Intelligence · doi:10.1007/s42484-020-00012-y · Cited by 489
All-optical reservoir computing
Duport, F., Schneider, B., Smerieri, A., Haelterman, M., & Massar, S. (2012) · Optics Express · doi:10.1364/oe.20.022783 · Cited by 502
Learning Recurrent Neural Networks with Hessian-Free Optimization
Martens, J., & Sutskever, I. (2011) · International Conference on Machine Learning · Cited by 525
Deep reservoir computing: A critical experimental analysis
Gallicchio, C., Micheli, A., & Pedrelli, L. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.089 · Cited by 519
Larger, L., Baylón-Fuentes, A., Martinenghi, R., Udaltsov, V. S., Chembo, Y. K., & Jacquot, M. (2017) · Physical Review X · doi:10.1103/physrevx.7.011015 · Cited by 509
“Among the numerous brain-inspired concepts, reservoir computing (RC) [1], also known originally as an echo state network [2] or a liquid state machine [3] (see Ref.”
BackgroundRe-visiting the echo state property
Yildiz, I. B., Jaeger, H., & Kiebel, S. J. (2012) · Neural Networks · doi:10.1016/j.neunet.2012.07.005 · Cited by 486
“Echo state networks (ESN) [8, 12] provide an architecture and supervised learning principle for recurrent neural networks (RNNs).”
BackgroundSelf-citationVlachas, P. R., Pathak, J., Hunt, B. R., Sapsis, T. P., Girvan, M., Ott, E., & Koumoutsakos, P. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.02.016 · Cited by 480
“Inthis work, we consider reservoir computers with f given by the functional form where and Larger et al. (2017, 2012) Following Jaeger and Haas (2004), where ω is a hyperparameter.”
MethodInformation Processing Capacity of Dynamical Systems
Dambre, J., Verstraeten, D., Schrauwen, B., & Massar, S. (2012) · Scientific Reports · doi:10.1038/srep00514 · Cited by 480
“In particular it provides theoretical justification for the widely used paradigm of reservoir computing with linear readout layer 20,3,21 , by proving that such systems can be universal.”
BackgroundDeep learning models for solar irradiance forecasting: A comprehensive review
Kumari, P., & Toshniwal, D. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2021.128566 · Cited by 445
Milano, G., Pedretti, G., Montano, K., Ricci, S., Hashemkhani, S., Boarino, L., Ielmini, D., & Ricciardi, C. (2022) · Nature Materials · doi:10.1038/s41563-021-01099-9 · Cited by 462
Jaeger, H. (2007) · Scholarpedia · doi:10.4249/scholarpedia.2330 · Cited by 459
Self-citationMalware classification with recurrent networks
Pascanu, R., Stokes, J. W., Sanossian, H., Marinescu, M., & Thomas, A. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178304 · Cited by 423
“Echo state networks (ESNs) have been successfully used for predicting chaotic systems [9].”
BackgroundMethodRobust timing and motor patterns by taming chaos in recurrent neural networks
Laje, R., & Buonomano, D. V. (2013) · Nature Neuroscience · doi:10.1038/nn.3405 · Cited by 454
“Indeed, a number of related theoretical frameworks (sometimes grouped under the term reservoir computing ) are to a large extent based on the notion that randomly connected recurrent networks (RRNs) can generate patterns of activity in high-dimensional space, and these patterns or trajectories can in turn be used for…”
BackgroundMethodInfluentialPhysical reservoir computing—an introductory perspective
Nakajima, K. (2020) · Japanese Journal of Applied Physics · doi:10.35848/1347-4065/ab8d4f · Cited by 422
Spintronic Nanodevices for Bioinspired Computing
Grollier, J., Querlioz, D., & Stiles, M. D. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2016.2597152 · Cited by 433
Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networksPreprint
Vlachas, P. R., Byeon, W., Koumoutsakos, P., Wan, Z. Y., & Sapsis, T. P. (2018) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 425
Reinforcement learning in a large-scale photonic recurrent neural network
Bueno, J., Maktoobi, S., Froehly, L., Fischer, I., Jacquot, M., Larger, L., & Brunner, D. (2018) · Optica · doi:10.1364/optica.5.000756 · Cited by 403
“Two hundred points of the chaotic Mackey–Glass (MG) sequence [3] are used as the injected training signal u n 1 .”
BackgroundMethodInfluentialLinking Connectivity, Dynamics, and Computations in Low-Rank Recurrent Neural Networks
Mastrogiuseppe, F., & Ostojic, S. (2018) · Neuron · doi:10.1016/j.neuron.2018.07.003 · Cited by 391
“In echo-state (Jaeger and Haas, 2004; Maass et al., 2007) and FORCE learning (Sussillo and Abbott, 2009), and similarly within the Neural Engineering Framework (Eliasmith and Anderson, 2004), computations are implemented through feedback loops from readout units to the bulk of the network.”
InfluentialLukoševičius, M., Jaeger, H., & Schrauwen, B. (2012) · KI - Künstliche Intelligenz · doi:10.1007/s13218-012-0204-5 · Cited by 392
“, [22,23,50,17,46].”
Self-citationArtificial Neural Networks for Neuroscientists: A Primer
Yang, G. R., & Wang, X.-J. (2020) · Neuron · doi:10.1016/j.neuron.2020.09.005 · Cited by 376
Emerging opportunities and challenges for the future of reservoir computing
Yan, M., Huang, C., Bienstman, P., Tino, P., Lin, W., & Sun, J. (2024) · Nature Communications · doi:10.1038/s41467-024-45187-1 · Cited by 346
“The target output are the corresponding labels which can be spoken digits 16,28,29,34,35,65,68,82 – 84 , image labels 33,35,85 – 87 , bit-symbols 16,29,68,88 – 93 and so on.”
BackgroundTraining recurrent neural networks
Hinton, G. E., & Sutskever, I. (2013) · Cited by 383
An overview of reservoir computing: theory, applications and implementations
Schrauwen, B., Verstraeten, D., & Campenhout, J. V. (2007) · Ghent University Academic Bibliography (Ghent University) · Cited by 552
Attractor reconstruction by machine learning
Lu, Z., Hunt, B. R., & Ott, E. (2018) · Chaos: An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5039508 · Cited by 374
Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks
Vlachas, P. R., Byeon, W., Wan, Z. Y., Sapsis, T. P., & Koumoutsakos, P. (2018) · Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rspa.2017.0844 · Cited by 358
“Similarly, other machine learning techniques based on Multi Layer Perceptrons (MLP) [44], Echo State Networks (ESNs) [45, 46] or radial basis functions [47, 48] have been successful, albeit only for low order dynamical systems.”
MethodThe cortex as a central pattern generator
Yuste, R., MacLean, J. N., Smith, J., & Lansner, A. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1686 · Cited by 355
Recurrent network models of sequence generation and memory
Rajan, K., Harvey, C. D., & Tank, D. W. (2016) · Neuron · doi:10.1016/j.neuron.2016.02.009 · Cited by 351
“…models typically compute the output of a network through weighted readout vectors (Buonomano and Merzenich, 1995; Maass et al., 2002; Jaeger, 2003; Jaeger and Haas, 2004) and feed the output back into the network as an additional current, leaving the synaptic weights within the dynamics-producing…”
BackgroundMethodPathak, J., Wikner, A., Fussell, R., Chandra, S., Hunt, B. R., Girvan, M., & Ott, E. (2018) · Chaos: An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5028373 · Cited by 307
Chaotic Time Series Prediction Based on a Novel Robust Echo State Network
Li, D., Han, M., & Wang, J. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2188414 · Cited by 341
“The trained ESN would behave very differently if test data change slightly away from the training data [15].”
BackgroundMethodInfluential100,000-spin coherent Ising machine
Honjo, T., Sonobe, T., Inaba, K., Inagaki, T., Ikuta, T., Yamada, Y., Kazama, T., Enbutsu, K., Umeki, T., Kasahara, R., et al. (2021) · Science Advances · doi:10.1126/sciadv.abh0952 · Cited by 341
Reservoir observers: Model-free inference of unmeasured variables in chaotic systems
Lu, Z., Pathak, J., Hunt, B., Girvan, M., Brockett, R., & Ott, E. (2017) · Chaos: An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.4979665 · Cited by 344
Harnessing Disordered-Ensemble Quantum Dynamics for Machine Learning
Fujii, K., & Nakajima, K. (2017) · Physical Review Applied · doi:10.1103/physrevapplied.8.024030 · Cited by 337
“It is reported that the computational power of ESN can be well characterised by the spectral radius of the internal connection weight matrix [18, 20, 57–59].”
BackgroundMethodInfluentialInformation processing via physical soft body
Nakajima, K., Hauser, H., Li, T., & Pfeifer, R. (2015) · Scientific Reports · doi:10.1038/srep10487 · Cited by 324
“A number of different implementations for reservoirs have been proposed, such as abstract dynamical systems for echo state networks 7,8 or models of neurons for liquid state machines 9 , the surface of water in a laminar state 14 , and nonlinear mass spring systems 15–19 .”
MethodGerstner, W., Lehmann, M., Liakoni, V., Corneil, D., & Brea, J. (2018) · Frontiers in Neural Circuits · doi:10.3389/fncir.2018.00053 · Cited by 290
“For example, memory traces hidden in the rich firing activity patterns of a recurrent network (Maass et al., 2002; Jaeger and Haas, 2004; Buonomano and Maass, 2009; Susillo and Abbott, 2009) or short-term synaptic plasticity in recurrent networks (Mongillo et al., 2008) could be involved in…”
BackgroundCheng, S., Quilodrán-Casas, C., Ouala, S., Farchi, A., Liu, C., Tandeo, P., Fablet, R., Lucor, D., Iooss, B., Brajard, J., et al. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123537 · Cited by 303
“For instance, reservoir computing approaches [247], [248] were shown to be well-suited for learning dynamical systems from data [249], [250].”
BackgroundAnalysis and Design of Echo State Networks
Ozturk, M. C., Xu, D., & Príncipe, J. C. (2007) · Neural Computation · doi:10.1162/neco.2007.19.1.111 · Cited by 308
“echo state network (ESN) by Jaeger (2001, 2002a; Jaeger & Hass, 2004 ) and the liquid state machine (LSM) by Maass (Maass, Natschl¨ ager, & Markram, 2002).,We also created W matrices with values uniformly distributed between −1 and 1 (U-ESN) and scaled to obtain a given spectral radius ( Jaeger & Hass, 2004 ).”
BackgroundMethodInfluentialMemory traces in dynamical systems
Ganguli, S., Huh, D., & Sompolinsky, H. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0804451105 · Cited by 303
“[6]”
On the computational power of circuits of spiking neurons
Maass, W., & Markram, H. (2004) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2004.04.001 · Cited by 301
“An essentially equivalent framework for real-time computing has been introduced independently by Jäger (see [19] and [20]) and applied to artificial neural network models.”
BackgroundMethodInfluentialMiglani, A., & Kumar, N. (2019) · Vehicular Communications · doi:10.1016/j.vehcom.2019.100184 · Cited by 295
Information processing in echo state networks at the edge of chaos
Boedecker, J., Obst, O., Lizier, J. T., Mayer, N. M., & Asada, M. (2012) · Theory in Biosciences · doi:10.1007/s12064-011-0146-8 · Cited by 297
“ESNs provide a specific architecture and a training procedure that aims to solve the problem of slow convergence (Jaeger, 2001a; Jaeger and Haas, 2004) of earlier recurrent neural network training algorithms.”
BackgroundTwo types of asynchronous activity in networks of excitatory and inhibitory spiking neurons
Ostojic, S. (2014) · Nature Neuroscience · doi:10.1038/nn.3658 · Cited by 303
“The radius λmax of that circle depends on network parameters (see Online Methods equation (16)).”
BackgroundRecurrent Broad Learning Systems for Time Series Prediction
Xu, M., Han, M., Chen, C. L. P., & Qiu, T. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2863020 · Cited by 284
“Remarkable results have been achieved for various network structures, such as extreme learning machines (ELMs) [25], [26] and echo state networks (ESNs) [27]–[29].”
BackgroundMethodComputing with Spiking Neuron Networks
Paugam-Moisy, H., & Bohte, S. (2012) · Handbook of Natural Computing · doi:10.1007/978-3-540-92910-9_10 · Cited by 272
Reservoir Computing Using Diffusive Memristors
Midya, R., Wang, Z., Asapu, S., Zhang, X., Rao, M., Song, W., Zhuo, Y., Upadhyay, N., Xia, Q., & Yang, J. J. (2019) · Advanced Intelligent Systems · doi:10.1002/aisy.201900084 · Cited by 279
Recurrent Neural Networks for Short-Term Load Forecasting: An Overview and Comparative Analysis
Bianchi, F. M., Maiorino, E., Kampffmeyer, M. C., Rizzi, A., & Jenssen, R. (2017) · SpringerBriefs in Computer Science · doi:10.1007/978-3-319-70338-1 · Cited by 280
“This result confirms the excellent and well-known capability of the ESN in predicting chaotic time series [180, 149].”
BackgroundMethodToward optical signal processing using Photonic Reservoir Computing
Vandoorne, K., Dierckx, W., Schrauwen, B., Verstraeten, D., Baets, R., Bienstman, P., & Campenhout, J. V. (2008) · Optics Express · doi:10.1364/oe.16.011182 · Cited by 276
Smart homes for the elderly dementia sufferers: identification and prediction of abnormal behaviour
Lotfi, A., Langensiepen, C., Mahmoud, S. M., & Akhlaghinia, M. J. (2012) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-010-0043-x · Cited by 271
Model Predictive Control of Unknown Nonlinear Dynamical Systems Based on Recurrent Neural Networks
Pan, Y., & Wang, J. (2012) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2011.2169636 · Cited by 273
“ESNs have been successfully applied for system identification and control; e.g. [35]–[37].”
MethodCrone, S. F., Hibon, M., & Nikolopoulos, K. (2011) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2011.04.001 · Cited by 271
“…passed since the start of the M3 competition, a decade which has seen the development of extended NN paradigms (e.g., recurrent Echo State NN, see Jaeger & Haas, 2004), theoretical advances in methodologies for specifying NNs (see, e.g., Crone & Kourentzes, 2010; Liao & Fildes, 2005; Qi & Zhang,…”
BackgroundMethodEcho state networks are universal
Grigoryeva, L., & Ortega, J.-P. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.08.025 · Cited by 274
Data-driven science and engineering : machine learning, dynamical systems, and control
Brunton, S. L., & Kutz, J. N. (2019) · doi:10.1017/978-1-009-08951-7 · Cited by 242
Metadata
- Authors
- Jaeger, H., & Haas, H.
- Affiliations
- Constructor University (DE)
- Corresponding author
- Jaeger, Herbert
- Venue
- Science
- Published
- 2004-04-02
- Volume
- 304
- Issue
- 5667
- Pages
- 78-80
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2118706537
- Cited by
- 3,811
- References
- 18
- Citations by year
- 2026: 257 · 2025: 314 · 2024: 315 · 2023: 332 · 2022: 300 · 2021: 355 · 2020: 259 · 2019: 309 · 2018: 237 · 2017: 160 · 2016: 142 · 2015: 128 · 2014: 120 · 2013: 107 · 2012: 100
- FWCI
- 33.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Reservoir Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Chaos control and synchronization
- Keywords
- echo state networks, chaotic time series prediction, recurrent neural networks, channel equalization, wireless communication, energy efficiency, reservoir computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.