The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics
Open accessAbstract
We examine the efficiency of Recurrent Neural Networks in forecasting the spatiotemporal dynamics of high dimensional and reduced order complex systems using Reservoir Computing (RC) and Backpropagation through time (BPTT) for gated network architectures. We highlight advantages and limitations of each method and discuss their implementation for parallel computing architectures. We quantify the relative prediction accuracy of these algorithms for the long-term forecasting of chaotic systems using as benchmarks the Lorenz-96 and the Kuramoto-Sivashinsky (KS) equations. We find that, when the full state dynamics are available for training, RC outperforms BPTT approaches in terms of predictive performance and in capturing of the long-term statistics, while at the same time requiring much less training time. However, in the case of reduced order data, large scale RC models can be unstable and more likely than the BPTT algorithms to diverge. In contrast, RNNs trained via BPTT show superior forecasting abilities and capture well the dynamics of reduced order systems. Furthermore, the present study quantifies for the first time the Lyapunov Spectrum of the KS equation with BPTT, achieving similar accuracy as RC. This study establishes that RNNs are a potent computational framework for the learning and forecasting of complex spatiotemporal systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 3×
“…present work some more recent RNN architectures, like Unitary (Arjovsky, Shah, & Bengio, 2016; Jing et al., 2017) and Gated Recurrent Units (GRUs) (Cho et al., 2014; Chung, Gulcehre, Cho, & Bengio, 2014) that have shown superior performance over LSTMs for a wide variety of language, speech signal…”
BackgroundMethodMastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
(2010) · Cited by 14,134
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Detecting strange attractors in turbulence
Takens, F. (1981) · Lecture Notes in Mathematics · doi:10.1007/bfb0091924 · Cited by 10,459 · cited 2×
“Classical approaches to forecast the observable dynamics based on Takens seminal work (Takens, 1981), rely on reconstructing the full dynamics in a high-dimensional phase–space .”
BackgroundDetermining Lyapunov exponents from a time series
Wolf, A., Swift, J. B., Swinney, H. L., & Vastano, J. A. (1985) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(85)90011-9 · Cited by 9,499
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · cited 1×
“RNNs are architectures designed to capture long-term dependencies in sequential data (Bengio, Simard, & Frasconi, 1994; Goodfellow, Bengio, & Courville, 2016; Hochreiter, 1998; Pascanu, Mikolov, & Bengio, 2013).”
Solutions of Ill-Posed Problems.
Bell, J. B., Tikhonov, A. N., & Arsenin, V. Y. (1978) · Mathematics of Computation · doi:10.2307/2006360 · Cited by 8,101
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955 · cited 1×
“This may not be attributed to the expressiveness of Unitary networks, but rather to the difficulty on identifying the right hyperparameters (Greff, Srivastava, Koutík, Steunebrink, & Schmidhuber, 2016).”
BackgroundBackpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879 · cited 1×
“Backpropagation dates back to the works of Dreyfus (1962), Linnainmaa (1976) and Rumelhart, Hinton, and Williams (1986), while its extension to RNNs termed Backpropagation through time (BPTT) was presented in Werbos (1988, 1990).”
MethodOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Maass, W., Natschläger, T., & Markram, H. (2002) · Neural Computation · doi:10.1162/089976602760407955 · Cited by 4,025 · cited 1×
“This technique has been proposed in the context of Echo-State-Networks (ESNs) (Jaeger & Haas, 2004) and Liquid State Machines with spiking neurons (LSM) (Maass, Natschläger, & Markram, 2002).”
MethodHarnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication
Jaeger, H., & Haas, H. (2004) · Science · doi:10.1126/science.1091277 · Cited by 3,811
Ott, E., & Wiesenfeld, K. (1994) · Physics Today · doi:10.1063/1.2808369 · Cited by 3,448
Self-citationMachine Learning for Fluid Mechanics
Brunton, S. L., Noack, B. R., & Koumoutsakos, P. (2020) · Annual Review of Fluid Mechanics · doi:10.1146/annurev-fluid-010719-060214 · Cited by 3,063 · cited 2×
“At the same time it may offer a bridge to the physics community to appreciate and explore the importance of advanced ML algorithms for solving challenging physical problems (Brunton et al., 2020).”
BackgroundSelf-citationReservoir computing approaches to recurrent neural network training
Lukoševičius, M., & Jaeger, H. (2009) · Computer Science Review · doi:10.1016/j.cosrev.2009.03.005 · Cited by 2,956
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
Hochreiter, S. (1998) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488598000094 · Cited by 2,939
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 2,557
Hamiltonian Systems and Transformation in Hilbert Space
Koopman, B. O. (1931) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.17.5.315 · Cited by 2,383
Sauer, T., Yorke, J. A., & Casdagli, M. (1991) · Journal of Statistical Physics · doi:10.1007/bf01053745 · Cited by 2,045
Nonlinear Time Series Analysis
(2012) · SpringerReference · doi:10.1007/springerreference_205493 · Cited by 1,983
Analysis of Observed Chaotic Data
Abarbanel, H. D. I., & Gollub, J. P. (1996) · Physics Today · doi:10.1063/1.881528 · Cited by 1,821
Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks
Sainath, T. N., Vinyals, O., Senior, A., & Sak, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178838 · Cited by 1,678
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557
Nonlinear analysis of hydrodynamic instability in laminar flames—I. Derivation of basic equations
Sivashinsky, G. I. (1977) · Acta Astronautica · doi:10.1016/0094-5765(77)90096-0 · Cited by 1,551
Deep learning for universal linear embeddings of nonlinear dynamics
Lusch, B., Kutz, J. N., & Brunton, S. L. (2018) · Nature Communications · doi:10.1038/s41467-018-07210-0 · Cited by 1,424
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · RePEc: Research Papers in Economics · Cited by 1,310
Pathak, J., Hunt, B., Girvan, M., Lu, Z., & Ott, E. (2018) · Physical Review Letters · doi:10.1103/physrevlett.120.024102 · Cited by 1,303
Self-citation(2011) · Validated Numerics · doi:10.2307/j.ctvcm4g18.8 · Cited by 1,187
A theoretically grounded application of dropout in recurrent neural networks
Gal, Y., & Ghahramani, Z. (2016) · Cambridge University Engineering Department Publications Database · Cited by 1,089
Measurement of the Lyapunov Spectrum from a Chaotic Time Series
Sano, M., & Sawada, Y. (1985) · Physical Review Letters · doi:10.1103/physrevlett.55.1082 · Cited by 982
Fourth-Order Time-Stepping for Stiff PDEs
Kassam, A.-K., & Trefethen, L. N. (2005) · SIAM Journal on Scientific Computing · doi:10.1137/s1064827502410633 · Cited by 950
Generalization of backpropagation with application to a recurrent gas market model
Werbos, P. J. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90007-x · Cited by 944
A Practical Guide to Applying Echo State Networks
Lukoševičius, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_36 · Cited by 867
Larger, L., Soriano, M. C., Brunner, D., Appeltant, L., Gutierrez, J. M., Pesquera, L., Mirasso, C. R., & Fischer, I. (2012) · Optics Express · doi:10.1364/oe.20.003241 · Cited by 864
On the Computational Power of Neural Nets
Siegelmann, H. T., & Sontag, E. D. (1995) · Journal of Computer and System Sciences · doi:10.1006/jcss.1995.1013 · Cited by 692
Weather Forecasting with Ensemble Methods
Gneiting, T., & Raftery, A. E. (2005) · Science · doi:10.1126/science.1115255 · Cited by 636 · cited 1×
“…exploration as it may have far-reaching implications for societal and scientific challenges in fields ranging from weather and climate prediction (Gneiting & Raftery, 2005; Weyn, Durran, & Caruana, 2019), medicine analytics (Esteva et al., 2017; Kurth et al., 2018), to modeling ocean dynamics…”
Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data
Pathak, J., Lu, Z., Hunt, B. R., Girvan, M., & Ott, E. (2017) · Chaos: An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5010300 · Cited by 620
Self-citationDiffusion-Induced Chaos in Reaction Systems
Kuramoto, Y. (1978) · Progress of Theoretical Physics Supplement · doi:10.1143/ptps.64.346 · Cited by 610
Parallel distributed computing using Python
Dalcin, L. D., Paz, R. R., Kler, P. A., & Cosimo, A. (2011) · Advances in Water Resources · doi:10.1016/j.advwatres.2011.04.013 · Cited by 590
Multi-Task Learning for Multiple Language Translation
Dong, D., Wu, H., He, W., Yu, D., & Wang, H. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1166 · Cited by 565
The limits and potentials of deep learning for robotics
Sünderhauf, N., Brock, O., Scheirer, W. J., Hadsell, R. T., Fox, D., Leitner, J., Upcroft, B., Abbeel, P., Burgard, W., Milford, M. J., et al. (2018) · QUT ePrints (Queensland University of Technology) · Cited by 519
Larger, L., Baylón-Fuentes, A., Martinenghi, R., Udaltsov, V. S., Chembo, Y. K., & Jacquot, M. (2017) · Physical Review X · doi:10.1103/physrevx.7.011015 · Cited by 509
Nonlinear signal processing using neural networks: Prediction and system modelling
Lapedes, A. S., & Farber, R. M. (1987) · University of North Texas Digital Library (University of North Texas) · Cited by 437
Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networksPreprint
Vlachas, P. R., Byeon, W., Koumoutsakos, P., Wan, Z. Y., & Sapsis, T. P. (2018) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 424
Self-citationAttractor reconstruction by machine learning
Lu, Z., Hunt, B. R., & Ott, E. (2018) · Chaos: An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5039508 · Cited by 374
Self-citationNonlinear time-series analysis revisited
Bradley, E., & Kantz, H. (2015) · Chaos: An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.4917289 · Cited by 368
Weyn, J. A., Durran, D. R., & Caruana, R. (2019) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2019ms001705 · Cited by 347
MPI for Python: Performance improvements and MPI-2 extensions
Dalcín, L., Paz, R., Storti, M., & D’Elía, J. (2008) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2007.09.005 · Cited by 328
Taylor expansion of the accumulated rounding error
Linnainmaa, S. (1976) · BIT · doi:10.1007/bf01931367 · Cited by 315
Pathak, J., Wikner, A., Fussell, R., Chandra, S., Hunt, B. R., Girvan, M., & Ott, E. (2018) · Chaos: An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5028373 · Cited by 307
Self-citationRecurrent Neural Networks Are Universal Approximators
Schäfer, A. M., & Zimmermann, H. G. (2006) · Lecture Notes in Computer Science · doi:10.1007/11840817_66 · Cited by 297
Axiom: A Householder-Parameterized Pure Unitary RNN for Long-Range Sequence Modeling
Chaudhary, S. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20108465 · Cited by 293
Echo state networks are universal
Grigoryeva, L., & Ortega, J.-P. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.08.025 · Cited by 274
Data-assisted reduced-order modeling of extreme events in complex dynamical systems
Wan, Z. Y., Vlachas, P., Koumoutsakos, P., & Sapsis, T. (2018) · PLOS ONE · doi:10.1371/journal.pone.0197704 · Cited by 225
Self-citationHa, D., & Schmidhuber, J. (2018) · Zenodo · doi:10.5281/zenodo.1207631 · Cited by 187
An overview and comparative analysis of Recurrent Neural Networks for Short Term Load Forecasting
Bianchi, F. M., Maiorino, E., Kampffmeyer, M., Rizzi, A., & Jenssen, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04378 · Cited by 182
The numerical solution of variational problems
Dreyfus, S. (1962) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(62)90004-5 · Cited by 172
Jiang, J., & Lai, Y. (2019) · Physical Review Research · doi:10.1103/physrevresearch.1.033056 · Cited by 168
Predicting chaotic time series with wavelet networks
Cao, L., Hong, Y., Fang, H., et al. (1995) · Physica D Nonlinear Phenomena · doi:10.1016/0167-2789(95)00119-o · Cited by 164
Reservoir computing with a single time-delay autonomous Boolean node
Haynes, N. D., Soriano, M. C., Rosin, D. P., Fischer, I., & Gauthier, D. J. (2015) · Physical Review E · doi:10.1103/physreve.91.020801 · Cited by 147
Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNsPreprint
Jing, L., Shen, Y., Dubček, T., Peurifoy, J., Skirlo, S. A., LeCun, Y., Tegmark, M., & Marin Soljacic, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.05231 · Cited by 142
Prediction of Spatiotemporal Time Series Based on Reconstructed Local States
Parlitz, U., & Merkwirth, C. (2000) · Physical Review Letters · doi:10.1103/physrevlett.84.1890 · Cited by 105
Model identification of a spatiotemporally varying catalytic reaction
Krischer, K., Rico-Martı́nez, R., Kevrekidis, I. G., Rotermund, H. H., Ertl, G., & Hudson, J. L. (1993) · AIChE Journal · doi:10.1002/aic.690390110 · Cited by 93
Antonik, P., Haelterman, M., & Massar, S. (2017) · Physical Review Applied · doi:10.1103/physrevapplied.7.054014 · Cited by 66
Machine learning the kinematics of spherical particles in fluid flows
Wan, Z. Y., & Sapsis, T. P. (2018) · Journal of Fluid Mechanics · doi:10.1017/jfm.2018.797 · Cited by 49
Self-citationThe limits and potentials of deep learning for robotics
Sünderhauf, N., Brock, O., Scheirer, W., Hadsell, R., Fox, D., Leitner, J., Upcroft, B., Abbeel, P., Burgard, W., Milford, M., et al. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364918770733 · Cited by 38 · cited 1×
“, 2018), and the dynamics of ocean dynamics and turbulent flows (Aksamit et al., 2019; Sünderhauf et al., 2018; Brunton et al., 2020).”
BackgroundEvaluating Lyapunov exponent spectra with neural networks
Maus, A., & Sprott, J. C. (2013) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2013.03.001 · Cited by 38
The topological invariance of Lyapunov exponents in embedded dynamics
Dechert, W. D., & Gençay, R. (1996) · Physica D Nonlinear Phenomena · doi:10.1016/0167-2789(95)00225-1 · Cited by 36
Noninvertibility and resonance in discrete-time neural networks for time-series processing
GICQUEL, N., Anderson, J. S., & Kevrekidis, I. G. (1998) · Physics Letters A · doi:10.1016/s0375-9601(97)00753-6 · Cited by 20
Cited by
Physics-informed machine learning
Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021) · Nature Reviews Physics · doi:10.1038/s42254-021-00314-5 · Cited by 8,718
“187 may be appropriate and offer a promising direction to follow.”
BackgroundNext generation reservoir computing
Gauthier, D. J., Bollt, E., Griffith, A., & Barbosa, W. A. S. (2021) · Nature Communications · doi:10.1038/s41467-021-25801-2 · Cited by 582
Physical reservoir computing—an introductory perspective
Nakajima, K. (2020) · Japanese Journal of Applied Physics · doi:10.35848/1347-4065/ab8d4f · Cited by 422
Machine learning and applications in ultrafast photonics
Genty, G., Salmela, L., Dudley, J. M., Brunner, D., Kokhanovskiy, A., Kobtsev, S., & Turitsyn, S. K. (2021) · Nature Photonics · doi:10.1038/s41566-020-00716-4 · Cited by 418
Emerging opportunities and challenges for the future of reservoir computing
Yan, M., Huang, C., Bienstman, P., Tino, P., Lin, W., & Sun, J. (2024) · Nature Communications · doi:10.1038/s41467-024-45187-1 · Cited by 346
Metadata
- Authors
- Vlachas, P. R., Pathak, J., Hunt, B. R., Sapsis, T. P., Girvan, M., Ott, E., & Koumoutsakos, P.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2020-06
- Volume
- 126
- Pages
- 191-217
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W3006036768
- Cited by
- 480
- References
- 80
- FWCI
- 24.81
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.