On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Machine learning has become a widely popular and successful paradigm, especially in data-driven science and engineering. A major application problem is data-driven forecasting of future states from a complex dynamical system. Artificial neural networks have evolved as a clear leader among many machine learning approaches, and recurrent neural networks are considered to be particularly well suited for forecasting dynamical systems. In this setting, the echo-state networks or reservoir computers (RCs) have emerged for their simplicity and computational complexity advantages. Instead of a fully trained network, an RC trains only readout weights by a simple, efficient least squares method. What is perhaps quite surprising is that nonetheless, an RC succeeds in making high quality forecasts, competitively with more intensively trained methods, even if not the leader. There remains an unanswered question as to why and how an RC works at all despite randomly selected weights. To this end, this work analyzes a further simplified RC, where the internal activation function is an identity function. Our simplification is not presented for the sake of tuning or improving an RC, but rather for the sake of analysis of what we take to be the surprise being not that it does not work better, but that such random methods work at all. We explicitly connect the RC with linear activation and linear readout to well developed time-series literature on vector autoregressive (VAR) averages that includes theorems on representability through the Wold theorem, which already performs reasonably for short-term forecasts. In the case of a linear activation and now popular quadratic readout RC, we explicitly connect to a nonlinear VAR, which performs quite well. Furthermore, we associate this paradigm to the now widely popular dynamic mode decomposition; thus, these three are in a sense different faces of the same thing. We illustrate our observations in terms of popular benchmark examples including Mackey–Glass differential delay equations and the Lorenz63 system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Detecting strange attractors in turbulence
Takens, F. (1981) · Lecture Notes in Mathematics · doi:10.1007/bfb0091924 · Cited by 10,459 · cited 1×
“f \stacked" (delayed) variables, called the companion system. This idea is familiar in dynamical systems as we see it is related to stating time-delay variables and the Taken’s embedding theorem [77,61,71,58,9,87]. Dene, X k+1 = AX k+ C+ e k; where, X k= 2 6 6 6 6 6 6 6 6 6 6 6 6 6 4 j x k j x k 1 j .. . j x 1 j…”
Maass, W., Natschläger, T., & Markram, H. (2002) · Neural Computation · doi:10.1162/089976602760407955 · Cited by 4,025 · cited 1×
“The reservoir computing (RC) [44, 58, 87] and the closely related echo state network (ESN) [43, 57] and liquid state machine (LSM) [60, 35] have emerged as a special variant of an RNN, where only the output layer is trained rather than the entire network of weights.”
BackgroundHarnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication
Jaeger, H., & Haas, H. (2004) · Science · doi:10.1126/science.1091277 · Cited by 3,811 · cited 1×
“ion, and especially the long short term memory (LSTM) approach demonstrate excellent delity, [82,35,81,20,18,86], but these are especially expensive to fully train, [63]. The reservoir computing (RC) [38,52,80] and the closely related echo state network (ESN) [37,51] and liquid state machine (LSM) [54,31] have emerged…”
BackgroundReservoir computing approaches to recurrent neural network training
Lukoševičius, M., & Jaeger, H. (2009) · Computer Science Review · doi:10.1016/j.cosrev.2009.03.005 · Cited by 2,956
Sauer, T., Yorke, J. A., & Casdagli, M. (1991) · Journal of Statistical Physics · doi:10.1007/bf01053745 · Cited by 2,045
Lasota, A., & Mackey, M. C. (1994) · Applied Mathematical Sciences · doi:10.1007/978-1-4612-4286-4 · Cited by 1,353
Using recurrent neural network models for early detection of heart failure onset
Choi, E., Schuetz, A., Stewart, W. F., & Sun, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw112 · Cited by 928
Cited by
Next generation reservoir computing
Gauthier, D. J., Bollt, E., Griffith, A., & Barbosa, W. A. S. (2021) · Nature Communications · doi:10.1038/s41467-021-25801-2 · Cited by 582
Self-citationEmerging opportunities and challenges for the future of reservoir computing
Yan, M., Huang, C., Bienstman, P., Tino, P., Lin, W., & Sun, J. (2024) · Nature Communications · doi:10.1038/s41467-024-45187-1 · Cited by 346
Metadata
- Authors
- Bollt, E.
- Venue
- Chaos: An Interdisciplinary Journal of Nonlinear Science
- Publisher
- AIP Publishing
- Published
- 2021-01-01
- Volume
- 31
- Issue
- 1
- Pages
- 013108-013108
- ISSN
- 1054-1500 (ISSN-L)
- 1089-7682
- OpenAlex
- W3049633943
- Cited by
- 179
- References
- 8
- FWCI
- 9.88
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.