The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Analysis and Design of Echo State Networks
Abstract
The design of echo state network (ESN) parameters relies on the selection of the maximum eigenvalue of the linearized system around zero (spectral radius). However, this procedure does not quantify in a systematic manner the performance of the ESN in terms of approximation error. This article presents a functional space approximation framework to better understand the operation of ESNs and proposes an information-theoretic metric, the average entropy of echo states, to assess the richness of the ESN dynamics. Furthermore, it provides an interpretation of the ESN dynamics rooted in system theory as families of coupled linearized systems whose poles move according to the input signal dynamics. With this interpretation, a design methodology for functional approximation is put forward where ESNs are designed with uniform pole distributions covering the frequency spectrum to abide by the richness metric, irrespective of the spectral radius. A single bias parameter at the ESN input, adapted with the modeling error, configures the ESN spectral radius to the input-output joint space. Function approximation examples compare the proposed design methodology versus the conventional design.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Haykin, S. (1986) · Cited by 12,603
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Detecting strange attractors in turbulence
Takens, F. (1981) · Lecture Notes in Mathematics · doi:10.1007/bfb0091924 · Cited by 10,459
An Information-Maximization Approach to Blind Separation and Blind Deconvolution
Bell, A. J., & Sejnowski, T. J. (1995) · Neural Computation · doi:10.1162/neco.1995.7.6.1129 · Cited by 8,986
Hopfield, J. J. (1984) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.81.10.3088 · Cited by 6,918
Kailath, T. (1980) · Cited by 6,215
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879 · cited 2×
“Various algorithms, such as backpropagation through time (Werbos, 1990) and real-time recurrent learning (Williams & Zipser, 1989), have been proposed to train RNNs; however, these algorithms suffer from computational complexity, resulting in slow training, complex performance surfaces, the…”
BackgroundA Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369 · cited 2×
“Various algorithms, such as backpropagation through time (Werbos, 1990) and real-time recurrent learning ( Williams & Zipser, 1989 ), have been proposed to train RNNs; however, these algorithms suffer from computational complexity, resulting in slow training, complex performance surfaces, the possibility of instabilit…”
BackgroundMaass, W., Natschläger, T., & Markram, H. (2002) · Neural Computation · doi:10.1162/089976602760407955 · Cited by 4,025 · cited 3×
“A similar problem exists in the design of the LSM. LSMs have been shown to possess universal approximation given the separation property (SP) for the liquid (reservoir in ESNs) and the approximation property (AP) for the readout (Maass et al., 2002).”
BackgroundHarnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication
Jaeger, H., & Haas, H. (2004) · Science · doi:10.1126/science.1091277 · Cited by 3,811
An invariant form for the prior probability in estimation problems
Jeffreys, H. (1946) · Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences · doi:10.1098/rspa.1946.0056 · Cited by 2,323
Differential-Geometrical Methods in Statistics
Amari, S.-I. (1985) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-5056-2 · Cited by 1,673 · cited 1×
“The probability density function (pdf) in a differential geometric framework should be thought of as a volume form; that is, in our case, the pdf of the state vector describes the metric of the state space manifold ( Amari, 1990 ).,Moreover, entropy has been proposed as an appropriate measure of the volume of the sign…”
BackgroundComputation at the edge of chaos: Phase transitions and emergent computation
Langton, C. G. (1990) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(90)90064-v · Cited by 1,527 · cited 1×
“It is interesting to relate this observation with the computational properties found in dynamical systems “at the edge of chaos” (Packard, 1988; Langton, 1990; Mitchell, Hraber, & Crutchfield, 1993; Bertschinger & Natschläger, 2004).”
BackgroundProbability, Frequency and Reasonable Expectation
Cox, R. T. (1946) · American Journal of Physics · doi:10.1119/1.1990764 · Cited by 1,308
Optimization and applications of echo state networks with leaky- integrator neurons
Jaeger, H., Lukoševičius, M., Popovici, D., & Siewert, U. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.016 · Cited by 948
Anderson, J. A., Silverstein, J. W., Ritz, S. A., & Jones, R. S. (1977) · Psychological Review · doi:10.1037//0033-295x.84.5.413 · Cited by 899
Real-Time Computation at the Edge of Chaos in Recurrent Neural Networks
Bertschinger, N., & Natschläger, T. (2004) · Neural Computation · doi:10.1162/089976604323057443 · Cited by 833
Rényi, A. (2007) · Cited by 727
Buehler, R. J., & Wilde, D. J. (1965) · Econometrica · doi:10.2307/1909808 · Cited by 695
Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks
Puskorius, G. V., & Feldkamp, L. A. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279191 · Cited by 528
Information Theoretic Learning
Erdogmus, D., & Principe, J. C. (2009) · Encyclopedia of Artificial Intelligence · doi:10.4018/978-1-59904-849-9.ch133 · Cited by 506
Spatial Transformations in the Parietal Cortex Using Basis Functions
Pouget, A., & Sejnowski, T. J. (1997) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1997.9.2.222 · Cited by 425
Revisiting the Edge of Chaos: Evolving Cellular Automata to Perform Computations
Mitchell, M., Hraber, P., & Crutchfield, J. P. (1993) · arXiv (Cornell University) · doi:10.48550/arxiv.adap-org/9303003 · Cited by 388
Turing computability with neural nets
Siegelmann, H. T., & Sontag, E. D. (1991) · Applied Mathematics Letters · doi:10.1016/0893-9659(91)90080-f · Cited by 377
Training Multilayer Perceptrons with the Extended Kalman Algorithm
Singhal, S., & Wu, L. (1988) · neural information processing systems · Cited by 320
Transient synthesis in the time domain
Kautz, W. (1954) · Transactions of the IRE Professional Group on Circuit Theory · doi:10.1109/tct.1954.1083588 · Cited by 269
Generalized information potential criterion for adaptive system training
Erdogmus, D., & Principe, J. C. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1031936 · Cited by 237
The gamma-filter-a new class of adaptive IIR filters with restricted feedback
Príncipe, J. C., de Vries, B., & Oliveira, P. (1993) · IEEE Transactions on Signal Processing · doi:10.1109/78.193206 · Cited by 158
Self-citationDynamic recurrent neural network for system identification and control
Delgado, A., Kambhampati, C., & Warwick, K. (1995) · IEE Proceedings - Control Theory and Applications · doi:10.1049/ip-cta:19951873 · Cited by 136
On the computational power of Elman-style recurrent networks
Kremer, S. C. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.392262 · Cited by 98
Online entropy manipulation: stochastic information gradient
Deniz Erdogmus, B., Hild, K. E., & Príncipe, J. C. (2003) · IEEE Signal Processing Letters · doi:10.1109/lsp.2003.814400 · Cited by 96
Self-citationEcho state networks: appeal and challenges
Prokhorov, D. V. (2006) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2005.1556091 · Cited by 79
Adaptive behavior with fixed weights in RNN: an overview
Prokhorov, D. V., Feldkarnp, L. A., & Tyukin, I. (2003) · doi:10.1109/ijcnn.2002.1007449 · Cited by 48
Brain-Machine Interface Engineering
Sanchez, J. C., & Príncipe, J. C. (2007) · Synthesis lectures on biomedical engineering · doi:10.1007/978-3-031-01621-9 · Cited by 44
Self-citationAn associative memory readout for ESNs with applications to dynamical pattern recognition
Ozturk, M. C., & Príncipe, J. C. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.012 · Cited by 39
Self-citationMethods for Estimating the Computational Power and Generalization Capability of Neural Microcircuits
Maass, W., Legenstein, R. A., & Bertschinger, N. (2004) · neural information processing systems · Cited by 37
Learning Mappings in Brain Machine Interfaces with Echo State Networks
Rao, Y. N., Kim, S.-P., Sanchez, J. C., Deniz Erdogmus, B., Príncipe, J. C., Carmena, J. M., Lebedev, M., & Nicolelis, M. A. L. (2005) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1416283 · Cited by 32
Self-citationNonlinearity creates linear independence
Ito, Y. (1996) · Advances in Computational Mathematics · doi:10.1007/bf02124743 · Cited by 31
Reinhart, F., & Steil, J. J. (2008) · ECSIS Symposium on Learning and Adaptive Behaviors for Robotic Systems (LAB-RS) · doi:10.1109/lab-rs.2008.17 · Cited by 26
Context discerning multifunction networks: reformulating fixed weight neural networks
Santiago, R. A. (2005) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2004.1379896 · Cited by 20
INFORMATION THEORETIC SELF-ORGANIZATION OF MULTIPLE AGENTS
Chandrasekar, R. (2003) · Cited by 1
Cited by
Reservoir computing approaches to recurrent neural network training
Lukoševičius, M., & Jaeger, H. (2009) · Computer Science Review · doi:10.1016/j.cosrev.2009.03.005 · Cited by 2,956
Recent advances in physical reservoir computing: A review
Tanaka, G., Yamane, T., Héroux, J. B., Nakane, R., Kanazawa, N., Takeda, S., Numata, H., Nakano, D., & Hirose, A. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.03.005 · Cited by 2,146
Optimization and applications of echo state networks with leaky- integrator neurons
Jaeger, H., Lukoševičius, M., Popovici, D., & Siewert, U. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.016 · Cited by 948
A Practical Guide to Applying Echo State Networks
Lukoševičius, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_36 · Cited by 867
Minimum Complexity Echo State Network
Rodan, A., & Tino, P. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2089641 · Cited by 710
Metadata
- Authors
- Ozturk, M. C., Xu, D., & Príncipe, J. C.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2007-01
- Volume
- 19
- Issue
- 1
- Pages
- 111-138
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2029967456
- Cited by
- 308
- References
- 44
- FWCI
- 11.86
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.