The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Design of deep echo state networks
Open accessAbstract
In this paper, we provide a novel approach to the architectural design of deep Recurrent Neural Networks using signal frequency analysis. In particular, focusing on the Reservoir Computing framework and inspired by the principles related to the inherent effect of layering, we address a fundamental open issue in deep learning, namely the question of how to establish the number of layers in recurrent architectures in the form of deep echo state networks (DeepESNs). The proposed method is first analyzed and refined on a controlled scenario and then it is experimentally assessed on challenging real-world tasks. The achieved results also show the ability of properly designed DeepESNs to outperform RC approaches on a speech recognition task, and to compete with the state-of-the-art in time-series prediction on polyphonic music tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
(2010) · Cited by 14,134
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
(2016) · Cited by 11,679
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication
Jaeger, H., & Haas, H. (2004) · Science · doi:10.1126/science.1091277 · Cited by 3,811
(2013) · Cited by 3,740
Reservoir computing approaches to recurrent neural network training
Lukoševičius, M., & Jaeger, H. (2009) · Computer Science Review · doi:10.1016/j.cosrev.2009.03.005 · Cited by 2,956
Neural Networks: Tricks of the Trade: Second Edition
(2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8 · Cited by 2,116
International Conference on Artificial Neural Networks
(1990) · Neural Networks · doi:10.1016/0893-6080(90)90065-s · Cited by 2,088
Acoustic Modeling Using Deep Belief Networks
Mohamed, A.-R., Dahl, G. E., & Hinton, G. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,727
An experimental unification of reservoir computing methods
Verstraeten, D., Schrauwen, B., D’Haene, M., & Stroobandt, D. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.003 · Cited by 1,135
Optimization and applications of echo state networks with leaky- integrator neurons
Jaeger, H., Lukoševičius, M., Popovici, D., & Siewert, U. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.016 · Cited by 948
A Practical Guide to Applying Echo State Networks
Lukoševičius, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_36 · Cited by 867
Real-Time Computation at the Edge of Chaos in Recurrent Neural Networks
Bertschinger, N., & Natschläger, T. (2004) · Neural Computation · doi:10.1162/089976604323057443 · Cited by 833
Minimum Complexity Echo State Network
Rodan, A., & Tino, P. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2089641 · Cited by 710
(2014) · International Conference on Learning Representations · Cited by 634
Training and Analysing Deep Recurrent Neural Networks
Hermans, M., & Schrauwen, B. (2013) · neural information processing systems · Cited by 631
Framewise phoneme classification with bidirectional LSTM networks
Graves, A., & Schmidhuber, J. (2006) · Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. · doi:10.1109/ijcnn.2005.1556215 · Cited by 618
Deep reservoir computing: A critical experimental analysis
Gallicchio, C., Micheli, A., & Pedrelli, L. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.089 · Cited by 519
Self-citationRe-visiting the echo state property
Yildiz, I. B., Jaeger, H., & Kiebel, S. J. (2012) · Neural Networks · doi:10.1016/j.neunet.2012.07.005 · Cited by 486
Advances in optimizing recurrent networks
Bengio, Y., Boulanger-Lewandowski, N., & Pascanu, R. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639349 · Cited by 469
Yamashita, Y., & Tani, J. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000220 · Cited by 468
Boulanger-Lewandowski, N., Bengio, Y., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6392 · Cited by 459
Edge of chaos and prediction of computational performance for neural circuit models
Legenstein, R., & Maass, W. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.017 · Cited by 453
A computational model of filtering, detection, and compression in the cochlea
Lyon, R. (2005) · ICASSP '82. IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1982.1171644 · Cited by 411
Hierarchical Recurrent Neural Networks for Long-Term Dependencies
Hihi, S. E., & Bengio, Y. (1995) · Cited by 325
Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on
Riedhammer, K., Bocklet, T., Ghoshal, A. K., & Povey, D. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · Cited by 265
Isolated word recognition with the Liquid State Machine: a case study
Verstraeten, D., Schrauwen, B., Stroobandt, D., & Van Campenhout, J. (2005) · Information Processing Letters · doi:10.1016/j.ipl.2005.05.019 · Cited by 262
Improving reservoirs using intrinsic plasticity
Schrauwen, B., Wardermann, M., Verstraeten, D., Steil, J. J., & Stroobandt, D. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.12.020 · Cited by 227
Architectural and Markovian factors of echo state networks
Gallicchio, C., & Micheli, A. (2011) · Neural Networks · doi:10.1016/j.neunet.2011.02.002 · Cited by 198
Self-citationGrowing Echo-State Network With Multiple Subreservoirs
Qiao, J., Li, F., Han, H., & Li, W. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2514275 · Cited by 192
Decoupled echo state networks with lateral inhibition
Xue, Y., Yang, L., & Haykin, S. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.014 · Cited by 186
Steil, J. J. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.011 · Cited by 165
Echo State Property of Deep Reservoir Computing Networks
Gallicchio, C., & Micheli, A. (2017) · Cognitive Computation · doi:10.1007/s12559-017-9461-9 · Cited by 164
Self-citationA Gradient Rule for the Plasticity of a Neuron’s Intrinsic Excitability
Triesch, J. (2005) · Lecture notes in computer science · doi:10.1007/11550822_11 · Cited by 154
Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps
Rodan, A., & Peter Tiňo, B. (2012) · Neural Computation · doi:10.1162/neco_a_00297 · Cited by 128
Multilayered Echo State Machine: A Novel Architecture and Algorithm
Malik, Z. K., Hussain, A., & Wu, Q. M. J. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2533545 · Cited by 125
Echo state networks with filter neurons and a delay&sum readout
Holzmann, G., & Häuser, H. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.07.004 · Cited by 98
Perspectives of Neural-Symbolic Integration
Hammer, B., & Hitzler, P. (2007) · Studies in computational intelligence · doi:10.1007/978-3-540-73954-8 · Cited by 95
Acoustic Modeling With Hierarchical Reservoirs
Triefenbach, F., Jalalvand, A., Demuynck, K., & Martens, J. (2013) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2013.2280209 · Cited by 80
Local Lyapunov exponents of deep echo state networks
Gallicchio, C., Micheli, A., & Silvestri, L. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.11.073 · Cited by 69
Self-citationGallicchio, C., & Micheli, A. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489464 · Cited by 69
Self-citationAngelov, P. P., & Sperduti, A. (2016) · Lancaster EPrints (Lancaster University) · Cited by 60
Pre-training of Recurrent Neural Networks via Linear Autoencoders
Pasa, L., & Sperduti, A. (2014) · Padua Research Archive (University of Padova) · Cited by 38
Deep reservoir computing: A critical analysis
Gallicchio, C., & Micheli, A. (2016) · CINECA IRIS Institutial research information system (University of Pisa) · Cited by 38
Self-citationwyffels, F., Schrauwen, B., Verstraeten, D., & Stroobandt, D. (2008) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2008.4634252 · Cited by 37
Randomized Machine Learning Approaches: Recent Developments and Challenges
Gallicchio, C., Martín‐Guerrero, J. D., Micheli, A., & Soria‐Olivas, E. (2017) · CINECA IRIS Institutial research information system (University of Pisa) · Cited by 35
Self-citationMarkovian Bias of Neural-based Architectures With Feedback Connections
Peter Tiňo, B., Hammer, B., & Mikael Bodén, A. (2007) · Studies in computational intelligence · doi:10.1007/978-3-540-73954-8_5 · Cited by 33
Why Layering in Recurrent Neural Networks? A DeepESN Survey
Gallicchio, C., & Micheli, A. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489368 · Cited by 17
Self-citationShort-term Memory of Deep RNNPreprint
Gallicchio, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00748 · Cited by 16
Self-citationNeural Advances in Processing Nonlinear Dynamic Signals
Esposito, A. A., Esposito, A., Esposito, A., Faundez-Zanuy, M., Morabito, F., & Pasero, E. (2018) · Smart innovation, systems and technologies · doi:10.1007/978-3-319-95098-3 · Cited by 12
Randomized Recurrent Neural Networks
Gallicchio, C., Micheli, A., & Peter Tiňo, B. (2018) · CINECA IRIS Institutial research information system (University of Pisa) · Cited by 9
Self-citation
Cited by
Deep learning modelling techniques: Current progress, applications, advantages, and challenges
Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., Mofijur, M., Shawkat Ali, A. B. M., & Gandomi, A. H. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10466-8 · Cited by 1,079
Emerging opportunities and challenges for the future of reservoir computing
Yan, M., Huang, C., Bienstman, P., Tino, P., Lin, W., & Sun, J. (2024) · Nature Communications · doi:10.1038/s41467-024-45187-1 · Cited by 346
Metadata
- Authors
- Gallicchio, C., Micheli, A., & Pedrelli, L.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2018-12
- Volume
- 108
- Pages
- 33-47
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2885420314
- Cited by
- 262
- References
- 58
- FWCI
- 13.86
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.