CoChat
Sign Up
Cited by 41Neural Networks and Reservoir ComputingNeural Networks and ApplicationsNonlinear Dynamics and Pattern Formation

Multiplex visibility graphs to investigate recurrent neural network dynamics

Open access
Bianchi, Filippo Maria, Livi, Lorenzo, Alippi, Cesare, Jenssen, Robert·2017·Scientific Reports
DOI 10.1038/srep44037PMC PMC5345088
Open this paper in CoChat — free accountOpen PDF

Abstract

A recurrent neural network (RNN) is a universal approximator of dynamical systems, whose performance often depends on sensitive hyperparameters. Tuning them properly may be difficult and, typically, based on a trial-and-error approach. In this work, we adopt a graph-based framework to interpret and characterize internal dynamics of a class of RNNs called echo state networks (ESNs). We design principled unsupervised methods to derive hyperparameters configurations yielding maximal ESN performance, expressed in terms of prediction error and memory capacity. In particular, we propose to model time series generated by each neuron activations with a horizontal visibility graph, whose topological properties have been shown to be related to the underlying system dynamics. Successively, horizontal visibility graphs associated with all neurons become layers of a larger structure called a multiplex. We show that topological properties of such a multiplex reflect important features of ESN dynamics that can be used to guide the tuning of its hyperparamers. Results obtained on several benchmarks and a real-world dataset of telephone call data records show the effectiveness of the proposed methods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/srep44037

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most studies using complex networks as preprocessing convert raw time series into graphs, extract topological statistics, and feed them into DNN models.

    “In particular, most of the studies in this area use complex networks to convert raw time series data into graphs and then extract the topological statistics (such as cluster coefficients, path lengths, and node edge distributions) from the graphs and feed them into the DNN models.”

    From TVGeAN: Tensor Visibility Graph-Enhanced Attention Network for Versatile Multivariant Time Series Learning Tasks (2024) · cites this paper as [79]

    The Abstract (page unavailable) describes converting time series of neuron activations into graphs, and the Discussion (page unavailable) analyzes topological properties such as clustering coefficients. The paper does not establish that most studies use this approach or that the extracted statistics are fed into DNN models; its proposed graph-based methods tune ESN hyperparameters. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. Barzel B. & Barabási A.-L. Universality in network dynamics. Nature Physics 9, 673– 681 (2013). PMC3852675 24319492
  2. Hammer B., Micheli A., Sperduti A. & Strickert M. Recursive self-organizing network models. Neural Networks 17, 1061– 1085 (2004). 15555852
  3. Maass W., Joshi P. & Sontag E. D. Computational aspects of feedback in neural circuits. PLoS Computational Biology 3, e165 (2007). PMC1779299 17238280
  4. Jordan M. I. Serial order: A parallel distributed processing approach. In Donahoe J. W. & Dorsel V. P. (eds) Neural-Network Models of Cognition: Biobehavioral Foundations, vol. 121 of Advances in Psychology, 471– 495 (North-Holland, 1997).
  5. Amit D. J., Gutfreund H. & Sompolinsky H. Spin-glass models of neural networks. Physical Review A 32, 1007– 1018 (1985). 9896156
  6. Enel P., Procyk E., Quilodran R. & Dominey P. F. Reservoir computing properties of neural dynamics in prefrontal cortex. PLoS Computational Biology 12, e1004967 (2016). 27286251 PMC4902312
  7. Barak O., Sussillo D., Romo R., Tsodyks M. & Abbott L. F. From fixed points to chaos: three models of delayed discrimination. Progress in Neurobiology 103, 214– 222 (2013). 23438479 PMC3622800
  8. Rajan K., Abbott L. F. & Sompolinsky H. Stimulus-dependent suppression of chaos in recurrent neural networks. Physical Review E 82, 011903 (2010). PMC10683875 20866644
  9. Fusi S., Miller E. K. & Rigotti M. Why neurons mix: high dimensionality for higher cognition. Current Opinion in Neurobiology 37, 66– 74 (2016). 26851755
  10. Schmidhuber J. Deep learning in neural networks: An overview. Neural Networks 61, 85– 117 (2015). 25462637
  11. Barra A., Bernacchia A., Santucci E. & Contucci P. On the equivalence of Hopfield networks and Boltzmann machines. Neural Networks 34, 1– 9 (2012). 22784924
  12. Charles A., Yin D. & Rozell C. Distributed sequence memory of multidimensional inputs in recurrent networks. arXiv preprint arXiv:1605.08346 (2016). · arXiv
  13. Legenstein R. & Maass W. What makes a dynamical system computationally powerful? In Haykin S., Principe J., Sejnowski & McWhirter (eds) New Directions in Statistical Signal Processing: From Systems to Brain, 127– 154 (MIT Press, Cambridge, 2007).
  14. Yildiz I. B., Jaeger H. & Kiebel S. J. Re-visiting the echo state property. Neural Networks 35, 1– 9 (2012). 22885243
  15. Dambre J., Verstraeten D., Schrauwen B. & Massar S. Information processing capacity of dynamical systems. Scientific Reports 2 (2012). PMC3400147 22816038
  16. Lajoie G., Lin K. K., Thivierge J.-P. & Shea-Brown E. Encoding in balanced networks: Revisiting spike patterns and chaos in stimulus-driven systems. PLoS Computational Biology 12, 1– 30 (2016). PMC5156368 27973557
  17. Livi L., Bianchi F. M. & Alippi C. Determination of the edge of criticality in echo state networks through Fisher information maximization. IEEE Transactions on Neural Networks and Learning Systems 1– 12, doi: 10.1109/TNNLS.2016.2644268 (2017). 28092580 · DOI
  18. Mayer N. M. Input-anticipating critical reservoirs show power law forgetting of unexpected input events. Neural Computation 27, 1102– 1119 (2015). 25774542
  19. Moretti P. & Muñoz M. A. Griffiths phases and the stretching of criticality in brain networks. Nature Communications 4 (2013). 24088740
  20. Mora T. & Bialek W. Are biological systems poised at criticality? Journal of Statistical Physics 144, 268– 302 (2011).
  21. Wang X., Lizier J. & Prokopenko M. Fisher information at the edge of chaos in random Boolean networks. Artificial Life 17, 315– 329 (2011). 21762019
  22. Tkačik G.. Thermodynamics and signatures of criticality in a network of neurons. Proceedings of the National Academy of Sciences 112, 11508– 11513 (2015). PMC4577210 26330611
  23. Grigoryeva L., Henriques J., Larger L. & Ortega J.-P. Optimal nonlinear information processing capacity in delay-based reservoir computers. Scientific Reports 5 (2015). PMC4566100 26358528
  24. Bianchi F. M., Scardapane S., Uncini A., Rizzi A. & Sadeghian A. Prediction of telephone calls load using echo state network with exogenous variables. Neural Networks 71, 204– 213 (2015). 26413714
  25. Bianchi F. M., Livi L. & Alippi C. Investigating echo state networks dynamics by means of recurrence analysis. IEEE Transactions on Neural Networks and Learning Systems 1– 13, doi: 10.1109/TNNLS.2016.2630802 (2016). 28114039 · DOI
  26. Boedecker J., Obst O., Lizier J. T., Mayer N. M. & Asada M. Information processing in echo state networks at the edge of chaos. Theory in Biosciences 131, 205– 213 (2012). 22147532
  27. Ozturk M. C., Xu D. & Prncipe J. C. Analysis and design of echo state networks. Neural Computation 19, 111– 138 (2007). 17134319
  28. Verstraeten D. & Schrauwen B. On the quantification of dynamics in reservoir computing. In Artificial Neural Networks-ICANN 2009 985– 994 (Springer: Berlin Heidelberg, 2009).
  29. Leisch F., Trapletti A. & Hornik K. Stationarity and stability of autoregressive neural network processes. In Kearns M. J., Solla S. A. & Cohn D. A. (eds) Advances in Neural Information Processing Systems 11, 267– 273 (MIT Press, 1999).
  30. Lee K.-M., Min B. & Goh K.-I. Towards real-world complexity: an introduction to multiplex networks. The European Physical Journal B 88, 1– 20 (2015).
  31. Boccaletti S.. The structure and dynamics of multilayer networks. Physics Reports 544, 1– 122 (2014). PMC7332224 32834429
  32. Menichetti G., Remondini D., Panzarasa P., Mondragón R. J. & Bianconi G. Weighted multiplex networks. PloS ONE 9, e97857 (2014). 24906003 PMC4048161
  33. Kivelä M.. Multilayer networks. Journal of Complex Networks 2, 203– 271 (2014).
  34. De Domenico M.. Mathematical formulation of multilayer networks. Physical Review X 3, 041022 (2013).
  35. Lacasa L., Nicosia V. & Latora V. Network structure of multivariate time series. Scientific Reports 5 (2015). PMC4614448 26487040
  36. Luque B., Lacasa L., Ballesteros F. & Luque J. Horizontal visibility graphs: Exact results for random time series. Physical Review E 80, 046103 (2009). 19905386
  37. Luque B., Lacasa L., Ballesteros F. J. & Robledo A. Feigenbaum graphs: A complex network perspective of chaos. PLoS One 6, e22411 (2011). 21915254 PMC3168432
  38. Luque B., Cordero-Gracia M., Gómez M. & Robledo A. Quasiperiodic graphs at the onset of chaos. Physical Review E 88, 062918 (2013). 24483542
  39. Lacasa L. & Toral R. Description of stochastic and chaotic series using visibility graphs. Physical Review E 82, 036120 (2010). 21230152
  40. Ravetti M. G., Carpi L. C., Gonçalves B. A., Frery A. C. & Rosso O. A. Distinguishing noise from chaos: Objective versus subjective criteria using horizontal visibility graph. PloS ONE 9, e108004 (2014). 25247303 PMC4172653
  41. Luque B., Lacasa L., Ballesteros F. J. & Robledo A. Analytical properties of horizontal visibility graphs in the Feigenbaum scenario. Chaos: An Interdisciplinary Journal of Nonlinear Science 22, 013109 (2012). 22462985
  42. Zhang S.. Architectural complexity measures of recurrent neural networks. arXiv preprint arXiv:1602.08210 (2016). · arXiv
  43. Hermans M. & Schrauwen B. Recurrent kernel machines: Computing with infinite echo state networks. Neural Computation 24, 104– 133 (2012). 21851278
  44. Lukoševičius M. A Practical Guide to Applying Echo State Networks, 659– 686 (Springer Berlin Heidelberg, Berlin, Heidelberg, 2012).
  45. Saramäki J., Kivelä M., Onnela J.-P., Kaski K. & Kertesz J. Generalizations of the clustering coefficient to weighted complex networks. Physical Review E 75, 027105 (2007). 17358454
  46. Pascanu R. & Jaeger H. A neurodynamical model for working memory. Neural Networks 24, 199– 207 (2011). 21036537
  47. Jaeger H. Adaptive nonlinear system identification with echo state networks. In Advances in Neural Information Processing Systems, 593– 600 (MIT Press, 2002).
  48. Butcher J., Verstraeten D., Schrauwen B., Day C. & Haycock P. Reservoir computing and extreme learning machines for non-linear time-series data analysis. Neural Networks 38, 76– 89 (2013). 23275138
  49. Blondel V. D.. Data for Development: the D4D Challenge on Mobile Phone Data. ArXiv preprint arXiv:1210.0137 (2012). · arXiv
  50. Restrepo J. G., Ott E. & Hunt B. R. Approximating the largest eigenvalue of network adjacency matrices. Physical Review E 76, 056119 (2007). 18233730
  51. Newman M. E. J. Assortative mixing in networks. Physical Review Letters 89, 208701 (2002). 12443515
  52. Marblestone A. H., Wayne G. & Kording K. P. Toward an integration of deep learning and neuroscience. Frontiers in Computational Neuroscience 10 (2016). PMC5021692 27683554
  53. Sussillo D., Churchland M. M., Kaufman M. T. & Shenoy K. V. A neural network that finds a naturalistic solution for the production of muscle activity. Nature Neuroscience 18, 1025– 1033 (2015). 26075643 PMC5113297
  54. Sporns O. Networks of the Brain (MIT press, Cambridge, MA, USA, 2011).
  55. Lopez-Fernandez L., Robles G. & Gonzalez-Barahona J. M. Applying social network analysis to the information in CVS repositories. In Proceedings of the International Workshop on Mining Software Repositories 101–105 (Edinburgh, UK, 2004).

Cited by

Sorted by citations

Authors
Bianchi, F. M., Livi, L., Alippi, C., & Jenssen, R.
Affiliations
  • UiT The Arctic University of Norway (NO)
  • University of Exeter (GB)
  • Università della Svizzera italiana (CH)
Corresponding author
Bianchi, Filippo Maria
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2017-03-10
Volume
7
Issue
1
Pages
44037-44037
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/srep44037
PMC
PMC5345088
OpenAlex
W2519273635
Cited by
41
References
5
Citations by year
2025: 2 · 2024: 4 · 2023: 3 · 2022: 5 · 2021: 4 · 2020: 6 · 2019: 7 · 2018: 6 · 2017: 3 · 2015: 1
FWCI
3.59
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Virtual Community of Pathological Anatomy (University of Castilla La Mancha) (submittedVersion, other-oa)
  • NORA - Norwegian Open Research Archives (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Reservoir Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Computer Networks and Communications › Nonlinear Dynamics and Pattern Formation
Keywords
recurrent neural networks, echo state networks, horizontal visibility graph, multiplex networks, hyperparameter optimization, memory capacity, unsupervised learning, graph topology, time series
MeSH terms
Unsupervised Machine Learning, Systems Analysis, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review