The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks
Open accessAbstract
Brains process information in spiking neural networks. Their intricate connections shape the diverse functions these networks perform. Yet how network connectivity relates to function is poorly understood, and the functional capabilities of models of spiking networks are still rudimentary. The lack of both theoretical insight and practical algorithms to find the necessary connectivity poses a major impediment to both studying information processing in the brain and building efficient neuromorphic hardware systems. The training algorithms that solve this problem for artificial neural networks typically rely on gradient descent. But doing so in spiking networks has remained challenging due to the nondifferentiable nonlinearity of spikes. To avoid this issue, one can employ surrogate gradients to discover the required connectivity. However, the choice of a surrogate is not unique, raising the question of how its implementation influences the effectiveness of the method. Here, we use numerical simulations to systematically study how essential design parameters of surrogate gradients affect learning performance on a range of classification problems. We show that surrogate gradient learning is robust to different shapes of underlying surrogate derivatives, but the choice of the derivative's scale can substantially affect learning performance. When we combine surrogate gradients with suitable activity regularization techniques, spiking networks perform robust information processing at the sparse activity limit. Our study provides a systematic account of the remarkable robustness of surrogate gradient learning and serves as a practical guide to model functional spiking neural networks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
Hochreiter, S. (1998) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488598000094 · Cited by 2,939
Towards spike-based machine intelligence with neuromorphic computing
Roy, K., Jaiswal, A., & Panda, P. (2019) · Nature · doi:10.1038/s41586-019-1677-2 · Cited by 2,190
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Context-dependent computation by recurrent dynamics in prefrontal cortex
Mante, V., Sussillo, D., Shenoy, K. V., & Newsome, W. T. (2013) · Nature · doi:10.1038/nature12742 · Cited by 1,847
Using goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
Spoken Language Processing: A Guide to Theory, Algorithm, and System Development
Huang, X., Acero, A., Hon, H.-W., & Reddy, R. (2001) · Cited by 1,593
Neftci, E. O., Mostafa, H., & Zenke, F. (2019) · IEEE Signal Processing Magazine · doi:10.1109/msp.2019.2931595 · Cited by 1,538
Self-citationNeuronal Dynamics: From Single Neurons to Networks and Models of Cognition
Gerstner, W., Kistler, W. M., Naud, R., & Paninski, L. (2014) · doi:10.1017/cbo9781107447615 · Cited by 1,515
Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
Warden, P. (2018) · arXiv · doi:10.48550/arxiv.1804.03209 · Cited by 1,033
A deep learning framework for neuroscience
Richards, B. A., Lillicrap, T. P., Beaudoin, P., Bengio, Y., Bogacz, R., Christensen, A., Clopath, C., Costa, R. P., de Berker, A., Ganguli, S., et al. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0520-2 · Cited by 1,027
Self-citationA Low Power, Fully Event-Based Gesture Recognition System
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.781 · Cited by 1,017
Training Deep Spiking Neural Networks Using Backpropagation
Lee, J. H., Delbruck, T., & Pfeiffer, M. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00508 · Cited by 976
Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades
Orchard, G., Jayawant, A., Cohen, G. K., & Thakor, N. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00437 · Cited by 882
The tempotron: a neuron that learns spike timing–based decisions
Gütig, R., & Sompolinsky, H. (2006) · Nature Neuroscience · doi:10.1038/nn1643 · Cited by 815
Convolutional networks for fast, energy-efficient neuromorphic computing
Esser, S. K., Merolla, P. A., Arthur, J. V., Cassidy, A. S., Appuswamy, R., Andreopoulos, A., Berg, D. J., McKinstry, J. L., Melano, T., Barch, D. R., et al. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1604850113 · Cited by 781
Deep Learning With Spiking Neurons: Opportunities and Challenges
Pfeiffer, M., & Pfeil, T. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00774 · Cited by 766
The recent excitement about neural networks
Crick, F. (1989) · Nature · doi:10.1038/337129a0 · Cited by 714
A wafer-scale neuromorphic hardware system for large-scale neural modeling
Schemmel, J., Briiderle, D., Griibl, A., Hock, M., Meier, K., & Millner, S. (2010) · Proceedings of 2010 IEEE International Symposium on Circuits and Systems · doi:10.1109/iscas.2010.5536970 · Cited by 682
Long short-term memory and learning-to-learn in networks of spiking neuronsPreprint
Bellec, G., Salaj, D., Subramoney, A., Legenstein, R. A., & Maass, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09574 · Cited by 668
Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks
Sussillo, D., & Barak, O. (2013) · Neural Computation · doi:10.1162/neco_a_00409 · Cited by 575
SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks
Zenke, F., & Ganguli, S. (2018) · Neural Computation · doi:10.1162/neco_a_01086 · Cited by 515
Self-citationSterling, P., & Laughlin, S. (2015) · doi:10.7551/mitpress/9780262028707.001.0001 · Cited by 482
Signal Propagation and Logic Gating in Networks of Integrate-and-Fire Neurons
Vogels, T. P., & Abbott, L. F. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.3508-05.2005 · Cited by 465
Self-citationSLAYER: Spike Layer Error Reassignment in Time
Shrestha, S. B., & Orchard, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08646 · Cited by 444
Flexible timing by temporal scaling of cortical responses
Wang, J., Narain, D., Hosseini, E. A., & Jazayeri, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-017-0028-6 · Cited by 443
Supervised Learning Based on Temporal Coding in Spiking Neural Networks
Mostafa, H. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2726060 · Cited by 331
Deep Learning Models of the Retinal Response to Natural Scenes
McIntosh, L., Maheswaranathan, N., Nayebi, A., Ganguli, S., & Baccus, S. A. (2017) · PubMed · doi:10.48550/arxiv.1702.01825 · Cited by 289
The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks
Cramer, B., Stradmann, Y., Schemmel, J., & Zenke, F. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3044364 · Cited by 288
Self-citationGradient Descent for Spiking Neural NetworksPreprint
Huh, D., & B, T. J. S. A. (2017) · arXiv (Cornell University) · doi:10.12751/nncn.bc2017.0088 · Cited by 212
Spiking Deep Networks with LIF Neurons
Hunsberger, E., & Eliasmith, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.08829 · Cited by 203
(2016) · Choice Reviews Online · doi:10.5860/choice.194507 · Cited by 202
Low-dimensional dynamics for working memory and time encoding
Cueva, C. J., Saez, A., Marcos, E., Genovesio, A., Jazayeri, M., Romo, R., Salzman, C. D., Shadlen, M. N., & Fusi, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915984117 · Cited by 161
Deep learning incorporating biologically inspired neural dynamics and in-memory computing
Woźniak, S., Pantazi, A., Bohnstingl, T., & Eleftheriou, E. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0187-0 · Cited by 160
Spiking neurons can discover predictive features by aggregate-label learning
Gütig, R. (2016) · Science · doi:10.1126/science.aab4113 · Cited by 127
Analyzing biological and artificial neural networks: challenges with opportunities for synergy?
Barrett, D. G. T., Morcos, A. S., & Macke, J. H. (2019) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2019.01.007 · Cited by 127
Local online learning in recurrent networks with random feedback
Murray, J. M. (2019) · eLife · doi:10.7554/elife.43299 · Cited by 126
Motor primitives in space and time via targeted gain modulation in cortical networks
Stroud, J. P., Porter, M. A., Hennequin, G., & Vogels, T. P. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0276-0 · Cited by 119
Self-citationBoahen, K. A. (2017) · Computing in Science & Engineering · doi:10.1109/mcse.2017.33 · Cited by 104
'Artiphysiology' reveals V4-like shape tuning in a deep network trained for image classification
Pospisil, D. A., Pasupathy, A., & Bair, W. (2018) · eLife · doi:10.7554/elife.38242 · Cited by 91
From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction
Tanaka, H., Nayebi, A., Maheswaranathan, N., McIntosh, L., Baccus, S. A., & Ganguli, S. (2019) · PubMed · doi:10.48550/arxiv.1912.06207 · Cited by 76
Williamson, R. C., Doiron, B., Smith, M. A., & Yu, B. M. (2019) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2018.12.009 · Cited by 75
(2017) · doi:10.3389/fncom.2016.00131 · Cited by 59
Data and Power Efficient Intelligence with Neuromorphic Learning Machines
Neftci, E. (2018) · iScience · doi:10.1016/j.isci.2018.06.010 · Cited by 52
Error-Backpropagation in Networks of Fractionally Predictive Spiking Neurons
Bohté, S. M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21735-7_8 · Cited by 50
Technical report: supervised training of convolutional spiking neural networks with PyTorchPreprint
Zimmer, R., Pellegrini, T., Singh, S. F., & Masquelier, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.10124 · Cited by 34
The dynamic neural code of the retina for natural scenesPreprint
Maheswaranathan, N., McIntosh, L., Tanaka, H., Grant, S., Kastner, D. B., Melander, J., Nayebi, A., Brezovec, L. E., Wang, J. H., Ganguli, S., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/340943 · Cited by 24
A modular neural network model of grasp movement generationPreprint
Michaels, J. A., Schaffelhofer, S., Agudelo-Toro, A., & Scherberger, H. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/742189 · Cited by 21
Training spiking multi-layer networks with surrogate gradients on an analog neuromorphic substratePreprint
Cramer, B., Billaudelle, S., Kanya, S., Leibfried, A., Grübl, A., Karasenko, V., Pehle, C., Schreiber, K., Stradmann, Y., Weis, J., et al. (2020) · arXiv (Cornell University) · Cited by 18
Self-citation
Cited by
Training Spiking Neural Networks Using Lessons From Deep Learning
Eshraghian, J. K., Ward, M., Neftci, E. O., Wang, X., Lenz, G., Dwivedi, G., Bennamoun, M., Jeong, D. S., & Lu, W. D. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3308088 · Cited by 782
Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook
Davies, M., Wild, A., Orchard, G., Sandamirskaya, Y., Guerra, G. A. F., Joshi, P., Plank, P., & Risbud, S. R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3067593 · Cited by 688
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks
Fang, W., Yu, Z., Chen, Y., Masquelier, T., Huang, T., & Tian, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00266 · Cited by 605
SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence
Fang, W., Chen, Y., Ding, J., Yu, Z., Masquelier, T., Chen, D., Huang, L., Zhou, H., Li, G., & Tian, Y. (2023) · Science Advances · doi:10.1126/sciadv.adi1480 · Cited by 444
The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks
Cramer, B., Stradmann, Y., Schemmel, J., & Zenke, F. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3044364 · Cited by 288
Self-citation
Metadata
- Authors
- Zenke, F., & Vogels, T. P.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 33
- Issue
- 4
- Pages
- 899-925
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W3124478039
- Cited by
- 261
- References
- 55
- FWCI
- 11.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.