The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Attention Spiking Neural Networks
Open accessAbstract
Brain-inspired spiking neural networks (SNNs) are becoming a promising energy-efficient alternative to traditional artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a significant hindrance to deploying SNNs ubiquitously. To leverage the full potential of SNNs, in this paper we study the attention mechanisms, which can help human focus on important information. We present our idea of attention in SNNs with a multi-dimensional attention module, which infers attention weights along the temporal, channel, as well as spatial dimension separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response. Extensive experimental results on event-based action recognition and image classification datasets demonstrate that attention facilitates vanilla SNNs to achieve sparser spiking firing, better performance, and energy efficiency concurrently. In particular, we achieve top-1 accuracy of 75.92% and 77.08% on ImageNet-1 K with single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. Compared with counterpart Res-ANN-104, the performance gap becomes -0.95/+0.21 percent and the energy efficiency is 31.8×/7.4×. To analyze the effectiveness of attention SNNs, we theoretically prove that the spiking degradation or the gradient vanishing, which usually holds in general SNNs, can be resolved by introducing the block dynamical isometry theory. We also analyze the efficiency of attention SNNs based on our proposed spiking response visualization method. Our work lights up SNN's potential as a general backbone to support various applications in the field of SNN research, with a great balance between effectiveness and energy efficiency.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
CBAM: Convolutional Block Attention Module
Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_1 · Cited by 25,853
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks
Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., & Hu, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01155 · Cited by 9,641 · cited 6×
“ECA has the smallest ΔMAC .”
BackgroundMethodInfluentialA million spiking-neuron integrated circuit with a scalable communication network and interface
Merolla, P. A., Arthur, J. V., Alvarez-Icaza, R., Cassidy, A. S., Sawada, J., Akopyan, F., Jackson, B. L., Imam, N., Guo, C., Nakamura, Y., et al. (2014) · Science · doi:10.1126/science.1254642 · Cited by 4,147
Loihi: A Neuromorphic Manycore Processor with On-Chip Learning
Davies, M., Srinivasa, N., Lin, T.-H., Chinya, G., Cao, Y., Choday, S. H., Dimou, G., Joshi, P., Imam, N., Jain, S., et al. (2018) · IEEE Micro · doi:10.1109/mm.2018.112130359 · Cited by 3,897
Networks of spiking neurons: The third generation of neural network models
Maass, W. (1997) · Neural Networks · doi:10.1016/s0893-6080(97)00011-7 · Cited by 3,481
ViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767
SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks
Yang, L., Zhang, R., Li, L., & Xie, X. (2021) · International Conference on Machine Learning · Cited by 2,735
Attention mechanisms in computer vision: A survey
Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng, M.-M., & Hu, S.-M. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0271-y · Cited by 2,566 · cited 1×
“Please refer to [44] for a comprehensive review of attention in CV.”
BackgroundGallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3008413 · Cited by 2,566
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E., Feng, J., & Yan, S. (2021) · arXiv · doi:10.48550/arxiv.2101.11986 · Cited by 2,415
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415
Towards spike-based machine intelligence with neuromorphic computing
Roy, K., Jaiswal, A., & Panda, P. (2019) · Nature · doi:10.1038/s41586-019-1677-2 · Cited by 2,190
1.1 Computing's energy problem (and what we can do about it)
Horowitz, M. (2014) · 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) · doi:10.1109/isscc.2014.6757323 · Cited by 2,069
Is Space-Time Attention All You Need for Video Understanding?
Bertasius, G., Wang, H., & Torresani, L. (2021) · arXiv · doi:10.48550/arxiv.2102.05095 · Cited by 1,663
Neftci, E. O., Mostafa, H., & Zenke, F. (2019) · IEEE Signal Processing Magazine · doi:10.1109/msp.2019.2931595 · Cited by 1,538
Going Deeper in Spiking Neural Networks: VGG and Residual Architectures
Sengupta, A., Ye, Y., Wang, R., Liu, C., & Roy, K. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00095 · Cited by 1,211
Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks
Wu, Y., Deng, L., Li, G., Zhu, J., & Shi, L. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00331 · Cited by 1,196
Self-citationTowards artificial general intelligence with hybrid Tianjic chip architecture
Pei, J., Deng, L., Song, S., Zhao, M., Zhang, Y., Wu, S., Wang, G., Zou, Z., Wu, Z., He, W., et al. (2019) · Nature · doi:10.1038/s41586-019-1424-8 · Cited by 1,082
Self-citationA Low Power, Fully Event-Based Gesture Recognition System
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.781 · Cited by 1,017
Communication in neuronal networks
Laughlin, S. B., & Sejnowski, T. J. (2003) · Science · doi:10.1126/science.1089662 · Cited by 950
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks
Fang, W., Yu, Z., Chen, Y., Masquelier, T., Huang, T., & Tian, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00266 · Cited by 605
Self-citationGlobal Second-Order Pooling Convolutional Networks
Gao, Z., Xie, J., Wang, Q., & Li, P. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00314 · Cited by 524 · cited 1×
“There are many optimizations from various aspects for the modeling of the channel and spatial attention, such as effectiveness [42], space complexity [30], computational complexity [43], etc.”
BackgroundGoing Deeper With Directly-Trained Larger Spiking Neural Networks
Zheng, H., Wu, Y., Deng, L., Hu, Y., & Li, G. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17320 · Cited by 506
Self-citationNeural mechanisms of selective visual attention.
Moore, T., & Zirnsak, M. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033400 · Cited by 471
SLAYER: Spike Layer Error Reassignment in Time
Shrestha, S. B., & Orchard, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08646 · Cited by 444
Rethinking the performance comparison between SNNS and ANNS
Deng, L., Wu, Y., Hu, X., Liang, L., Ding, Y., Li, G., Zhao, G., Li, P., & Xie, Y. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.09.005 · Cited by 343
Self-citationNeuronal Mechanisms of Visual Attention
Maunsell, J. H. R. (2015) · Annual Review of Vision Science · doi:10.1146/annurev-vision-082114-035431 · Cited by 331
Han, B., Srinivasan, G., & Roy, K. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01357 · Cited by 330
Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingPreprint
Deng, S., Li, Y., Zhang, S., & Gu, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.11946 · Cited by 277
Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks
Yin, B., Corradi, F., & Bohté, S. M. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00397-w · Cited by 259
Attention enhances synaptic efficacy and the signal-to-noise ratio in neural circuits
Briggs, F., Mangun, G. R., & Usrey, W. M. (2013) · Nature · doi:10.1038/nature12276 · Cited by 217
Deep Residual Learning in Spiking Neural Networks
Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., & Tian, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.04159 · Cited by 209
Self-citationOptimal Conversion of Conventional Artificial Neural Networks to Spiking Neural NetworksPreprint
Deng, S., & Gu, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00476 · Cited by 196
Temporal-wise Attention Spiking Neural Networks for Event Streams Classification
Yao, M., Gao, H., Zhao, G., Wang, D., Lin, Y., Yang, Z., & Li, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01006 · Cited by 183
Self-citationStöckl, C., & Maass, W. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00311-4 · Cited by 176
EV-Gait: Event-Based Robust Gait Recognition Using Dynamic Vision Sensors
Wang, Y., Du, B., Shen, Y., Wu, K., Zhao, G., Sun, J., & Wen, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00652 · Cited by 160
Self-citationA Simple and Light-Weight Attention Module for Convolutional Neural Networks
Park, J., Woo, S., Lee, J.-Y., & Kweon, I. S. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01283-0 · Cited by 159
Progressive Tandem Learning for Pattern Recognition With Deep Spiking Neural Networks
Wu, J., Xu, C., Han, X. J., Zhou, D., Zhang, M., Li, H., & Tan, K. C. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3114196 · Cited by 153
Kundu, S., Datta, G., Pedram, M., & Beerel, P. A. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00400 · Cited by 128
Optimized Potential Initialization for Low-Latency Spiking Neural Networks
Bu, T., Ding, J., Yu, Z., & Huang, T. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19874 · Cited by 103
Event-Stream Representation for Human Gaits Identification Using Deep Neural Networks
Wang, Y., Zhang, X., Shen, Y., Du, B., Zhao, G., Lizhen, L. C. C., & Wen, H. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3054886 · Cited by 94
Self-citationVisual explanations from spiking neural networks using inter-spike intervals
Kim, Y., & Panda, P. (2021) · Scientific Reports · doi:10.1038/s41598-021-98448-0 · Cited by 91
High Speed and High Dynamic Range Video with an Event Camera
Rebecq, H., Ranftl, R., Koltun, V., & Scaramuzza, D. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2963386 · Cited by 50 · cited 1×
“high-speed object tracking [39], low-latency interaction [26], etc.”
Chen, Z., Deng, L., Wang, B., Li, G., & Xie, Y. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3010201 · Cited by 43
Self-citationCovert Attention with a Spiking Neural Network
Chevallier, S., & Tarroux, P. (2008) · Springer eBooks · doi:10.1007/978-3-540-79547-6_6 · Cited by 13
Selective Attention and Consciousness: Investigating Their Relation Through Computational Modelling
Neokleous, K. C., Avraamides, M. N., Neocleous, C. K., & Schizas, C. N. (2010) · Cognitive Computation · doi:10.1007/s12559-010-9063-2 · Cited by 13
Xu, C., Liu, Y., Chen, D., & Yang, Y. (2022) · Symmetry · doi:10.3390/sym14091933 · Cited by 13
MEFNet: Multi-scale Event Fusion Network for Motion DeblurringPreprint
Sun, L., Sakaridis, C., Liang, J., Jiang, Q., Yang, K., Sun, P., Ye, Y., Wang, K., & Gool, L. V. (2021) · arXiv (Cornell University) · Cited by 1
Cited by
SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence
Fang, W., Chen, Y., Ding, J., Yu, Z., Masquelier, T., Chen, D., Huang, L., Zhou, H., Li, G., & Tian, Y. (2023) · Science Advances · doi:10.1126/sciadv.adi1480 · Cited by 444
Self-citationMemristor‐Based Neuromorphic Chips
Duan, X., Cao, Z., Gao, K., Yan, W., Sun, S., Zhou, G., Wu, Z., Ren, F., & Sun, B. (2024) · Advanced Materials · doi:10.1002/adma.202310704 · Cited by 432
Metadata
- Authors
- Yao, M., Zhao, G., Zhang, H., Hu, Y., Deng, L., Tian, Y., Xu, B., & Li, G.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-08
- Volume
- 45
- Issue
- 8
- Pages
- 9393-9410
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4319866562
- Cited by
- 266
- References
- 59
- FWCI
- 19.50
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.