The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Rethinking the performance comparison between SNNS and ANNS
Abstract
Artificial neural networks (ANNs), a popular path towards artificial intelligence, have experienced remarkable success via mature models, various benchmarks, open-source datasets, and powerful computing platforms. Spiking neural networks (SNNs), a category of promising models to mimic the neuronal dynamics of the brain, have gained much attention for brain inspired computing and been widely deployed on neuromorphic devices. However, for a long time, there are ongoing debates and skepticisms about the value of SNNs in practical applications. Except for the low power attribute benefit from the spike-driven processing, SNNs usually perform worse than ANNs especially in terms of the application accuracy. Recently, researchers attempt to address this issue by borrowing learning methodologies from ANNs, such as backpropagation, to train high-accuracy SNN models. The rapid progress in this domain continuously produces amazing results with ever-increasing network size, whose growing path seems similar to the development of deep learning. Although these ways endow SNNs the capability to approach the accuracy of ANNs, the natural superiorities of SNNs and the way to outperform ANNs are potentially lost due to the use of ANN-oriented workloads and simplistic evaluation metrics. In this paper, we take the visual recognition task as a case study to answer the questions of "what workloads are ideal for SNNs and how to evaluate SNNs makes sense". We design a series of contrast tests using different types of datasets (ANN-oriented and SNN-oriented), diverse processing models, signal conversion methods, and learning algorithms. We propose comprehensive metrics on the application accuracy and the cost of memory & compute to evaluate these models, and conduct extensive experiments. We evidence the fact that on ANN-oriented workloads, SNNs fail to beat their ANN counterparts; while on SNN-oriented workloads, SNNs can fully perform better. We further demonstrate that in SNNs there exists a trade-off between the application accuracy and the execution cost, which will be affected by the simulation time window and firing threshold. Based on these abundant analyses, we recommend the most suitable model for each scenario. To the best of our knowledge, this is the first work using systematical comparisons to explicitly reveal that the straightforward workload porting from ANNs to SNNs is unwise although many works are doing so and a comprehensive evaluation indeed matters. Finally, we highlight the urgent need to build a benchmarking framework for SNNs with broader tasks, datasets, and metrics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
(2008) · Journal of Machine Learning Research · Cited by 33,051
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Hodgkin, A. L., & Huxley, A. F. (1952) · The Journal of Physiology · doi:10.1113/jphysiol.1952.sp004764 · Cited by 23,191
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Simple model of spiking neurons
Izhikevich, E. M. (2003) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2003.820440 · Cited by 4,828
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537
A million spiking-neuron integrated circuit with a scalable communication network and interface
Merolla, P. A., Arthur, J. V., Alvarez-Icaza, R., Cassidy, A. S., Sawada, J., Akopyan, F., Jackson, B. L., Imam, N., Guo, C., Nakamura, Y., et al. (2014) · Science · doi:10.1126/science.1254642 · Cited by 4,147
Loihi: A Neuromorphic Manycore Processor with On-Chip Learning
Davies, M., Srinivasa, N., Lin, T.-H., Chinya, G., Cao, Y., Choday, S. H., Dimou, G., Joshi, P., Imam, N., Jain, S., et al. (2018) · IEEE Micro · doi:10.1109/mm.2018.112130359 · Cited by 3,897
Networks of spiking neurons: The third generation of neural network models
Maass, W. (1997) · Neural Networks · doi:10.1016/s0893-6080(97)00011-7 · Cited by 3,481
Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks
Chen, Y.-H., Krishna, T., Emer, J. S., & Sze, V. (2017) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2016.2616357 · Cited by 3,247
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
A 128$\times$128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor
Lichtsteiner, P., Posch, C., & Delbruck, T. (2008) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2007.914337 · Cited by 2,485
Convolutional Neural Networks for Speech Recognition
Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., Deng, L., Penn, G., & Yu, D. (2014) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2014.2339736 · Cited by 2,262
Self-citationTrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip
Akopyan, F., Sawada, J., Cassidy, A., Alvarez-Icaza, R., Arthur, J., Merolla, P., Imam, N., Nakamura, Y., Datta, P., Nam, G.-J., et al. (2015) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2015.2474396 · Cited by 1,687
Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition
Gerstner, W., Kistler, W. M., Naud, R., & Paninski, L. (2014) · doi:10.1017/cbo9781107447615 · Cited by 1,515
Unsupervised learning of digit recognition using spike-timing-dependent plasticity
Diehl, P. U., & Cook, M. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00099 · Cited by 1,452
Hybrid computing using a neural network with dynamic external memory
Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016) · Nature · doi:10.1038/nature20101 · Cited by 1,412
Furber, S. B., Galluppi, F., Temple, S., & Plana, L. A. (2014) · Proceedings of the IEEE · doi:10.1109/jproc.2014.2304638 · Cited by 1,350
DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · Proceedings of the 19th international conference on Architectural support for programming languages and operating systems · doi:10.1145/2541940.2541967 · Cited by 1,297
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080246 · Cited by 1,240
Going Deeper in Spiking Neural Networks: VGG and Residual Architectures
Sengupta, A., Ye, Y., Wang, R., Liu, C., & Roy, K. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00095 · Cited by 1,211
Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks
Wu, Y., Deng, L., Li, G., Zhu, J., & Shi, L. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00331 · Cited by 1,196
Self-citationRecent Trends in Deep Learning Based Natural Language Processing
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02709 · Cited by 1,192
Rueckauer, B., Lungu, I.-A., Hu, Y., Pfeiffer, M., & Liu, S.-C. (2017) · Frontiers in Neuroscience · doi:10.3389/fnins.2017.00682 · Cited by 1,072
GHOSH-DASTIDAR, S., & ADELI, H. (2009) · International Journal of Neural Systems · doi:10.1142/s0129065709002002 · Cited by 1,057
Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing
Diehl, P. U., Neil, D., Binas, J., Cook, M., Liu, S.-C., & Pfeiffer, M. (2015) · 2015 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280696 · Cited by 991
Training Deep Spiking Neural Networks Using Backpropagation
Lee, J. H., Delbruck, T., & Pfeiffer, M. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00508 · Cited by 976
Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition
Cao, Y., Chen, Y., & Khosla, D. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0788-3 · Cited by 917
Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades
Orchard, G., Jayawant, A., Cohen, G. K., & Thakor, N. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00437 · Cited by 882
Convolutional networks for fast, energy-efficient neuromorphic computing
Esser, S. K., Merolla, P. A., Arthur, J. V., Cassidy, A. S., Appuswamy, R., Andreopoulos, A., Berg, D. J., McKinstry, J. L., Melano, T., Barch, D. R., et al. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1604850113 · Cited by 781
Vidal, A. R., Rebecq, H., Horstschaefer, T., & Scaramuzza, D. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2793357 · Cited by 541
CIFAR10-DVS: An Event-Stream Dataset for Object Classification
Li, H., Liu, H., Ji, X., Li, G., & Shi, L. (2017) · Frontiers in Neuroscience · doi:10.3389/fnins.2017.00309 · Cited by 538
Self-citationEV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
Zhu, A., Yuan, L., Chaney, K., & Daniilidis, K. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.062 · Cited by 470
Neural Networks (IJCNN), The 2011 International Joint Conference on
Pokharel, R., Seth, S., & Principe, J. C. (2011) · CiteSeer X (The Pennsylvania State University) · Cited by 429
Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on
Ferrari, V., Marín-Jiménez, M. J., & Zisserman, A. (2008) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 397
2016 International Joint Conference on Neural Networks (IJCNN)
(2016) · doi:10.1109/ijcnn38550.2016 · Cited by 303
Yin, S., Ouyang, P., Tang, S., Tu, F., Li, X., Zheng, S., Lu, T., Gu, J., Liu, L., & Wei, S. (2018) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2017.2778281 · Cited by 212
Lee, C., Panda, P., Srinivasan, G., & Roy, K. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00435 · Cited by 205
Noise as a Resource for Computation and Learning in Networks of Spiking Neurons
Maass, W. (2014) · Proceedings of the IEEE · doi:10.1109/jproc.2014.2310593 · Cited by 198
Training Spiking Deep Networks for Neuromorphic HardwarePreprint
Hunsberger, E., & Eliasmith, C. (2016) · arXiv (Cornell University) · doi:10.13140/rg.2.2.10967.06566 · Cited by 166
Deng, L., Jiao, P., Pei, J., Wu, Z., & Li, G. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.01.010 · Cited by 165
Self-citationPoker-DVS and MNIST-DVS. Their History, How They Were Made, and Other Details
Serrano-Gotarredona, T., & Linares-Barranco, B. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00481 · Cited by 161
Spiking neurons can discover predictive features by aggregate-label learning
Gütig, R. (2016) · Science · doi:10.1126/science.aab4113 · Cited by 127
Stromatias, E., Neil, D., Pfeiffer, M., Galluppi, F., Furber, S. B., & Liu, S. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00222 · Cited by 118
Spiking Optical Flow for Event-Based Sensors Using IBM's TrueNorth Neurosynaptic System
Haessig, G., Cassidy, A. S., Alvarez-Icaza, R., Benosman, R. B., & Orchard, G. (2018) · IEEE Transactions on Biomedical Circuits and Systems · doi:10.1109/tbcas.2018.2834558 · Cited by 112
Solving Constraint Satisfaction Problems with Networks of Spiking Neurons
Jonke, Z., Habenschuss, S., & Maass, W. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00118 · Cited by 76
A self-driving robot using deep convolutional neural networks on neuromorphic hardware
Hwu, T. J., Isbell, J., Oros, N., & Krichmar, J. L. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7965912 · Cited by 66
Gaussian Process Lstm Recurrent Neural Network Language Models for Speech Recognition
Lam, M. W. Y., Chen, X., Hu, S., Yu, J., Liu, X., & Meng, H. M. L. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683660 · Cited by 37
Exploiting Multi-layer Features Using a CNN-RNN Approach for RGB-D Object Recognition
Çağlayan, A., & Can, A. B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11015-4_51 · Cited by 31
An Improved Approach Based on CNN-RNNs for Mathematical Expression Recognition
Zhang, W., Bai, Z., & Zhu, Y. (2019) · International Conference on Multimedia Systems and Signal Processing (ICMSSP) · doi:10.1145/3330393.3330410 · Cited by 27
Object-dependent sparse representation for extracellular spike detection
Shi, G., Liu, Z., Wang, X., et al. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.05.082 · Cited by 10
Research works printed but not linked: 17
- Banner, R., Hubara, I., Hoffer, E., & Soudry, D. (2018). Scalable methods for 8-bit training of neural networks, arXiv preprint arXiv:1805.11046. · arXiv
- Ghaeini, R., Hasan, S. A., Datla, V., Liu, J., Lee, K., & Qadir, A., et al. (2018). Dr-bilstm: Dependent reading bidirectional lstm for natural language inference, arXiv preprint arXiv:1802.05577. · DOI
- Hu, B., Lu, Z., Li, H., & Chen, Q. (2014). Convolutional neural network architectures for matching natural language sentences, In: Advances in neural information processing systems, pp. 2042–2050.
- Hu, Y., Tang, H., Wang, Y., & Pan, G. (2018). Spiking deep residual network, arXiv preprint arXiv:1805.01352. · arXiv
- Iyer, L. R., Chua, Y., & Li, H. (2018). Is neuromorphic mnist neuromorphic? analyzing the discriminative power of neuromorphic datasets in the time domain, arXiv preprint arXiv:1807.01013. · arXiv
- Jahani, 2018, A comparison between the application of empirical and ann methods for estimation of daily global solar radiation in iran, Theoretical and Applied Climatology, 1
- Jin, Y., Li, P., & Zhang, W. (2018). Hybrid macro/micro level backpropagation for training deep spiking neural networks, arXiv preprint arXiv:1805.07866. · arXiv
- Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016). Pruning filters for efficient convnets, arXiv preprint arXiv:1608.08710. · arXiv
- Liu, L., Deng, L., Hu, X., Zhu, M., Li, G., & Ding, Y., et al. (2018). Dynamic sparse graph for efficient deep learning, arXiv preprint arXiv:1810.00859. · arXiv
- Mozafari, M., Ganjtabesh, M., Nowzari-Dalini, A., Thorpe, S. J., & Masquelier, T. (2018). Combining stdp and reward-modulated stdp in deep convolutional spiking neural networks for digit recognition arXiv preprint arXiv:1804.00227. · arXiv
- Mueggler, 2017, The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and slam, International Journal of Robotics Research
- Novikov, A., Podoprikhin, D., Osokin, A., & Vetrov, D. P. (2015). Tensorizing neural networks, In: Advances in neural information processing systems, pp. 442–450.
- Shi, L., Pei, J., Deng, N., Wang, D., Deng, L., & Wang, Y., et al. (2015). Development of a neuromorphic computing system, In: 2015 IEEE international electron devices meeting (IEDM), pp. 4.3.1–4.3.4. · DOI
- Shrestha, S. B., & Orchard, G. (2018). Slayer: Spike layer error reassignment in time, In: Advances in neural information processing systems, pp. 1412–1421.
- Tavanaei, A., & Maida, A. S. (2017). Bp-stdp: Approximating backpropagation using spike timing dependent plasticity, arXiv preprint arXiv:1711.04214. · arXiv
- Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., & Shi, L. (2019). Direct training for spiking neural networks: Faster, larger, better, In: Proceedings of the AAAI conference on artificial intelligence, Vol. 33, pp. 1311–1318. · DOI · Paper page
- Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018). Learning transferable architectures for scalable image recognition, In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697–8710. · DOI · Paper page
Cited by
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks
Fang, W., Yu, Z., Chen, Y., Masquelier, T., Huang, T., & Tian, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00266 · Cited by 605
Attention Spiking Neural Networks
Yao, M., Zhao, G., Zhang, H., Hu, Y., Deng, L., Tian, Y., Xu, B., & Li, G. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3241201 · Cited by 266
Self-citationAbavisani, M., Khoshrou, A., Foroushan, S. K., Ebadpour, N., & Sahebkar, A. (2024) · Current Research in Biotechnology · doi:10.1016/j.crbiot.2024.100211 · Cited by 75
Spiking Neural Networks—Part III: Neuromorphic Communications
Skatchkovsky, N., Jang, H., & Simeone, O. (2021) · IEEE Communications Letters · doi:10.1109/lcomm.2021.3050212 · Cited by 14
Predictive coding with spiking neurons and feedforward gist signaling
Lee, K., Dora, S., Mejias, J. F., Bohte, S. M., & Pennartz, C. M. A. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1338280 · Cited by 14
“2.6.”
Influential
Metadata
- Authors
- Deng, L., Wu, Y., Hu, X., Liang, L., Ding, Y., Li, G., Zhao, G., Li, P., & Xie, Y.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2020-01
- Volume
- 121
- Pages
- 294-307
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2974328520
- Cited by
- 343
- References
- 58
- FWCI
- 8.46
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- Funders
- National Natural Science Foundation of China
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.