The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Rethinking the performance comparison between SNNS and ANNS
Abstract
Artificial neural networks (ANNs), a popular path towards artificial intelligence, have experienced remarkable success via mature models, various benchmarks, open-source datasets, and powerful computing platforms. Spiking neural networks (SNNs), a category of promising models to mimic the neuronal dynamics of the brain, have gained much attention for brain inspired computing and been widely deployed on neuromorphic devices. However, for a long time, there are ongoing debates and skepticisms about the value of SNNs in practical applications. Except for the low power attribute benefit from the spike-driven processing, SNNs usually perform worse than ANNs especially in terms of the application accuracy. Recently, researchers attempt to address this issue by borrowing learning methodologies from ANNs, such as backpropagation, to train high-accuracy SNN models. The rapid progress in this domain continuously produces amazing results with ever-increasing network size, whose growing path seems similar to the development of deep learning. Although these ways endow SNNs the capability to approach the accuracy of ANNs, the natural superiorities of SNNs and the way to outperform ANNs are potentially lost due to the use of ANN-oriented workloads and simplistic evaluation metrics. In this paper, we take the visual recognition task as a case study to answer the questions of "what workloads are ideal for SNNs and how to evaluate SNNs makes sense". We design a series of contrast tests using different types of datasets (ANN-oriented and SNN-oriented), diverse processing models, signal conversion methods, and learning algorithms. We propose comprehensive metrics on the application accuracy and the cost of memory & compute to evaluate these models, and conduct extensive experiments. We evidence the fact that on ANN-oriented workloads, SNNs fail to beat their ANN counterparts; while on SNN-oriented workloads, SNNs can fully perform better. We further demonstrate that in SNNs there exists a trade-off between the application accuracy and the execution cost, which will be affected by the simulation time window and firing threshold. Based on these abundant analyses, we recommend the most suitable model for each scenario. To the best of our knowledge, this is the first work using systematical comparisons to explicitly reveal that the straightforward workload porting from ANNs to SNNs is unwise although many works are doing so and a comprehensive evaluation indeed matters. Finally, we highlight the urgent need to build a benchmarking framework for SNNs with broader tasks, datasets, and metrics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
(2008) · Journal of Machine Learning Research · Cited by 33,051
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Hodgkin, A. L., & Huxley, A. F. (1952) · The Journal of Physiology · doi:10.1113/jphysiol.1952.sp004764 · Cited by 23,191
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Research works printed but not linked: 17
- Banner, R., Hubara, I., Hoffer, E., & Soudry, D. (2018). Scalable methods for 8-bit training of neural networks, arXiv preprint arXiv:1805.11046. · arXiv
- Ghaeini, R., Hasan, S. A., Datla, V., Liu, J., Lee, K., & Qadir, A., et al. (2018). Dr-bilstm: Dependent reading bidirectional lstm for natural language inference, arXiv preprint arXiv:1802.05577. · DOI
- Hu, B., Lu, Z., Li, H., & Chen, Q. (2014). Convolutional neural network architectures for matching natural language sentences, In: Advances in neural information processing systems, pp. 2042–2050.
- Hu, Y., Tang, H., Wang, Y., & Pan, G. (2018). Spiking deep residual network, arXiv preprint arXiv:1805.01352. · arXiv
- Iyer, L. R., Chua, Y., & Li, H. (2018). Is neuromorphic mnist neuromorphic? analyzing the discriminative power of neuromorphic datasets in the time domain, arXiv preprint arXiv:1807.01013. · arXiv
- Jahani, 2018, A comparison between the application of empirical and ann methods for estimation of daily global solar radiation in iran, Theoretical and Applied Climatology, 1
- Jin, Y., Li, P., & Zhang, W. (2018). Hybrid macro/micro level backpropagation for training deep spiking neural networks, arXiv preprint arXiv:1805.07866. · arXiv
- Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016). Pruning filters for efficient convnets, arXiv preprint arXiv:1608.08710. · arXiv
- Liu, L., Deng, L., Hu, X., Zhu, M., Li, G., & Ding, Y., et al. (2018). Dynamic sparse graph for efficient deep learning, arXiv preprint arXiv:1810.00859. · arXiv
- Mozafari, M., Ganjtabesh, M., Nowzari-Dalini, A., Thorpe, S. J., & Masquelier, T. (2018). Combining stdp and reward-modulated stdp in deep convolutional spiking neural networks for digit recognition arXiv preprint arXiv:1804.00227. · arXiv
- Mueggler, 2017, The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and slam, International Journal of Robotics Research
- Novikov, A., Podoprikhin, D., Osokin, A., & Vetrov, D. P. (2015). Tensorizing neural networks, In: Advances in neural information processing systems, pp. 442–450.
- Shi, L., Pei, J., Deng, N., Wang, D., Deng, L., & Wang, Y., et al. (2015). Development of a neuromorphic computing system, In: 2015 IEEE international electron devices meeting (IEDM), pp. 4.3.1–4.3.4. · DOI
- Shrestha, S. B., & Orchard, G. (2018). Slayer: Spike layer error reassignment in time, In: Advances in neural information processing systems, pp. 1412–1421.
- Tavanaei, A., & Maida, A. S. (2017). Bp-stdp: Approximating backpropagation using spike timing dependent plasticity, arXiv preprint arXiv:1711.04214. · arXiv
- Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., & Shi, L. (2019). Direct training for spiking neural networks: Faster, larger, better, In: Proceedings of the AAAI conference on artificial intelligence, Vol. 33, pp. 1311–1318. · DOI · Paper page
- Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018). Learning transferable architectures for scalable image recognition, In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697–8710. · DOI · Paper page
Cited by
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks
Fang, W., Yu, Z., Chen, Y., Masquelier, T., Huang, T., & Tian, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00266 · Cited by 605
Attention Spiking Neural Networks
Yao, M., Zhao, G., Zhang, H., Hu, Y., Deng, L., Tian, Y., Xu, B., & Li, G. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3241201 · Cited by 266
Self-citationAbavisani, M., Khoshrou, A., Foroushan, S. K., Ebadpour, N., & Sahebkar, A. (2024) · Current Research in Biotechnology · doi:10.1016/j.crbiot.2024.100211 · Cited by 75
Spiking Neural Networks—Part III: Neuromorphic Communications
Skatchkovsky, N., Jang, H., & Simeone, O. (2021) · IEEE Communications Letters · doi:10.1109/lcomm.2021.3050212 · Cited by 14
Predictive coding with spiking neurons and feedforward gist signaling
Lee, K., Dora, S., Mejias, J. F., Bohte, S. M., & Pennartz, C. M. A. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1338280 · Cited by 14
“2.6.”
Influential
Metadata
- Authors
- Deng, L., Wu, Y., Hu, X., Liang, L., Ding, Y., Li, G., Zhao, G., Li, P., & Xie, Y.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2020-01
- Volume
- 121
- Pages
- 294-307
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2974328520
- Cited by
- 343
- References
- 58
- FWCI
- 8.46
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- Funders
- National Natural Science Foundation of China
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Rethinking the performance comparison between SNNS and ANNS. https://app.cochat.ai/papers/10.1016/j.neunet.2019.09.005
- The paper
- Deng, L., Wu, Y., Hu, X., Liang, L., Ding, Y., Li, G., Zhao, G., Li, P., & Xie, Y. (2020). Rethinking the performance comparison between SNNS and ANNS. Neural Networks. https://doi.org/10.1016/j.neunet.2019.09.005
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.