Understanding error propagation in deep learning neural network (DNN) accelerators and applications
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
Deep learning neural networks (DNNs) have been successful in solving a wide range of machine learning problems. Specialized hardware accelerators have been proposed to accelerate the execution of DNN algorithms for high-performance and energy efficiency. Recently, they have been deployed in datacenters (potentially for business-critical or industrial applications) and safety-critical systems such as self-driving cars. Soft errors caused by high-energy particles have been increasing in hardware systems, and these can lead to catastrophic failures in DNN systems. Traditional methods for building resilient systems, e.g., Triple Modular Redundancy (TMR), are agnostic of the DNN algorithm and the DNN accelerator's architecture. Hence, these traditional resilience approaches incur high overheads, which makes them challenging to deploy. In this paper, we experimentally evaluate the resilience characteristics of DNN systems (i.e., DNN software running on specialized accelerators). We find that the error resilience of a DNN system depends on the data types, values, data reuses, and types of layers in the design. Based on our observations, we propose two efficient protection techniques for DNN systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011 · cited 1×
“Deep learning neural network (DNN) applications are widely used in high-performance computing systems and datacenters [4, 18, 47].”
BackgroundConvolutional networks and applications in vision
LeCun, Y., Kavukcuoglu, K., & Farabet, C. (2010) · Proceedings of 2010 IEEE International Symposium on Circuits and Systems · doi:10.1109/iscas.2010.5537907 · Cited by 2,213 · cited 1×
“A deep learning neural network (DNN) is a directed acyclic graph consisting of multiple computation layers [34].”
BackgroundOptimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks
Zhang, C., Li, P., Sun, G., Guan, Y., Xiao, B., & Cong, J. (2015) · Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/2684746.2689060 · Cited by 2,158 · cited 1×
“[61], Diannao [12], Dadiannao [14] x x x”
BackgroundDaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1109/micro.2014.58 · Cited by 1,492
Research works printed but not linked: 11
- Autonomous car facts 2016. Keynote: Autonomous Car A New Driver for Resilient Computing and Design-for-Test. (2016). Retrieved Oct. 2016 from https://nepp.nasa.gov/workshops/etw2016/talks/15WED/20160615-0930-Autonomous_Saxena-Nirmal-Saxena-Rec2016Jun16-nasaNEPP.pdf Autonomous car facts 2016. Keynote: Autonomous Car A New Driver for Resilient Computing and Design-for-Test. (2016). Retrieved Oct. 2016 from https://nepp.nasa.gov/workshops/etw2016/talks/15WED/20160615-0930-Autonomous_Saxena-Nirmal-Saxena-Rec2016Jun16-nasaNEPP.pdf
- Caffe Model 2014. Caffe Model Zoo. (2014). Retrieved Oct. 2016 from http://caffe.berkeleyvision.org/model_zoo.html Caffe Model 2014. Caffe Model Zoo. (2014). Retrieved Oct. 2016 from http://caffe.berkeleyvision.org/model_zoo.html
- CaffeNet 2014. CaffeNet . ( 2014 ). Retrieved Oct. 2016 from http://caffe.berkeleyvision.org/model_zoo.html CaffeNet 2014. CaffeNet. (2014). Retrieved Oct. 2016 from http://caffe.berkeleyvision.org/model_zoo.html
- ConvNet 2014. High-performance C++/CUDA implementation of convolutional neural networks. ( 2014 ). Retrieved Oct. 2016 from https://code.google.com/p/cuda-convnet ConvNet 2014. High-performance C++/CUDA implementation of convolutional neural networks. (2014). Retrieved Oct. 2016 from https://code.google.com/p/cuda-convnet
- 10.5555/2354410.2355132 · doi:10.5555/2354410.2355132
- IEC 61508 2016. Functional Safety and IEC 61508. (2016). Retrieved Oct. 2016 from http://www.iec.ch/functionalsafety/ IEC 61508 2016. Functional Safety and IEC 61508. (2016). Retrieved Oct. 2016 from http://www.iec.ch/functionalsafety/
- Lin , Min and Chen , Qiang and Yan , Shuicheng . 2013. Network in network. arXiv preprint arXiv:1312.4400 ( 2013 ). Lin, Min and Chen, Qiang and Yan, Shuicheng. 2013. Network in network. arXiv preprint arXiv:1312.4400 (2013). · arXiv
- Neale Adam and Manoj Sachdev. 2016. Neutron Radiation Induced Soft Error Rates for an Adjacent-ECC Protected SRAM in 28 nm CMOS. (2016). Neale Adam and Manoj Sachdev. 2016. Neutron Radiation Induced Soft Error Rates for an Adjacent-ECC Protected SRAM in 28 nm CMOS. (2016). · doi:10.1109/tns.2016.2547963
- 10.1109/CGO.2005.34 · doi:10.1109/cgo.2005.34
- TPU {n. d.}. Google supercharges machine learning tasks with TPU custom chip. ({n. d.}). Retrieved Oct. 2016 from https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html TPU {n. d.}. Google supercharges machine learning tasks with TPU custom chip. ({n. d.}). Retrieved Oct. 2016 from https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html
- Yann LeCun . 2000. Deep Learning and the Future of AI. (2000). Retrieved Apr. 2017 from https://indico.cern.ch/event/510372/ Yann LeCun. 2000. Deep Learning and the Future of AI. (2000). Retrieved Apr. 2017 from https://indico.cern.ch/event/510372/
Cited by
A Survey on Distributed Machine Learning
Verbraeken, J., Wolting, M., Katzy, J., Kloppenburg, J., Verbelen, T., & Rellermeyer, J. S. (2021) · ACM Computing Surveys · doi:10.1145/3377454 · Cited by 872
The Hidden Vulnerability of Distributed Learning in Byzantium
Mhamdi, E. M. E., Guerraoui, R., & Rouault, S. (2018) · arXiv · doi:10.48550/arxiv.1802.07927 · Cited by 648
Analyzing and Increasing the Reliability of Convolutional Neural Networks on GPUs
Santos, F. F. D., Pimenta, P. F., Lunardi, C., Draghetti, L., Carro, L., Kaeli, D., & Rech, P. (2019) · IEEE Transactions on Reliability · doi:10.1109/tr.2018.2878387 · Cited by 226
“Some solutions require specific hardware mod-ifications [10], [16].”
BackgroundAres: A framework for quantifying the resilience of deep neural networks
Reagen, B., Gupta, U., Pentecost, L., Whatmough, P., Lee, S. K., Mulholland, N., Brooks, D., & Wei, G.-Y. (2018) · 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC) · doi:10.1109/dac.2018.8465834 · Cited by 189
Perez-Cerrolaza, J., Abella, J., Borg, M., Donzella, C., Cerquides, J., Cazorla, F. J., Englund, C., Tauber, M., Nikolakopoulos, G., & Flores, J. L. (2024) · ACM Computing Surveys · doi:10.1145/3626314 · Cited by 185
A survey on modeling and improving reliability of DNN algorithms and accelerators
Mittal, S. (2020) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2019.101689 · Cited by 175
PyTorchFI: A Runtime Perturbation Tool for DNNs
Mahmoud, A., Aggarwal, N., Nobbe, A., Vicarte, J. R. S., Adve, S. V., Fletcher, C. W., Frosio, I., & Hari, S. K. S. (2020) · 2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W) · doi:10.1109/dsn-w50199.2020.00014 · Cited by 140
Self-citationA Reliability Analysis of a Deep Neural Network
Bosio, A., Bernardi, P., Ruospo, A., & Sanchez, E. (2019) · 2019 IEEE Latin American Test Symposium (LATS) · doi:10.1109/latw.2019.8704548 · Cited by 114
“As already discussed in [12] this is the most critical observed fault;”
BackgroundResultInfluentialSoft errors in DNN accelerators: A comprehensive review
Ibrahim, Y., Wang, H., Liu, J., Wei, J., Chen, L., Rech, P., Adam, K., & Guo, G. (2020) · Microelectronics Reliability · doi:10.1016/j.microrel.2020.113969 · Cited by 102
TensorFI: A Flexible Fault Injection Framework for TensorFlow Applications
Chen, Z., Narayanan, N., Fang, B., Li, G., Pattabiraman, K., & DeBardeleben, N. (2020) · 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre5003.2020.00047 · Cited by 93
“, mutation testing [24]) or as a single bit-flip [18]–[21].”
BackgroundMethodResultInfluentialSelf-citationBinFI: an efficient fault injector for safety-critical machine learning systems
Chen, Z., Li, G., Pattabiraman, K., & DeBardeleben, N. (2019) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3295500.3356177 · Cited by 91
Self-citationFIdelity: Efficient Resilience Analysis Framework for Deep Learning Accelerators
He, Y., Balaprakash, P., & Li, Y. (2020) · 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00033 · Cited by 83
Rech, P. (2024) · IEEE Transactions on Nuclear Science · doi:10.1109/tns.2024.3349956 · Cited by 75
“Some studies also show that transient faults can result in a correct detection (the BB area and position are correct) but in a wrong classification [62], as depicted in Fig.”
BackgroundSoft Error Resilience of Deep Residual Networks for Object Recognition
Ibrahim, Y., Wang, H., Bai, M., Liu, Z., Wang, J., Yang, Z., & Chen, Z. (2020) · IEEE Access · doi:10.1109/access.2020.2968129 · Cited by 72
Towards Energy-Efficient and Secure Edge AI: A Cross-Layer Framework ICCAD Special Session Paper
Shafique, M., Marchisio, A., Wicaksana Putra, R. V., & Hanif, M. A. (2021) · 2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD) · doi:10.1109/iccad51958.2021.9643539 · Cited by 58
“Transient Faults (Soft Errors): Soft error rates have been increasing in HW systems [52].”
BackgroundMachine learning modeling practices to support the principles of AI and ethics in nutrition research
Thomas, D. M., Kleinberg, S., Brown, A. W., Crow, M., Bastian, N. D., Reisweber, N., Lasater, R., Kendall, T., Shafto, P., Blaine, R., et al. (2022) · Nutrition & Diabetes · doi:10.1038/s41387-022-00226-y · Cited by 48
“Specific methods to address individual AI/ML models exist, but there does not exist a one size fits all solution to generally characterize error propagation within AI/ML models [51].”
BackgroundMethodSoft Error Reliability Analysis of Vision Transformers
Xue, X., Liu, C., Wang, Y., Yang, B., Luo, T., Zhang, L., Li, H., & Li, X. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2023.3317138 · Cited by 35
Maillard, P., Chen, Y. P., Vidmar, J., Fraser, N., Gambardella, G., Sawant, M., & Voogel, M. L. (2023) · IEEE Transactions on Nuclear Science · doi:10.1109/tns.2022.3216360 · Cited by 28
Nitika, N., Keerthiveena, B., Thakur, G., & Rathore, A. S. (2024) · Pharmaceutical Research · doi:10.1007/s11095-024-03663-9 · Cited by 26
“Four machine learning frameworks were included in the current study to estimate the charge variants: Support vector regression (SVR), Random Forest (RF), Deep neural network (DNN), and Convolutional neural network (CNN) [55, 56, 59].”
Fine-Grained Fault Sensitivity Analysis of Vision Transformers Under Soft Errors
He, J., Liu, Y., Xu, C., Liao, X., & Yang, Y. (2025) · Electronics · doi:10.3390/electronics14122418 · Cited by 4
Fast and Accurate Error Simulation for CNNs Against Soft Errors
Bolchini, C., Cassano, L., Miele, A., & Toschi, A. (2023) · IEEE Transactions on Computers · doi:10.1109/tc.2022.3184274 · Cited by 1
Metadata
- Authors
- Li, G., Hari, S. K. S., Sullivan, M., Tsai, T., Pattabiraman, K., Emer, J., & Keckler, S. W.
- Venue
- Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- Publisher
- ACM
- Published
- 2017-11-12
- Pages
- 1-12
- ISSN
- 2167-4337 (ISSN-L)
- OpenAlex
- W2767260595
- Cited by
- 492
- References
- 52
- FWCI
- 50.59
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.