Analog/Mixed-Signal Hardware Error Modeling for Deep Learning Inference
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Analog/mixed-signal (AMS) computation can be more energy efficient than digital approaches for deep learning inference, but incurs an accuracy penalty from precision loss. Prior AMS approaches focus on small networks/datasets, which can maintain accuracy even with 2b precision. We analyze applicability of AMS approaches to larger networks by proposing a generic AMS error model, implementing it in an existing training framework, and investigating its effect on ImageNet classification with ResNet-50. We demonstrate significant accuracy recovery by exposing the network to AMS error during retraining, and we show that batch normalization layers are responsible for this accuracy recovery. We also introduce an energy model to predict the requirements of high-accuracy AMS hardware running large networks and use it to show that for ADC-dominated designs, there is a direct tradeoff between energy efficiency and network accuracy. Our model predicts that achieving < 0.4% accuracy loss on ResNet-50 with AMS hardware requires a computation energy of at least ~300 fJ/MAC. Finally, we propose methods for improving the energy-accuracy tradeoff.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We are aware of one recent work that analyzed more complex problems such as ImageNet classification23, however the hardware model used was rather abstract and no experimental validation was presented.”
From Accurate deep neural network inference using computational phase-change memory (2020) · cites this paper as [23]
In the citing paper
“Consistent with previous observations23,33, we found that the network recovers high accuracy extremely quickly when retraining with additive noise due to quick updates of the batch normalization parameters (see Supplementary Note 1), and obtained satisfactory convergence after only 8 epochs.”
From Accurate deep neural network inference using computational phase-change memory (2020) · cites this paper as [23]
In the citing paper
“Based on a recent ADC survey23, 8-bit data conversion is a good tradeoff between precision and energy consumption.”
From Accurate deep neural network inference using computational phase-change memory (2020) · cites this paper as [23]
In the citing paper
“we notice that retraining ResNet with additive noise results mainly in adapting the batch normalization parameters, whereas the weights stay close to the full-precision weights trained without noise.”
From Accurate deep neural network inference using computational phase-change memory (2020) · cites this paper as [23]
Citations
Cites
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The former includes all network-level methods, such as pruning [1, 2], distillation [3], and quantization [4], and all other approaches im-”
MethodEfficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · cited 1×
“that the reuse of weights and/or activations significantly reduces the energy cost of memory access [5].”
Background(1989) · Cited by 3,214
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690 · cited 1×
“Distiller provides out-of-the-box support for several methods of quantization of weights and activations; we use DoReFa, which compresses and quantizes convolutional layer weights, clips and quantizes activations, and allows for quantization in the training loop using a straight-through estimator [28].”
MethodISAAC: a convolutional neural network accelerator with in-situ analog arithmetic in crossbars
Shafiee, A., Nag, A., Muralimanohar, N., Balasubramonian, R., Strachan, J. P., Hu, M., Williams, R. S., & Srikumar, V. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001139 · Cited by 1,536
Learning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481 · cited 1×
“The former includes all network-level methods, such as pruning [1, 2], distillation [3], and quantization [4], and all other approaches im-”
MethodSelf-citationISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars
Shafiee, A., Nag, A., Muralimanohar, N., Balasubramonian, R., Strachan, J. P., Hu, M., Williams, R. S., & Srikumar, V. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.12 · Cited by 775
Hu, M., Strachan, J. P., Li, Z., Grafals, E. M., Davila, N., Graves, C., Lam, S., Ge, N., Yang, J. J., & Williams, R. S. (2016) · Proceedings of the 53rd Annual Design Automation Conference · doi:10.1145/2897937.2898010 · Cited by 686
Analog Versus Digital: Extrapolating from Electronics to Neurobiology
Sarpeshkar, R. (1998) · Neural Computation · doi:10.1162/089976698300017052 · Cited by 496
Accelerator-friendly neural-network training: Learning variations and defects in RRAM crossbar
Chen, L., Li, J., Chen, Y., Deng, Q., Shen, J., Liang, X., & Jiang, L. (2017) · Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017 · doi:10.23919/date.2017.7926952 · Cited by 245 · cited 1×
“The majority of AMS approaches to date have focused on the use of resistive arrays to perform dot product computation - weights are stored as conductances, activations are loaded in as voltages on wordlines, and their products are read out as currents on bitlines [9–21].”
BackgroundAres: a framework for quantifying the resilience of deep neural networks
Reagen, B., Gupta, U., Pentecost, L., Whatmough, P., Lee, S. K., Mulholland, N., Brooks, D., & Wei, G.-Y. (2018) · Proceedings of the 55th Annual Design Automation Conference · doi:10.1145/3195970.3195997 · Cited by 228 · cited 1×
“neural network evaluation, these drawbacks can be ameliorated by the fact that deep networks are often resilient to small amounts of noise [7].”
BackgroundBinary convolutional neural network on RRAM
Tang, T., Xia, L., Li, B., Wang, Y., & Yang, H. (2017) · 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/aspdac.2017.7858419 · Cited by 176
MNSIM: Simulation Platform for Memristor-based Neuromorphic Computing System
Xia, L., Li, B., Tang, T., Gu, P., Chen, P.-Y., Yu, S., Cao, Y., Wang, Y., Xie, Y., & Yang, H. (2017) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2017.2729466 · Cited by 171
Pelgrom, M. J. M. (2022) · Springer eBooks · doi:10.1007/978-3-030-90808-9 · Cited by 168
Bankman, D., Yang, L., Moons, B., Verhelst, M., & Murmann, B. (2018) · 2018 IEEE International Solid - State Circuits Conference - (ISSCC) · doi:10.1109/isscc.2018.8310264 · Cited by 157
Efficient training algorithms for neural networks based on memristive crossbar circuits
Kataeva, I., Merrikh-Bayat, F., Zamanidoost, E., & Strukov, D. (2015) · 2015 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280785 · Cited by 103 · cited 1×
“The majority of AMS approaches to date have focused on the use of resistive arrays to perform dot product computation - weights are stored as conductances, activations are loaded in as voltages on wordlines, and their products are read out as currents on bitlines [9–21].”
BackgroundSwitched-capacitor neural networks
Tsividis, Y., & Anastassiou, D. (1987) · Electronics Letters · doi:10.1049/el:19870674 · Cited by 102 · cited 1×
“Another approach is to use switched-capacitor multipliers, which use passive charge redistribution to convert digital inputs into an analog output representing the scaled product of the inputs [22–24].”
Method2021 55th Asilomar Conference on Signals, Systems, and Computers
(2021) · doi:10.1109/ieeeconf53345.2021 · Cited by 102
Li, B., Xia, L., Gu, P., Wang, Y., & Yang, H. (2015) · Proceedings of the 52nd Annual Design Automation Conference · doi:10.1145/2744769.2744870 · Cited by 94
Ankit, A., Sengupta, A., Panda, P., & Roy, K. (2017) · Proceedings of the 54th Annual Design Automation Conference 2017 · doi:10.1145/3061639.3062311 · Cited by 92
Analysis and Design of a Passive Switched-Capacitor Matrix Multiplier for Approximate Computing
Lee, E. H., & Wong, S. S. (2016) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2016.2599536 · Cited by 60 · cited 1×
“Prior work has focused on demonstrating the functionality of AMS circuits both as standalone computational units and as the hardware fabric on which neural networks are implemented [25, 26].”
NEUTRAMS: neural network transformation and co-design under neuromorphic hardware constraints
Ji, Y., Zhang, Y., Li, S., Chi, P., Jiang, C., Qu, P., Xie, Y., & Chen, W. (2016) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.5555/3195638.3195663 · Cited by 57
Pelgrom, M. J. M. (2012) · Springer eBooks · doi:10.1007/978-1-4614-1371-4_8 · Cited by 54
Mixed-signal circuits for embedded machine-learning applications
Murmann, B., Bankman, D., Chai, E., Miyashita, D., & Yang, L. (2015) · 2015 49th Asilomar Conference on Signals, Systems and Computers · doi:10.1109/acssc.2015.7421361 · Cited by 51 · cited 1×
“…and network inputs and outputs are often digital in nature, approaches in the literature tend to focus on analog/mixed-signal (AMS) implementations, in which the core dot product computation is performed in the analog domain while the inputs and outputs to these analog blocks are digital [6, 8].”
BackgroundAn 8-bit, 16 input, 3.2 pJ/op switched-capacitor dot product circuit in 28-nm FDSOI CMOS
Bankman, D. J., & Murmann, B. (2016) · IEEE Asian Solid-State Circuits Conference (A-SSCC) · doi:10.1109/asscc.2016.7844125 · Cited by 49
Zmora, N., Jacob, G., & Novik, G. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1297430 · Cited by 40
TraNNsformer: Neural network transformation for memristive crossbar based neuromorphic system design
Ankit, A., Sengupta, A., & Roy, K. (2017) · 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) · doi:10.1109/iccad.2017.8203823 · Cited by 39
NervanaSystems/distiller v0.3.2
Zmora, N., Jacob, G., Bar, Zlotnik, L., Novik, G., Barad, H., Chen, Y.-S., Muchsel, R., 103yiran, Tacker-Oh, et al. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1297429 · Cited by 39
Kim, Y., Kim, H., Ahn, D., & Kim, J. (2018) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1145/3218603.3218605 · Cited by 33 · cited 3×
“Our use of Murmann’s ADC survey extends prior work based on first-order ADC energy models [19] to reflect the current state of the art in ADC design.”
MethodTraNNsformer: Neural Network Transformation for memristive crossbar based neuromorphic system designPreprint
Ankit, A., Sengupta, A., & Roy, K. (2017) · arXiv (Cornell University) · doi:10.5555/3199700.3199771 · Cited by 27
Neural Network-Hardware Co-design for Scalable RRAM-based BNN AcceleratorsPreprint
Kim, Y., Kim, H., & Kim, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.02187 · Cited by 26 · cited 1×
“The majority of AMS approaches to date have focused on the use of resistive arrays to perform dot product computation - weights are stored as conductances, activations are loaded in as voltages on wordlines, and their products are read out as currents on bitlines [9–21].”
BackgroundPassive charge redistribution digital‐to‐analogue multiplier
Bankman, D. J., & Murmann, B. (2015) · Electronics Letters · doi:10.1049/el.2014.3995 · Cited by 18
Rx-Caffe: Framework for evaluating and training Deep Neural Networks on Resistive Crossbars.Preprint
Jain, S., Sengupta, A., Roy, K., & Raghunathan, A. (2018) · arXiv (Cornell University) · Cited by 12
Research works printed but not linked: 1
- S. Han , H. Mao , and W. J. Dally . 2016. Deep Compression: Compressing Deep Neural Networks with Pruning , Trained Quantization and Huffman Coding. In International Conference on Learning Representations. San Juan. S. Han, H. Mao, and W. J. Dally. 2016. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In International Conference on Learning Representations. San Juan.
Cited by
Accurate deep neural network inference using computational phase-change memory
Joshi, V., Le Gallo, M., Haefeli, S., Boybat, I., Nandakumar, S. R., Piveteau, C., Dazzi, M., Rajendran, B., Sebastian, A., & Eleftheriou, E. (2020) · Nature Communications · doi:10.1038/s41467-020-16108-9 · Cited by 490
Cai, F., Kumar, S., Van Vaerenbergh, T., Sheng, X., Liu, R., Li, C., Liu, Z., Foltin, M., Yu, S., Xia, Q., et al. (2020) · Nature Electronics · doi:10.1038/s41928-020-0436-6 · Cited by 393
“This peripheral circuitry (particularly the digital/analogue conversions) dominates the system area and power consumption 43 in analogue deep learning accelerators that need to maintain fairly high bit precision across the feed-forward layers.”
BackgroundAccurate deep neural network inference using computational phase-change memory
Joshi, V., Gallo, M. L., Haefeli, S., Boybat, I., Nandakumar, S. R., Piveteau, C., Dazzi, M., Rajendran, B., Sebastian, A., & Eleftheriou, E. S. (2020) · RePEc: Research Papers in Economics · doi:10.3929/ethz-b-000416797 · Cited by 379
A survey on modeling and improving reliability of DNN algorithms and accelerators
Mittal, S. (2020) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2019.101689 · Cited by 175
Mixed-Signal Computing for Deep Neural Network Inference
Murmann, B. (2021) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2020.3020286 · Cited by 160
“Unfortunately, the requirements depend on the DNN algorithm, inference accuracy, signal statistics, and the employed training methods, which can help desensitize the network [39]–[42].”
Method
Metadata
- Authors
- Rekhi, A. S., Zimmer, B., Nedovic, N., Liu, N., Venkatesan, R., Wang, M., Khailany, B., Dally, W. J., & Gray, C. T.
- Affiliations
- Stanford University (US)
- Nvidia (United Kingdom) (GB)
- University of Virginia (US)
- Massachusetts Institute of Technology (US)
- ORCID
- Venue
- Proceedings of the 56th Annual Design Automation Conference 2019
- Publisher
- ACM
- Published
- 2019-06-02
- Pages
- 1-6
- ISSN
- 0738-100X (ISSN-L)
- OpenAlex
- W2945445229
- Cited by
- 67
- References
- 33
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 10 · 2023: 7 · 2022: 10 · 2021: 10 · 2020: 15 · 2019: 3
- FWCI
- 15.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE Design Automation Conference (DAC) (publishedVersion)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Analog and Mixed-Signal Circuit Design
- Keywords
- deep learning inference, error modeling, precision loss, batch normalization, energy-accuracy trade-off, energy modeling, ResNet50, ImageNet classification, energy efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.