Training and Inference with Integers in Deep Neural Networks
Open accessThe DOI matches its OpenAlex record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Researches on deep neural networks with discrete parameters and their deployment in embedded systems have been active and promising topics. Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously. In this work, we develop a new method termed as "WAGE" to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers. To perform pure discrete dataflow for fixed-point devices, we further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation. Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization. Empirically, we demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, another framework (known as DarkFPGA) makes assessments using the Maxeler MAX5 platform that is built up with Xilinx Ultrascale+ VU9P FPGA.”
From Embedded Intelligence on FPGA: Survey, Applications and Challenges (2021) · cites this paper as [79]
In the citing paper
“The key features for the framework consist of a scalable accelerator architecture, a software definable hardware accelerator and an optimization hardware design tool.”
From Embedded Intelligence on FPGA: Survey, Applications and Challenges (2021) · cites this paper as [79]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408
Efficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215
Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks
Chen, Y.-H., Krishna, T., Emer, J. S., & Sze, V. (2017) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2016.2616357 · Cited by 3,247
Cited by
Towards artificial general intelligence with hybrid Tianjic chip architecture
Pei, J., Deng, L., Song, S., Zhao, M., Zhang, Y., Wu, S., Wang, G., Zou, Z., Wu, Z., He, W., et al. (2019) · Nature · doi:10.1038/s41586-019-1424-8 · Cited by 1,082
Self-citationModel Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
Deng, B. L., Li, G., Han, S., Shi, L., & Xie, Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2976475 · Cited by 996
“In a nutshell, the heuristic methods are able to achieve sufficiently good accuracy on small or medium data sets such as MNIST, SVHN, and CIFAR10/CIFAR100 [150], [151], [168], [182].”
BackgroundMethodInfluentialSelf-citationPruning and quantization for deep neural network acceleration: A survey
Liang, T., Glossner, J., Wang, L., Shi, S., & Zhang, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.045 · Cited by 943
A Survey on Federated Learning for Resource-Constrained IoT Devices
Imteaj, A., Thakker, U., Wang, S., Li, J., & Amini, M. H. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3095077 · Cited by 774
“Wu et al. [106] proposed an approach to reduce the cost of training and inference by using lower bitwidth integers for both stages of the application.”
Compute-in-Memory Chips for Deep Learning: Recent Trends and Prospects
Yu, S., Jiang, H., Huang, S., Peng, X., & Lu, A. (2021) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2021.3092533 · Cited by 391
Yang, J., Shen, X., Xing, J., Tian, X., Li, H., Deng, B., Huang, J., & Hua, X.-S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00748 · Cited by 387
Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss
Jung, S., Son, C., Lee, S., Son, J., Han, J.-J., Kwak, Y., Hwang, S. J., & Choi, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00448 · Cited by 385
Peng, X., Huang, S., Jiang, H., Lu, A., & Yu, S. (2021) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2020.3043731 · Cited by 317
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
Tang, R., Lu, Y., Liu, L., Mou, L., Olga Vechtomova, A., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.12136 · Cited by 297
TASO: optimizing deep learning computation with automatic generation of graph substitutions
Jia, Z., Padon, O., Thomas, J., Warszawski, T., Zaharia, M., & Aiken, A. (2019) · Proceedings of the 27th ACM Symposium on Operating Systems Principles · doi:10.1145/3341301.3359630 · Cited by 261
Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
Zhao, R., Hu, Y., Dotzel, J., De Sa, C., & Zhang, Z. (2019) · arXiv · doi:10.48550/arxiv.1901.09504 · Cited by 231
Fully Quantized Network for Object Detection
Li, R., Wang, Y., Liang, F., Qin, H., Yan, J., & Fan, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00292 · Cited by 214
“The first is that critical operations such as batch normalization are either not handled [35] or ablated [29] during training, leading to a mismatch between training and evaluation behaviors, or causing training convergence difficulties.”
BackgroundScalable Methods for 8-bit Training of Neural Networks
Banner, R., Hubara, I., Hoffer, E., & Soudry, D. (2018) · arXiv · doi:10.48550/arxiv.1805.11046 · Cited by 193
Sun, X., & Yu, S. (2019) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2019.2933148 · Cited by 170
“To analyze the impact of weight precision on the training accuracy, we adopt a methodology inspired from [24] to quantize the weight and the gradient during training, where the continuous values are quantized to k-bit with the uniform step θ :”
Method$L1$ -Norm Batch Normalization for Efficient Training of Deep Neural Networks
Wu, S., Li, G., Deng, L., Liu, L., Wu, D., Xie, Y., & Shi, L. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876179 · Cited by 166
Self-citationA computing-in-memory macro based on three-dimensional resistive random-access memory
Huo, Q., Yang, Y., Wang, Y., Lei, D., Fu, X., Ren, Q., Xu, X., Luo, Q., Xing, G., Chen, C., et al. (2022) · Nature Electronics · doi:10.1038/s41928-022-00795-x · Cited by 159
Mixed-Precision Deep Learning Based on Computational Memory
Nandakumar, S. R., Le Gallo, M., Piveteau, C., Joshi, V., Mariani, G., Boybat, I., Karunaratne, G., Khaddam-Aljameh, R., Egger, U., Petropoulos, A., et al. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00406 · Cited by 120
Post-training 4-bit quantization of convolution networks for rapid-deployment
Banner, R., Nahshan, Y., Hoffer, E., & Soudry, D. (2018) · arXiv · doi:10.48550/arxiv.1810.05723 · Cited by 115
“Finally, Wu et al. (2018) has proposed a heuristic approach to adjust the clipping values according to the desired precision.”
BackgroundMethodEmbedded Intelligence on FPGA: Survey, Applications and Challenges
Seng, K. P., Lee, P. J., & Ang, L. M. (2021) · Electronics · doi:10.3390/electronics10080895 · Cited by 105
Compiling KB-sized machine learning models to tiny IoT devices
Gopinath, S., Ghanathe, N., Seshadri, V., & Sharma, R. (2019) · Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/3314221.3314597 · Cited by 92
Federated Learning for Resource-Constrained IoT Devices: Panoramas and State of the Art
Imteaj, A., Mamun Ahmed, K., Thakker, U., Wang, S., Li, J., & Amini, M. H. (2023) · Adaptation, Learning, and Optimization · doi:10.1007/978-3-031-11748-0_2 · Cited by 86
Li, W., Sun, X., Huang, S., Jiang, H., & Yu, S. (2022) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2022.3163197 · Cited by 77
Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
McKinstry, J. L., Esser, S. K., Appuswamy, R., Bablani, D., Arthur, J. V., Yildiz, I. B., & Modha, D. S. (2018) · arXiv · doi:10.48550/arxiv.1809.04191 · Cited by 74
“Recent attempts (Wu et al., 2018) at training low-precision networks with integer constraints have hinted at the possibility of porting such networks to commercially available hardware for inference3.”
Mixed Precision Training With 8-bit Floating Point
Mellempudi, N., Srinivasan, S., Das, D., & Kaul, B. (2019) · arXiv · doi:10.48550/arxiv.1905.12334 · Cited by 38
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
Cambier, L., Bhiwandiwalla, A., Gong, T., Nekuii, M., Elibol, O. H., & Tang, H. (2020) · arXiv · doi:10.48550/arxiv.2001.05674 · Cited by 23
“Wu et al. (2018) demonstrate using integers for training LeNet-5 (LeCun et al., 1998) and AlexNet with 8-bits for activations, error and gradients and 2-bits for weights.”
MethodNeural gradients are near-lognormal: improved quantized and sparse training
Chmiel, B., Ben-Uri, L., Shkolnik, M., Hoffer, E., Banner, R., & Soudry, D. (2020) · arXiv · doi:10.48550/arxiv.2006.08173 · Cited by 17
“, 2019), there were also major advances in quantization during training, many of them faced difficulty trying to represent the high dynamic range of the gradients (Banner et al., 2018a; Wu et al., 2018).”
BackgroundOflamaz, C. U., & Yalçın, M. E. (2025) · Electronics · doi:10.3390/electronics14183686 · Cited by 4
Metadata
- Authors
- Wu, S., Li, G., Chen, F., & Shi, L.
- Venue
- arXiv (Cornell University)
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1802.04680
- OpenAlex
- W2787513570
- Cited by
- 306
- References
- 23
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.