Deep Learning with Limited Numerical Precision
Open accessAbstract
Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training
Murray, A. F., & Edwards, P. J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.317730 · Cited by 201 · see citing works
Flagged first, then by citations
Research works printed but not linked: 24
- Audhkhasi, Kartik, Osoba, Osonde, and Kosko, Bart. Noise benefits in backpropagation and deep bidirectional pre-training. In Neural Networks (IJCNN), The 2013 International Joint Conference on , pp. 18. IEEE, 2013.
- Baboulin, Marc, Buttari, Alfredo, Dongarra, Jack, Kurzak, Jakub, Langou, Julie, Langou, Julien, Luszczek, Piotr, and Tomov, Stanimire. Accelerating scientific computations with mixed precision algorithms. Computer Physics Communications , 180(12):2526-2533, 2009.
- Bishop, Chris M. Training with noise is equivalent to tikhonov regularization. Neural computation , 7(1): 108-116, 1995.
- Bottou, L´ eon and Bousquet, Olivier. The tradeoffs of large scale learning. In NIPS , volume 4, pp. 2, 2007.
- Chen, Yunji, Luo, Tao, Liu, Shaoli, Zhang, Shijin, He, Liqiang, Wang, Jia, Li, Ling, Chen, Tianshi, Xu, Zhiwei, Sun, Ninghui, et al. Dadiannao: A machinelearning supercomputer. In Microarchitecture (MICRO), 2014 47th Annual IEEE/ACM International Symposium on , pp. 609-622. IEEE, 2014.
- Chilimbi, Trishul, Suzue, Yutaka, Apacible, Johnson, and Kalyanaraman, Karthik. Project adam: Building an efficient and scalable deep learning training system. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14) , pp. 571-582, Broomfield, CO, October 2014.
- Coates, Adam, Huval, Brody, Wang, Tao, Wu, David, Catanzaro, Bryan, and Andrew, Ng. Deep learning with cots hpc systems. In Proceedings of The 30th International Conference on Machine Learning , pp. 1337-1345, 2013.
- Courbariaux, Matthieu, Bengio, Yoshua, and David, Jean-Pierre. Low precision arithmetic for deep learning. arXiv preprint arXiv:1412.7024 , 2014. · arXiv
- Dean, Jeffrey, Corrado, Greg, Monga, Rajat, Chen, Kai, Devin, Matthieu, Mao, Mark, Senior, Andrew, Tucker, Paul, Yang, Ke, Le, Quoc V, et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems , pp. 12231231, 2012.
- Farabet, Cl´ ement, Martini, Berin, Corda, Benoit, Akselrod, Polina, Culurciello, Eugenio, and LeCun, Yann. Neuflow: A runtime reconfigurable dataflow processor for vision. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on , pp. 109116. IEEE, 2011.
- Gokhale, Vinayak, Jin, Jonghoon, Dundar, Aysegul, Martini, Berin, and Culurciello, Eugenio. A 240 gops/s mobile coprocessor for deep neural networks. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on , pp. 696-701. IEEE, 2014.
- Hammerstrom, Dan. A vlsi architecture for highperformance, low-cost, on-chip learning. In Neural Networks, 1990., 1990 IJCNN International Joint Conference on , pp. 537-544. IEEE, 1990.
- Hinton, Geoffrey E, Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 , 2012. · arXiv
- H¨ ohfeld, Markus and Fahlman, Scott E. Probabilistic rounding in neural network learning with limited precision. Neurocomputing , 4(6):291-299, 1992.
- Holt, JL and Hwang, Jenq-Neng. Finite precision error analysis of neural network hardware implementations. Computers, IEEE Transactions on , 42(3): 281-290, 1993.
- Iwata, Akira, Yoshida, Yukio, Matsuda, Satoshi, Sato, Yukimasa, and Suzumura, Nobuo. An artificial neural network accelerator using general purpose 24 bit floating point digital signal processors. In Neural Networks, 1989. IJCNN., International Joint Conference on , pp. 171-175. IEEE, 1989.
- Kim, Jonghong, Hwang, Kyuyeon, and Sung, Wonyong. X1000 real-time phoneme recognition vlsi using feed-forward deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on , pp. 7510-7514. IEEE, 2014.
- Krizhevsky, Alex and Hinton, Geoffrey. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep , 1(4):7, 2009.
- Kung, H.T. Why systolic architectures? Computer , 15(1):37-46, Jan 1982. doi: 10.1109/MC.1982. 1653825. · DOI
- LeCun, Yann, Bottou, L´ eon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.
- Merolla, Paul A, Arthur, John V, Alvarez-Icaza, Rodrigo, Cassidy, Andrew S, Sawada, Jun, Akopyan, Filipp, Jackson, Bryan L, Imam, Nabil, Guo, Chen, Nakamura, Yutaka, et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science , 345(6197):668-673, 2014.
- Recht, Benjamin, Re, Christopher, Wright, Stephen, and Niu, Feng. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems , pp. 693-701, 2011.
- Vanhoucke, Vincent, Senior, Andrew, and Mao, Mark Z. Improving the speed of neural networks on cpus. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop , 2011.
- Wu, Ren, Yan, Shengen, Shan, Yi, Dang, Qingqing, and Sun, Gang. Deep image: Scaling up image recognition. arXiv preprint arXiv:1501.02876 , 2015. · arXiv
Cited by
Sorted by citations
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Efficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · see citing works
“Other examples of output stationary are found in [88, 90].”
InfluentialOpportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310 · see citing works
Memory devices and applications for in-memory computing
Sebastian, A., Le Gallo, M., Khaddam-Aljameh, R., & Eleftheriou, E. (2020) · Nature Nanotechnology · doi:10.1038/s41565-020-0655-z · Cited by 2,289 · see citing works
Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks
Chen, Y.-H., Emer, J., & Sze, V. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001177 · Cited by 1,182 · see citing works
Neuro-Inspired Computing With Emerging Nonvolatile Memorys
Yu, S. (2018) · Proceedings of the IEEE · doi:10.1109/jproc.2018.2790840 · Cited by 1,170 · see citing works
Equivalent-accuracy accelerated neural-network training using analogue memory
Ambrogio, S., Narayanan, P., Tsai, H., Shelby, R. M., Boybat, I., di Nolfo, C., Sidler, S., Giordano, M., Bodini, M., Farinha, N. C. P., et al. (2018) · Nature · doi:10.1038/s41586-018-0180-5 · Cited by 1,151 · see citing works
Self-citationA Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219 · see citing works
“Quantized Training: Perhaps the most important use of quantization has been to accelerate NN training with half-precision [41, 72, 78, 171].”
InfluentialPre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994 · see citing works
Efficient and self-adaptive in-situ learning in multilayer memristor neural networks
Li, C., Belkin, D., Li, Y., Yan, P., Hu, M., Ge, N., Jiang, H., Montgomery, E., Lin, P., Wang, Z., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-04484-2 · Cited by 931 · see citing works
A Survey of Model Compression and Acceleration for Deep Neural Networks
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2017) · arXiv · doi:10.48550/arxiv.1710.09282 · Cited by 857 · see citing works
ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars
Shafiee, A., Nag, A., Muralimanohar, N., Balasubramonian, R., Strachan, J. P., Hu, M., Williams, R. S., & Srikumar, V. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.12 · Cited by 775 · see citing works
Fixed Point Quantization of Deep Convolutional Networks
Lin, D. D., Talathi, S. S., & Annapureddy, V. S. (2015) · arXiv · doi:10.48550/arxiv.1511.06393 · Cited by 684 · see citing works
Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
Mohammadi Amiri, M., & Gunduz, D. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.2981904 · Cited by 548 · see citing works
Gokmen, T., & Vlasov, Y. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00333 · Cited by 491 · see citing works
Communication-efficient edge ai: algorithms and systems
Shi, Y., Yang, K., Jiang, T., Zhang, J., & Letaief, K. B. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.3007787 · Cited by 488 · see citing works
In-Memory Computing: Advances and Prospects
Verma, N., Jia, H., Valavi, H., Tang, Y., Ozatay, M., Chen, L.-Y., Zhang, B., & Deaville, P. (2019) · IEEE Solid-State Circuits Magazine · doi:10.1109/mssc.2019.2922889 · Cited by 473 · see citing works
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436 · see citing works
PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference
Ankit, A., Hajj, I. E., Chalamalasetti, S. R., Ndu, G., Foltin, M., Williams, R. S., Faraboschi, P., Hwu, W.-M. W., Strachan, J. P., Roy, K., et al. (2019) · Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3297858.3304049 · Cited by 424 · see citing works
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · see citing works
Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
Ioannou, Y., Robertson, D., Cipolla, R., & Criminisi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.633 · Cited by 321 · see citing works
Bai, X., Wang, X., Liu, X., Liu, Q., Song, J., Sebe, N., & Kim, B. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108102 · Cited by 313 · see citing works
A Reconfigurable Streaming Deep Convolutional Neural Network Accelerator for Internet of Things
Du, L., Du, Y., Li, Y., Su, J., Kuan, Y.-C., Liu, C.-C., & Chang, M.-C. F. (2018) · IEEE Transactions on Circuits and Systems I: Regular Papers · doi:10.1109/tcsi.2017.2735490 · Cited by 205 · see citing works
Design and Analysis of Uplink and Downlink Communications for Federated Learning
Zheng, S., Shen, C., & Chen, X. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3041388 · Cited by 191 · see citing works
Filter-based deep-compression with global average pooling for convolutional networks
Hsiao, T.-Y., Chang, Y.-C., Chou, H.-H., & Chiu, C.-T. (2019) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2019.02.008 · Cited by 190 · see citing works
A Study of BFLOAT16 for Deep Learning Training
Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D. T., Jammalamadaka, N., Huang, J., Yuen, H., et al. (2019) · arXiv · doi:10.48550/arxiv.1905.12322 · Cited by 169 · see citing works
“Numerous studies have shown that 16-bits of precision is sufficient for training deep neural networks[16],[26],[21],[8].”
BackgroundBringing ai to edge: from deep learning’s perspective
Liu, D., Kong, H., Luo, X., Liu, W., & Subramaniam, R. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.04.141 · Cited by 171 · see citing works
“Some early work has shown that using FP16 to train DNN models can reduce the computational cost while retaining accuracy [109].”
BackgroundBinary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
Zhu, S., Dong, X., & Su, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00506 · Cited by 153 · see citing works
Mixed-Precision Deep Learning Based on Computational Memory
Nandakumar, S. R., Le Gallo, M., Piveteau, C., Joshi, V., Mariani, G., Boybat, I., Karunaratne, G., Khaddam-Aljameh, R., Egger, U., Petropoulos, A., et al. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00406 · Cited by 120 · see citing works
Characterizing the deployment of deep neural networks on commercial edge devices
Hadidi, R., Cao, J., Xie, Y., Asgari, B., Krishna, T., & Kim, H. (2019) · 2019 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc47752.2019.9041955 · Cited by 118 · see citing works
DeepShift: Towards Multiplication-Less Neural Networks
Elhoushi, M., Chen, Z., Shafiq, F., Tian, Y. H., & Li, J. Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00268 · Cited by 117 · see citing works
DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
Wiedemann, S., Kirchhoffer, H., Matlage, S., Haase, P., Marban, A., Marinc, T., Neumann, D., Nguyen, T., Schwarz, H., Wiegand, T., et al. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2969554 · Cited by 113 · see citing works
A Review of Privacy-Preserving Federated Learning for the Internet-of-Things
Briggs, C., Fan, Z., & Andras, P. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-70604-3_2 · Cited by 101 · see citing works
A Compact Convolutional Neural Network for Surface Defect Inspection
Huang, Y., Qiu, C., Wang, X., Wang, S., & Yuan, K. (2020) · Sensors · doi:10.3390/s20071974 · Cited by 85 · see citing works
Deep Convolutional Neural Network Inference with Floating-point Weights and Fixed-point Activations
Lai, L., Suda, N., & Chandra, V. (2017) · arXiv · doi:10.48550/arxiv.1703.03073 · Cited by 81 · see citing works
FPGA-Accelerated Dense Linear Machine Learning: A Precision-Convergence Trade-Off
Kara, K., Alistarh, D., Alonso, G., Mutlu, O., & Zhang, C. (2017) · 2017 IEEE 25th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm.2017.39 · Cited by 77 · see citing works
Mathur, A., Zhang, T., Bhattacharya, S., Velickovic, P., Joffe, L., Lane, N. D., Kawsar, F., & Lio, P. (2018) · 2018 17th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn.2018.00048 · Cited by 70 · see citing works
The cascading neural network: building the Internet of Smart Things
Leroux, S., Bohez, S., De Coninck, E., Verbelen, T., Vankeirsbilck, B., Simoens, P., & Dhoedt, B. (2017) · Knowledge and Information Systems · doi:10.1007/s10115-017-1029-1 · Cited by 59 · see citing works
Towards Energy-Efficient and Secure Edge AI: A Cross-Layer Framework ICCAD Special Session Paper
Shafique, M., Marchisio, A., Wicaksana Putra, R. V., & Hanif, M. A. (2021) · 2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD) · doi:10.1109/iccad51958.2021.9643539 · Cited by 58 · see citing works
Make RepVGG Greater Again: A Quantization-Aware Approach
Chu, X., Li, L., & Zhang, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29045 · Cited by 53 · see citing works
Bit Efficient Quantization for Deep Neural Networks
Nayak, P., Zhang, D., & Chai, S. (2019) · 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS) · doi:10.1109/emc2-nips53020.2019.00020 · Cited by 51 · see citing works
IF-CNN: Image-Aware Inference Framework for CNN With the Collaboration of Mobile Devices and Cloud
Shu, G., Liu, W., Zheng, X., & Li, J. (2018) · IEEE Access · doi:10.1109/access.2018.2880196 · Cited by 30 · see citing works
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., et al. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3476008 · Cited by 32 · see citing works
Approximate Computing Methods for Embedded Machine Learning
Ibrahim, A., Osta, M., Alameh, M., Saleh, M., Chible, H., & Valle, M. (2018) · 2018 25th IEEE International Conference on Electronics, Circuits and Systems (ICECS) · doi:10.1109/icecs.2018.8617877 · Cited by 31 · see citing works
Boosting Convolutional Neural Networks Performance Based on FPGA Accelerator
Al-Shamma, O., Fadhel, M. A., Hameed, R. A., Alzubaidi, L., & Zhang, J. (2020) · Advances in Intelligent Systems and Computing · doi:10.1007/978-3-030-16657-1_47 · Cited by 27 · see citing works
Framework for In-Memory Computing Based on Memristor and Memcapacitor for On-Chip Training
Singh, A., & Lee, B.-G. (2023) · IEEE Access · doi:10.1109/access.2023.3324375 · Cited by 19 · see citing works
“While DNN do exhibit some level of tolerance to calculation imprecision [12], [13], [14] it is vital to acknowledge the limitations of this resilience.”
BackgroundFLightNNs: Lightweight Quantized Deep Neural Networks for Fast and Accurate Inference
Ding, R., Liu, Z., Chin, T.-W., Marculescu, D., & Blanton, R. D. (. (2019) · Proceedings of the 56th Annual Design Automation Conference 2019 · doi:10.1145/3316781.3317828 · Cited by 19 · see citing works
- Authors
- Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P.
- Affiliations
- IBM (United States) (US)
- IBM Research - Almaden (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1502.02551
- OpenAlex
- W1841592590
- Cited by
- 1,364
- References
- 1
- Citations by year
- 2025: 39 · 2024: 98 · 2023: 122 · 2022: 161 · 2021: 184 · 2020: 184 · 2019: 183 · 2018: 200 · 2017: 109 · 2016: 66 · 2015: 16 · 2014: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- low-precision training, fixed-point arithmetic, stochastic rounding, neural network training, rounding schemes, classification accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
