CoChat
Sign Up
PreprintCited by 410Numerical Methods and AlgorithmsAdvanced Neural Network ApplicationsLow-power high-performance VLSI design

Training deep neural networks with low precision multiplications

Open access
Courbariaux, Matthieu, Bengio, Yoshua, David, Jean-Pierre·2014·arXiv
DOI 10.48550/arxiv.1412.7024arXiv 1412.7024
Open this paper in CoChat — free accountOpen PDF

Abstract

Multipliers are the most space and power-hungry arithmetic operators of the digital implementation of deep neural networks. We train a set of state-of-the-art neural networks (Maxout networks) on three benchmark datasets: MNIST, CIFAR-10 and SVHN. They are trained with three distinct formats: floating point, fixed point and dynamic fixed point. For each of those datasets and for each of those formats, we assess the impact of the precision of the multiplications on the final error after training. We find that very low precision is sufficient not just for running trained networks but also for training them. For example, it is possible to train Maxout networks with 10 bits multiplications.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1412.7024

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Floating-point backward passes are important because quantized gradient accumulation can produce zero or high-error gradients, especially at low precision.

    “Performing the backward pass with floating point is important, as accumulating the gradients in quantized precision can result in zerogradient or gradients that have high error, especially in low-precision [42,80,81,107,159,186,204,231].”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [42]

    Section 6 (p. 4) says parameter updates need higher precision to accumulate small changes, and Section 2 (p. 2) describes using high-precision accumulators. But Section 6 explicitly uses lower precision for backward propagation than for parameter updates. The cited passages do not report zero or high-error gradients from quantized accumulation or establish that the backward pass must use floating point. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • BinaryConnect constrains weights to either +1 or -1.

    “An important work here is BinaryConnect [42] which constrains the weights to either +1 or -1.”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [42]

    The Abstract (p. 1) describes this paper's use of floating-point, fixed-point, and dynamic fixed-point formats, including 10-bit multiplications. The Related Works section (p. 8) does not mention BinaryConnect. Searches for BinaryConnect and weights restricted to +1 or -1 found no passage in our copy addressing the claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 13

  1. Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Bergeron, A., Bouchard, N., and Bengio, Y. (2012). Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.
  2. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., WardeFarley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy) . Oral Presentation.
  3. Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., and Temam, O. (2014a). Diannao: A smallfootprint high-throughput accelerator for ubiquitous machine-learning. In Proceedings of the 19th international conference on Architectural support for programming languages and operating systems , pages 269-284. ACM.
  4. Farabet, C., Martini, B., Corda, B., Akselrod, P., Culurciello, E., and LeCun, Y. (2011). Neuflow: A runtime reconfigurable dataflow processor for vision. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on , pages 109-116. IEEE.
  5. Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep sparse rectifier neural networks. In AISTATS'2011 .
  6. Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013a). Maxout networks. Technical Report Arxiv report 1302.4389, Universit´ e de Montr´ eal.
  7. Goodfellow, I. J., Warde-Farley, D., Lamblin, P., Dumoulin, V., Mirza, M., Pascanu, R., Bergstra, J., Bastien, F., and Bengio, Y. (2013b). Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214 . · arXiv
  8. Holt, J. L. and Baker, T. E. (1991). Back propagation simulations using limited precision calculations. In Neural Networks, 1991., IJCNN-91-Seattle International Joint Conference on , volume 2, pages 121-126. IEEE.
  9. Krizhevsky, A., Sutskever, I., and Hinton, G. (2012a). ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS'2012) .
  10. Krizhevsky, A., Sutskever, I., and Hinton, G. (2012b). ImageNet classification with deep convolutional neural networks. In NIPS'2012 .
  11. Nair, V. and Hinton, G. (2010). Rectified linear units improve restricted Boltzmann machines. In ICML'2010 .
  12. Pham, P.-H., Jelaca, D., Farabet, C., Martini, B., LeCun, Y., and Culurciello, E. (2012). Neuflow: dataflow vision processing system-on-a-chip. In Circuits and Systems (MWSCAS), 2012 IEEE 55th International Midwest Symposium on , pages 1044-1047. IEEE.
  13. Williamson, D. (1991//). Dynamically scaled fixed point arithmetic. pages 315 - 18, New York, NY, USA. dynamic scaling;iteration stages;digital filters;overflow probability;fixed point arithmetic;fixed-point filter;.

Cited by

Sorted by citations

Related

Authors
Courbariaux, M., Bengio, Y., & David, J.-P.
Affiliations
  • Polytechnique Montréal (CA)
  • Université de Montréal (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2014
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1412.7024
arXiv
1412.7024
OpenAlex
W1825672851
Cited by
407
References
18
Citations by year
2025: 12 · 2024: 17 · 2023: 23 · 2022: 30 · 2021: 65 · 2020: 64 · 2019: 80 · 2018: 62 · 2017: 36 · 2016: 15 · 2015: 2 · 2014: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • PolyPublie (École Polytechnique de Montréal) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Canadian Institute for Advanced Research
  • Compute Canada
  • Canada Research Chairs
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
Keywords
low-precision arithmetic, deep neural network training, fixed-point arithmetic, floating-point precision, energy-efficient neural networks, MNIST, CIFAR-10, SVHN

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review