FPGA-Accelerated Dense Linear Machine Learning: A Precision-Convergence Trade-Off
Abstract
Stochastic gradient descent (SGD) is a commonly used algorithm for training linear machine learning models. Based on vector algebra, it benefits from the inherent parallelism available in an FPGA. In this paper, we first present a single-precision floating-point SGD implementation on an FPGA that provides similar performance as a 10-core CPU. We then adapt the design to make it capable of processing low-precision data. The low-precision data is obtained from a novel compression scheme-called stochastic quantization, specifically designed for machine learning applications. We test both full-precision and low-precision designs on various regression and classification data sets. We achieve up to an order of magnitude training speedup when using low-precision data compared to a full-precision SGD on the same FPGA and a state-of-the-art multi-core solution, while maintaining the quality of training. We open source the designs presented in this paper.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A comparison of SGD with quantization techniques in FPGA (using single-precision) has similar performance compared to a 10-core CPU.
“The authors in [8] gave a comparison of the SGD with quantization techniques in FPGA (in their approach using single-precision) having similar performance compared to 10-core CPU.”
From Embedded Intelligence on FPGA: Survey, Applications and Challenges (2021) · cites this paper as [8]
Cites
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795 · see citing works
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · see citing works
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211 · see citing works
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · Cited by 707 · see citing works
Training deep neural networks with low precision multiplications
Courbariaux, M., Bengio, Y., & David, J.-P. (2014) · arXiv · doi:10.48550/arxiv.1412.7024 · Cited by 407 · see citing works
TABLA: A unified template-based framework for accelerating statistical machine learning
Mahajan, D., Park, J., Amaro, E., Sharma, H., Yazdanbakhsh, A., Kim, J. K., & Esmaeilzadeh, H. (2016) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2016.7446050 · Cited by 156 · see citing works
Large-Scale FPGA-Based Convolutional Networks
Farabet, C., LeCun, Y., Kavukcuoglu, K., Martini, B., Akselrod, P., Talay, S., & Culurciello, E. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9781139042918.020 · Cited by 121 · see citing works
Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
De, C., Zhang, C., Olukotun, K., & Ré, C. (2015) · PubMed · doi:10.48550/arxiv.1506.06438 · Cited by 107 · see citing works
Self-citationEmbedded Hardware-Efficient Real-Time Classification With Cascade Support Vector Machines
Kyrkou, C., Bouganis, C.-S., Theocharides, T., & Polycarpou, M. M. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2428738 · Cited by 82 · see citing works
A Massively Parallel FPGA-Based Coprocessor for Support Vector Machines
Cadambi, S., Đurđanović, I., Jakkula, V., Sankaradass, M., Cosatto, E., Chakradhar, S., & Graf, H. P. (2009) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm.2009.34 · Cited by 75 · see citing works
QSGD: Randomized Quantization for Communication-Optimal Stochastic Gradient DescentPreprint
Alistarh, D., Li, J. Z., Tomioka, R., & Vojnović, M. (2016) · arXiv (Cornell University) · Cited by 64 · see citing works
Self-citationFPGA Simulation of Linear and Nonlinear Support Vector Machine
Mahmoodi, D., Soleimani, A., Khosravi, H., et al. (2011) · Journal of Software Engineering and Applications · doi:10.4236/jsea.2011.45036 · Cited by 58 · see citing works
A High Throughput FPGA-Based Floating Point Conjugate Gradient Implementation for Dense Matrices
Roldao, A., & Constantinides, G. A. (2010) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/1661438.1661439 · Cited by 52 · see citing works
A Reconfigurable Computing System Based on a Cache-Coherent Fabric
Oliver, N., Sharma, R. R., Chang, S., et al. (2011) · International Conference on ReConFigurable Computing and FPGAs (ReConFig) · doi:10.1109/reconfig.2011.4 · Cited by 50 · see citing works
Pipelined compressor tree optimization using integer linear programming
Kumm, M., & Zipf, P. (2014) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl.2014.6927468 · Cited by 48 · see citing works
A Massively Parallel, Energy Efficient Programmable Accelerator for Learning and Classification
Majumdar, A., Cadambi, S., Becchi, M., et al. (2012) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/2133382.2133388 · Cited by 44 · see citing works
Effects of Reduced Precision on Floating-Point SVM Classification Accuracy
Lesser, B., Mücke, M., & Gansterer, W. N. (2011) · Procedia Computer Science · doi:10.1016/j.procs.2011.04.053 · Cited by 38 · see citing works
Asynchronous Parallel Greedy Coordinate Descent
You, Y., Lian, X., Liu, J., et al. (2016) · neural information processing systems · Cited by 25 · see citing works
Hardware acceleration of iterative image reconstruction for X-ray computed tomography
Kim, J. K., Zhang, Z., & Fessler, J. A. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5946827 · Cited by 10 · see citing works
ZipML: An End-to-end Bitwise Framework for Dense Generalized Linear Models.Preprint
Zhang, H., Kara, K., Li, J., et al. (2016) · arXiv (Cornell University) · Cited by 10 · see citing works
Self-citationMultilayer perceptron network with integrated training algorithm in FPGA
Perez-Garcia, A. N., Xavier, G. M. T., Flores-Nava, L. M., et al. (2014) · International Conference on Electrical Engineering, Computing Science and Automatic Control (CCE) · doi:10.1109/iceee.2014.6978300 · Cited by 8 · see citing works
A hardware acceleration technique for gradient descent and conjugate gradient
Kesler, D., Deka, B., & Kumar, R. (2011) · doi:10.1109/sasp.2011.5941086 · Cited by 8 · see citing works
Peak Performance Model for a Custom Precision Floating-Point Dot Product on FPGAs
Mücke, M., Lesser, B., & Gansterer, W. N. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21878-1_49 · Cited by 7 · see citing works
FPGASVM: A Framework for Accelerating Kernelized Support Vector Machine
Rabieah, B., span, M., & Bouganis, C.-S. (2016) · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Embedded Intelligence on FPGA: Survey, Applications and Challenges
Seng, K. P., Lee, P. J., & Ang, L. M. (2021) · Electronics · doi:10.3390/electronics10080895 · Cited by 105 · see citing works
“The authors in [8] gave a comparison of the SGD with quantization techniques in FPGA (in their approach using single-precision) having similar performance compared to 10-core CPU.”
ResultParallel Implementation on FPGA of Support Vector Machines Using Stochastic Gradient Descent
Lopes, F. F., Ferreira, J. C., & Fernandes, M. A. C. (2019) · Electronics · doi:10.3390/electronics8060631 · Cited by 29 · see citing works
- Authors
- Kara, K., Alistarh, D., Alonso, G., Mutlu, O., & Zhang, C.
- Affiliations
- ETH Zurich (CH)
- Institute of Science and Technology Austria (AT)
- Venue
- 2017 IEEE 25th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)
- Publisher
- IEEE
- Published
- 2017-04
- Pages
- 160-167
- OpenAlex
- W2732291649
- Cited by
- 77
- References
- 24
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 4 · 2023: 5 · 2022: 6 · 2021: 12 · 2020: 13 · 2019: 13 · 2018: 13 · 2017: 4 · 2016: 1
- FWCI
- 8.29
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- FPGA acceleration, stochastic gradient descent, low-precision arithmetic, stochastic quantization, floating-point arithmetic, training acceleration, regression analysis, classification, hardware design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
