Student or educator? Sign up with your school email and get 6 months of Pro free.
PreprintCited by 62Embedded Systems Design TechniquesAdvanced Neural Network ApplicationsEmbedded Systems and FPGA Design

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices

Open access

Abstract

Accessible machine learning algorithms, software, and diagnostic tools for energy-efficient devices and systems are extremely valuable across a broad range of application domains. In scientific domains, real-time near-sensor processing can drastically improve experimental design and accelerate scientific discoveries. To support domain scientists, we have developed hls4ml, an open-source software-hardware codesign workflow to interpret and translate machine learning algorithms for implementation with both FPGA and ASIC technologies. We expand on previous hls4ml work by extending capabilities and techniques towards low-power implementations and increased usability: new Python APIs, quantization-aware pruning, end-to-end FPGA workflows, long pipeline kernels for low power, and new device backends include an ASIC workflow. Taken together, these and continued efforts in hls4ml will arm a new generation of domain scientists with accessible, efficient, and powerful tools for machine-learning-accelerated discovery.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2103.05579

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 70

  1. [1][1] C. R. Banbury, et al. 2020. Benchmarking TinyML Systems: Challenges and Direction. (2020). arXiv:2003.04821 - · arXiv
  2. [2][2] S. Zhang, et al. 2020. NeckSense: A multi-sensor necklace for detecting eating activities in free-living conditions. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4 (2020), 1. https://doi.org/10.1145/3397313 arXiv:1911.07179 - · DOI
  3. [3][3] A. R. Elias, et al. 2017. Where's the bear? - Automating wildlife image processing using IoT and edge cloud systems. In 2017 IEEE/ACM Second International Conference on Internet-of-Things Design and Implementation (IoTDI) . IEEE, 247. -
  4. [4][4] D. Vasisht, et al. 2017. Farmbeats: An IoT platform for data-driven agriculture. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17) . 515. -
  5. [5][5] J. Duarte et al. 2018. Fast inference of deep neural networks in FPGAs for particle physics. J. Instrum. 13 (2018), P07027. https://doi.org/10.1088/17480221/13/07/P07027 arXiv:1804.06913 - · DOI
  6. [7][7] R. Nane, et al. 2016. A survey and evaluation of FPGA high-level synthesis tools. IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 35, 10 (2016), 1591. https://doi.org/10.1109/TCAD.2015.2513673 - · DOI
  7. [8][8] S. Lahti, et al. 2019. Are we there yet? A study on the state of high-level synthesis. IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 38 (2019), 898. https://doi. org/10.1109/TCAD.2018.2834439 - · DOI
  8. [9][9] J. Ngadiuba, et al. 2020. Compressing deep neural networks on FPGAs to binary and ternary precision with hls4ml . Mach. Learn.: Sci. Technol. 2 (2020), 015001. https://doi.org/10.1088/2632-2153/aba042 arXiv:2003.06308 - · DOI
  9. [10][10] S. Summers et al. 2020. Fast inference of boosted decision trees in FPGAs for particle physics. J. Instrum. 15 (2020), P05026. https://doi.org/10.1088/17480221/15/05/P05026 arXiv:2002.02534 - · DOI
  10. [11][11] C. N. Coelho, et al. 2020. Automatic deep heterogeneous quantization of deep neural networks for ultra low-area, low-latency inference on the edge at particle colliders. (2020). arXiv:2006.10159 Submitted to Nat. Mach. Intell. - · arXiv
  11. [12][12] T. Aarrestad, et al. 2021. Fast convolutional neural networks on FPGAs with hls4ml . (2021). arXiv:2101.05108 Submitted to Mach. Learn.: Sci. Technol. - · arXiv
  12. [13][13] Y. Iiyama et al. 2021. Distance-weighted graph neural networks on FPGAs for real-time particle reconstruction in high energy physics. Front. Big Data 3 (2021), 44. https://doi.org/10.3389/fdata.2020.598927 arXiv:2008.03601 - · DOI
  13. [14][14] B. Hawks, et al. 2021. Ps and Qs: Quantization-aware pruning for efficient low latency neural network inference. (2021). arXiv:2102.11289 Submitted to Front. AI . - · arXiv
  14. [15][15] I. Kuon et al. 2007. Measuring the gap between FPGAs and ASICs. IEEE Trans. Comput.-Aided Design Integr. Circuits Syst. 26 (2007), 203. https://doi.org/10.1109/ TCAD.2006.884574 -
  15. [16][16] M. Horowitz. 2014. 1.1 Computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) . IEEE, 10. https://doi.org/10.1109/ISSCC.2014.6757323 - · DOI
  16. [17][17] S. I. Venieris, et al. 2018. Toolflows for mapping convolutional neural networks on FPGAs: A survey and future directions. ACM Comput. Surv. 51, Article 56 (2018), 39 pages. https://doi.org/10.1145/3186332 arXiv:1803.05900 - · DOI
  17. [18][18] K. Guo, et al. 2019. A survey of FPGA-based neural network inference accelerators. ACM Trans. Reconfigurable Technol. Syst. 12, Article 2 (2019), 26 pages. https: //doi.org/10.1145/3289185 arXiv:1712.08934 - · DOI · Paper page
  18. [19][19] A. Shawahna, et al. 2019. FPGA-based accelerators of deep learning networks for learning and classification: A review. IEEE Access 7 (2019), 7823. https: //doi.org/10.1109/access.2018.2890150 arXiv:1901.00121 - · DOI
  19. [21][21] S. I. Venieris et al. 2017. fpgaConvNet: A Toolflow for Mapping Diverse Convolutional Neural Networks on Embedded FPGAs. In NIPS 2017 Workshop on Machine Learning on the Phone and other Consumer Devices . arXiv:1711.08740 - · arXiv
  20. [22][22] S. I. Venieris et al. 2017. fpgaConvNet: Automated mapping of convolutional neural networks on FPGAs. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays . ACM, 291. https://doi.org/10. 1145/3020078.3021791 -
  21. [23][23] S. I. Venieris et al. 2017. Latency-driven design for FPGA-based convolutional neural networks. In 2017 27th International Conference on Field Programmable Logic and Applications (FPL) . 1. https://doi.org/10.23919/FPL.2017.8056828 - · DOI
  22. [24][24] S. I. Venieris et al. 2016. fpgaConvNet: A framework for mapping convolutional neural networks on FPGAs. In 2016 IEEE 24th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM) . IEEE, 40. https: //doi.org/10.1109/FCCM.2016.22 - · DOI
  23. [25][25] Y. Jia, et al. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM International Conference on Multimedia . ACM, New York, NY, USA, 675. https://doi.org/10.1145/2647868.2654889 arXiv:1408.5093 - · DOI · Paper page
  24. [26][26] Y. Guan, et al. 2017. FP-DNN: An automated framework for mapping deep neural networks onto FPGAs with RTL-HLS hybrid templates. In 2017 IEEE 25th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM) . 152. https://doi.org/10.1109/FCCM.2017.25 - · DOI
  25. [27][27] M. Abadi, et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. arXiv:1603.04467 https://www.tensorflow.org/ - · arXiv
  26. [28][28] H. Sharma, et al. 2016. From high-level deep neural models to FPGAs. In 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) . IEEE, 1. -
  27. [29][29] R. DiCecco, et al. 2016. Caffeinated FPGAs: FPGA framework for convolutional neural networks. In 2016 International Conference on Field-Programmable Technology (FPT) . IEEE, 265. https://doi.org/10.1109/FPT.2016.7929549 arXiv:1609.09671 - · DOI
  28. [30][30] V. Gokhale, et al. 2017. Snowflake: An efficient hardware accelerator for convolutional neural networks. In 2017 IEEE International Symposium on Circuits and Systems (ISCAS) . 1. https://doi.org/10.1109/ISCAS.2017.8050809 arXiv:1708.02579 - · DOI
  29. [31][31] R. Collobert, et al. 2011. Torch7: A Matlab-like environment for machine learning. In BigLearn, NIPS Workshop . -
  30. [32][32] K. Majumder et al. 2019. A flexible FPGA accelerator for convolutional neural networks. (2019). arXiv:1912.07284 - · arXiv
  31. [33][33] A. Rahman, et al. 2016. Efficient FPGA acceleration of convolutional neural networks using logical-3D compute array. In 2016 Design, Automation Test in Europe Conference Exhibition (DATE) . IEEE, 1393. -
  32. [35][35] M. Blott, et al. 2018. FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks. ACM Trans. Reconfigurable Technol. Syst. 11, 3, Article 16 (2018), 23 pages. https://doi.org/10.1145/3242897 arXiv:1809.04570 - · DOI
  33. [36][36] Y. Umuroglu, et al. 2017. FINN: A framework for fast, scalable binarized neural network inference. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays . ACM, 65. https://doi.org/10.1145/ 3020078.3021744 arXiv:1612.07119 - · arXiv
  34. [40][40] P. N. Whatmough, et al. 2019. FixyNN: Efficient hardware for mobile computer vision via transfer learning. In 2nd SysML Conference . 13. arXiv:1902.11128 https://mlsys.org/Conferences/2019/doc/2019/69.pdf - · arXiv
  35. [42][42] G. B. Hacene, et al. 2020. Quantized guided pruning for efficient hardware implementations of convolutional neural networks. In 2020 18th IEEE International New Circuits and Systems Conference (NEWCAS) . 206. https://doi.org/10.1109/ NEWCAS49341.2020.9159769 arXiv:1812.11337 - · arXiv
  36. [43][43] S.-E. Chang, et al. 2021. Mix and Match: A novel FPGA-centric deep neural network quantization framework. In 27th IEEE International Symposium on HighPerformance Computer Architecture (HPCA) . arXiv:2012.04240 - · arXiv
  37. [44][44] M. W. Numan, et al. 2020. Towards automatic high-level code deployment on reconfigurable platforms: A survey of high-level synthesis tools and toolchains. IEEE Access 8 (2020), 174692. https://doi.org/10.1109/ACCESS.2020.3024098 - · DOI
  38. [45][45] S. Ioffe et al. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In 32nd International Conference on Machine Learning , F. Bach et al. (Eds.), Vol. 37. PMLR, Lille, France, 448. arXiv:1502.03167 http://proceedings.mlr.press/v37/ioffe15.html - · arXiv
  39. [47][47] V. Nair et al. 2010. Rectified linear units improve restricted Boltzmann machines. In 27th International Conference on International Conference on Machine Learning (ICML'10) . Omnipress, Madison, WI, USA, 807. -
  40. [48][48] X. Glorot, et al. 2011. Deep sparse rectifier neural networks. In 14th International Conference on Artificial Intelligence and Statistics , G. Gordon, et al. (Eds.), Vol. 15. JMLR, Fort Lauderdale, FL, USA, 315. http://proceedings.mlr.press/v15/glorot11a. html -
  41. [51][51] M. Pierini, et al. 2020. hls4ml LHC jet dataset (150 particles). https://doi.org/10. 5281/zenodo.3602260 -
  42. [52][52] E. Coleman, et al. 2018. The importance of calorimetry for highly-boosted jet substructure. J. Instrum. 13 (2018), T01003. https://doi.org/10.1088/17480221/13/01/T01003 arXiv:1709.08705 - · DOI
  43. [54][54] Y. Gong, et al. 2014. Compressing deep convolutional networks using vector quantization. (2014). arXiv:1412.6115 - · arXiv
  44. [55][55] J. Wu, et al. 2016. Quantized convolutional neural networks for mobile devices. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 4820. https://doi.org/10.1109/CVPR.2016.521 arXiv:1512.06473 - · DOI
  45. [56][56] V. Vanhoucke, et al. 2011. Improving the speed of neural networks on CPUs. In Deep Learning and Unsupervised Feature Learning Workshop at the 25th Conference on Neural Information Processing Systems . -
  46. [57][57] S. Gupta, et al. 2015. Deep Learning with Limited Numerical Precision. In 32nd International Conference on Machine Learning , F. Bach et al. (Eds.), Vol. 37. PMLR, Lille, France, 1737. arXiv:1502.02551 http://proceedings.mlr.press/v37/gupta15. html - · arXiv
  47. [58][58] S. Han, et al. 2016. Deep compression: Compressing deep neural network with pruning, trained quantization and Huffman Coding. In 4th International Conference on Learning Representations . arXiv:1510.00149 - · arXiv
  48. [59][59] I. Hubara, et al. 2018. Quantized neural networks: Training neural networks with low precision weights and activations. J. Mach. Learn. Res. 18, 187 (2018), 1. 31. [arXiv:1609.07061 http://jmlr.org/papers/v18/16-456.html](http://jmlr.org/papers/v18/16-456.html) - · arXiv
  49. [60][60] M. Courbariaux, et al. 2015. BinaryConnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems , C. Cortes, et al. (Eds.), Vol. 28. Curran Associates, Inc., 3123. arXiv:1511.00363 https://proceedings.neurips.cc/paper/2015/file/ 3e15cc11f979ed25912dff5b0669f2cd-Paper.pdf - · arXiv
  50. [61][61] I. Hubara, et al. 2016. Binarized neural networks. In Advances in Neural Information Processing Systems , D. D. Lee, et al. (Eds.), Vol. 29. Curran Associates, Inc., 4107. arXiv:1602.02830 https://proceedings.neurips.cc/paper/2016/file/ d8330f857a17c53d217014ee776bfd50-Paper.pdf - · arXiv
  51. [62][62] M. Rastegari, et al. 2016. XNOR-Net: ImageNet classification using binary convolutional neural networks. In ECCV 2016 , B. Leibe, et al. (Eds.). Springer, Cham, Switzerland, 525. arXiv:1603.05279 - · arXiv
  52. [63][63] P. Merolla, et al. 2016. Deep neural networks are robust to weight binarization and other non-linear distortions. (2016). arXiv:1606.01981 - · arXiv
  53. [64][64] B. Moons, et al. 2017. Minimum energy quantized neural networks. In 2017 51st Asilomar Conference on Signals, Systems, and Computers . 1921. https://doi.org/10. 1109/ACSSC.2017.8335699 arXiv:1711.00215 - · arXiv
  54. [65][65] F. Li et al. 2016. Ternary weight networks. (2016). arXiv:1605.04711 - · arXiv
  55. [66][66] K. Wang, et al. 2019. HAQ: Hardware-aware automated quantization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 8604. arXiv:1811.08886 - · arXiv
  56. [67][67] Z. Dong, et al. 2019. HAWQ: Hessian aware quantization of neural networks with mixed-precision. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . IEEE, 293. arXiv:1905.03696 - · arXiv
  57. [68][68] Y. Cheng, et al. 2018. A Survey of Model compression and acceleration for deep neural networks. IEEE Signal Process. Mag. 35 (2018), 126. https://doi.org/10. 1109/MSP.2017.2765695 arXiv:1710.09282 - · arXiv
  58. [69][69] L. Deng, et al. 2020. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proc. IEEE 108 (2020), 485. https://doi.org/ 10.1109/JPROC.2020.2976475 - · DOI · Paper page
  59. [70][70] T. Choudhary, et al. 2020. A comprehensive survey on model compression and acceleration. Artif. Intell. Rev. 53, 7 (2020), 5113. https://doi.org/10.1007/s10462020-09816-7 - · DOI
  60. [71][71] Y. LeCun, et al. 1990. Optimal brain damage. In Advances in Neural Information Processing Systems , D. S. Touretzky (Ed.), Vol. 2. Morgan-Kaufmann, 598. http: //papers.nips.cc/paper/250-optimal-brain-damage -
  61. [72][72] C. Louizos, et al. 2018. Learning sparse neural networks through 𝐿 0 regularization. In 6th International Conference on Learning Representations . arXiv:1712.01312 https://openreview.net/forum?id=H1Y8hhg0b - · arXiv
  62. [74][74] A. Renda, et al. 2020. Comparing rewinding and fine-tuning in neural network pruning. In 8th International Conference on Learning Representations . arXiv:2003.02389 - · arXiv
  63. [75][75] D. Blalock, et al. 2020. What is the state of neural network pruning?. In Proceedings of Machine Learning and Systems , I. Dhillon, et al. (Eds.), Vol. 2. 129. arXiv:2003.03033 https://proceedings.mlsys.org/paper/2020/file/ d2ddea18f00665ce8623e36bd4e3c7c5-Paper.pdf - · arXiv
  64. [76][76] S. Han, et al. 2015. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems , C. Cortes, et al. (Eds.), Vol. 28. Curran Associates, Inc., 1135. arXiv:1506.02626 https://proceedings. neurips.cc/paper/2015/file/ae0eb3eed39d2bcef4622b2499a05fe6-Paper.pdf - · arXiv
  65. [77][77] A. Y. Ng. 2004. Feature selection, L1 vs. L2 regularization, and rotational invariance. In 21st International Conference On Machine Learning (ICML '04) . ACM, New York, NY, USA, 78. https://doi.org/10.1145/1015330.1015435 - · DOI · Paper page
  66. [78][78] C. Baskin, et al. 2018. UNIQ: Uniform noise injection for the quantization of neural networks. (2018). arXiv:1804.10969 - · arXiv
  67. [80][80] D. S. Rankin, et al. 2020. FPGAs-as-a-Service Toolkit (FaaST). In 2020 IEEE/ACM International Workshop on Heterogeneous High-performance Reconfigurable Computing . arXiv:2010.08556 - · arXiv
  68. [81][81] B. Khailany, et al. 2018. A modular digital VLSI flow for high-productivity SoC design. In ACM/ESDA/IEEE Design Automation Conference (DAC) . IEEE, 1. -
  69. [83][83] C. Herwig, et al. 2020. Design of a reconfigurable autoencoder algorithm for detector front-end ASICs. In IEEE Nuclear Science Symposium & Medical Imaging Conference . IEEE. -
  70. [84][84] F. Fahim, et al. 2020. High-level synthesis to on-chip implementation of a reconfigurable AI accelerator for front-end data analysis at the HL-LHC. In IEEE Nuclear Science Symposium & Medical Imaging Conference . IEEE.

Cited by

Metadata

Authors
Fahim, F., Hawks, B., Herwig, C., Hirschauer, J., Jindariani, S., Tran, N., Carloni, L. P., Di Guglielmo, G., Harris, P., Krupa, J., Rankin, D., Valentin, M. B., Hester, J., Luo, Y., Mamish, J., Orgrenci-Memik, S., Aarrestad, T., Javed, H., Loncar, V., Pierini, M., Pol, A. A., Summers, S., Duarte, J., Hauck, S., Hsu, S.-C., Ngadiuba, J., Liu, M., Hoang, D., Kreinar, E., & Wu, Z.
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
OpenAlex
W3129272413
Cited by
62
References
0
Citations by year
2026: 1 · 2025: 4 · 2024: 14 · 2023: 22 · 2022: 14 · 2021: 7
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
  • CERN Document Server (European Organization for Nuclear Research) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (DE-AC02-07CH11359, 1764000)
  • U.S. Department of Energy (AC02-07CH11359, DESC0021187, DE‐AC02‐07CH11359, DE-AC02-)
  • European Commission (772369, HORIZON2020)
  • Defense Advanced Research Projects Agency (FA8650-18-2-7862)
  • Office of Science (DE-AC02, DE–AC02–07CH11359, DESC0021187)
  • Advanced Research Projects Agency
  • High Energy Physics (DE-SC0021187, DE-AC02–07CH11359)
  • Air Force Research Laboratory (FA8650-18-2-7862)
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Control and Systems Engineering › Embedded Systems and FPGA Design
Keywords
hls4ml, hardware-software co-design, low-power machine learning, FPGA implementation, ASIC implementation, near-sensor processing, energy-efficient hardware, Python interface, open source tools

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.