A Novel FPGA Accelerator Design for Real-Time and Ultra-Low Power Deep Convolutional Neural Networks Compared With Titan X GPU
Open accessAbstract
Convolutional neural networks (CNNs) based deep learning algorithms require high data flow and computational intensity. For real-time industrial applications, they need to overcome challenges such as high data bandwidth requirement and power consumption on hardware platforms. In this work, we have analyzed in detail the data dependency in the CNN accelerator and propose specific pipelined operations and data organized manner to design a high throughput CNN accelerator on FPGA. Besides, we have optimized the kernel operations to obtain a high power efficiency. The proposed CNN accelerator supports image classification and real-time object detection with high accuracy. The evaluation results show that our CNN-based FPGA accelerator can achieve 740 Giga operations per second (GOPS) at 200 MHz with kernel power of 12.2 watts on Intel Arria 10 FPGA. For object detection tasks, our system can achieve 105 fps with 56.5 mAP or 25 fps with 73.6 mAP on VOC dataset. Since we use the mixed fixed-point data representation, the detection accuracy is comparable with the GPU-based YOLO V2 framework. The power efficiency of our system is 3.3× better than Titan X GPU and 418× better than Intel E5-2620 V4 CPU.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large, reconfigurable processing elements with low power consumption have make FPGA widely used recently due to the rapid growth in demand in AI, so that 92% of the execution time is spent on MACcomputations during forward inference in YOLO framework.
“In conclusion, large, reconfigurable processing elements with low power consumption have make FPGA widely used recently due to the rapid growth in demand in AI, so that 92% of the execution time is spent on MACcomputations during forward inference in YOLO framework [85].”
From Embedded Intelligence on FPGA: Survey, Applications and Challenges (2021) · cites this paper as [85]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892 · see citing works
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · see citing works
Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks
Zhang, C., Li, P., Sun, G., Guan, Y., Xiao, B., & Cong, J. (2015) · Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/2684746.2689060 · Cited by 2,158 · see citing works
EIE: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001163 · Cited by 1,937 · see citing works
Pruning Convolutional Neural Networks for Resource Efficient InferencePreprint
Molchanov, P. A., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06440 · Cited by 1,487 · see citing works
Angel-Eye: A Complete Design Flow for Mapping CNN Onto Embedded FPGA
Guo, K., Sui, L., Qiu, J., Yu, J., Wang, J., Yao, S., Han, S., Wang, Y., & Yang, H. (2017) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2017.2705069 · Cited by 559 · see citing works
Throughput-Optimized OpenCL-based FPGA Accelerator for Large-Scale Convolutional Neural Networks
Suda, N., Chandra, V., Dasika, G. S., Mohanty, A., Ma, Y., Vrudhula, S. B. K., Seo, J.-S., & Cao, Y. K. (2016) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/2847263.2847276 · Cited by 556 · see citing works
Minimizing Computation in Convolutional Neural Networks
Cong, J., & Xiao, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11179-7_36 · Cited by 379 · see citing works
CNP: An FPGA-based processor for Convolutional Networks
Farabet, C., Poulet, C., Han, J. Y., & LeCun, Y. (2009) · 2009 International Conference on Field Programmable Logic and Applications · doi:10.1109/fpl.2009.5272559 · Cited by 367 · see citing works
Optimizing the Convolution Operation to Accelerate Deep Neural Networks on FPGA
Ma, Y., Cao, Y. K., Vrudhula, S. B. K., & Seo, J.-S. (2018) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2018.2815603 · Cited by 346 · see citing works
Guan, Y., Liang, H., Xu, N., Wang, W., Shi, S., Chen, X., Sun, G., Zhang, W., & Cong, J. (2017) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm.2017.25 · Cited by 318 · see citing works
A high performance FPGA-based accelerator for large-scale convolutional neural networks
Li, H., Fan, X., Li, J., Cao, W., Zhou, X., & Wang, L. (2016) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl.2016.7577308 · Cited by 306 · see citing works
A 240 G-ops/s Mobile Coprocessor for Deep Neural Networks
Gokhale, V., Jin, J., Dündar, A., Martini, B., & Culurciello, E. (2014) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2014.106 · Cited by 300 · see citing works
Caffeine: Toward Uniformed Representation and Acceleration for Deep Convolutional Neural Networks
Zhang, C., Sun, G., Fang, Z., Zhou, P., Pan, P., & Cong, J. (2019) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2017.2785257 · Cited by 270 · see citing works
A CNN Accelerator on FPGA Using Depthwise Separable Convolution
Bai, L., Zhao, Y., & Huang, X. (2018) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2018.2865896 · Cited by 253 · see citing works
An OpenCL™ Deep Learning Accelerator on Arria 10
Aydonat, U., O’Connell, S., Capalija, D., Ling, A. C., & Chiu, G. R. (2017) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3020078.3021738 · Cited by 229 · see citing works
Design space exploration of FPGA-based Deep Convolutional Neural Networks
Motamedi, M., Gysel, P., Akella, V., & Ghiasi, S. (2016) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/aspdac.2016.7428073 · Cited by 211 · see citing works
A Lightweight YOLOv2: A Binarized CNN with A Parallel Support Vector Regression for an FPGA
Nakahara, H., Yonekawa, H., Fujii, T., & Sato, S. (2018) · Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3174243.3174266 · Cited by 139 · see citing works
GraphGen: An FPGA Framework for Vertex-Centric Graph Computation
Nurvitadhi, E., Weisz, G., Wang, Y., et al. (2014) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm.2014.15 · Cited by 122 · see citing works
Caffeinated FPGAs: FPGA framework For Convolutional Neural Networks
DiCecco, R., Lacey, G., Vasiljevic, J., Chow, P., Taylor, G. W., & Areibi, S. M. (2016) · International Conference on Field-Programmable Technology (FPT) · doi:10.1109/fpt.2016.7929549 · Cited by 107 · see citing works
A Framework for Generating High Throughput CNN Implementations on FPGAs
Zeng, H., Chen, R., Zhang, C., et al. (2018) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3174243.3174265 · Cited by 99 · see citing works
Low-Power Computer Vision: Status, Challenges, and Opportunities
Alyamkin, S., Ardi, M., Berg, A. C., Brighton, A., Chen, B., Chen, Y., Cheng, H.-P., Fan, Z., Feng, C., Fu, B., et al. (2019) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2019.2911899 · Cited by 83 · see citing works
Automatic Compilation of Diverse CNNs Onto High-Performance FPGA Accelerators
Ma, Y., Cao, Y. K., Vrudhula, S. B. K., & Seo, J.-S. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2018.2884972 · Cited by 69 · see citing works
Geng, T., Wang, T., Sanaullah, A., et al. (2018) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl.2018.00074 · Cited by 53 · see citing works
Inference of quantized neural networks on heterogeneous all-programmable devices
Preuser, T. B., Gambardella, G., Fraser, N., & Blott, M. (2018) · Design, Automation, and Test in Europe · doi:10.23919/date.2018.8342121 · Cited by 50 · see citing works
FPGA-Based Convolutional Neural Network Architecture with Reduced Parameter Requirements
Hailesellasie, M., Hasan, S. R., Khalid, F., Wad, F. A., & Shafique, M. (2018) · 2018 IEEE International Symposium on Circuits and Systems (ISCAS) · doi:10.1109/iscas.2018.8351283 · Cited by 38 · see citing works
Motamedi, M., Gysel, P., & Ghiasi, S. (2017) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3131289 · Cited by 28 · see citing works
A Threshold Neuron Pruning for a Binarized Deep Neural Network on an FPGA
Fujii, T., Sato, S., & Nakahara, H. (2018) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2017rcp0013 · Cited by 21 · see citing works
Li, S., Luo, Y., Sun, K., & Choi, K. (2018) · International Conference on Electronics, Information, and Communication (ICEIC) · doi:10.23919/elinfocom.2018.8330645 · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Embedded Intelligence on FPGA: Survey, Applications and Challenges
Seng, K. P., Lee, P. J., & Ang, L. M. (2021) · Electronics · doi:10.3390/electronics10080895 · Cited by 105 · see citing works
“In conclusion, large, reconfigurable processing elements with low power consumption have make FPGA widely used recently due to the rapid growth in demand in AI, so that 92% of the execution time is spent on MAC computations during forward inference in YOLO framework [85].”
Method
- Authors
- Li, S., Luo, Y., Sun, K., Yadav, N., & Choi, K. K.
- Affiliations
- Illinois Institute of Technology (US)
- University of Illinois Chicago (US)
- Rice University (US)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 105455-105471
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3033155721
- Cited by
- 75
- References
- 33
- Citations by year
- 2026: 11 · 2025: 17 · 2024: 12 · 2023: 9 · 2022: 14 · 2021: 9 · 2020: 3
- FWCI
- 3.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Industrial Core Technology Development Program of the Ministry of Trade, Industry and Energy (MOTIE)/Korea Electronics Technology Institute (KETI), South Korea (10083639)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Embedded Systems and FPGA Applications
- Keywords
- FPGA accelerator, convolutional neural networks, deep learning hardware acceleration, real-time object detection, image classification, low power design, pipelined architecture, data dependence analysis, fixed-point arithmetic, GPU comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
