Layer-Specific Optimization for Mixed Data Flow With Mixed Precision in FPGA Design for CNN-Based Object Detectors
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Convolutional neural networks (CNNs) require both intensive computation and frequent memory access, which lead to a low processing speed and large power dissipation. Although the characteristics of the different layers in a CNN are frequently quite different, previous hardware designs have employed common optimization schemes for them. This paper proposes a layer-specific design that employs different organizations that are optimized for the different layers. The proposed design employs two layer-specific optimizations: layer-specific mixed data flow and layer-specific mixed precision. The mixed data flow aims to minimize the off-chip access while demanding a minimal on-chip memory (BRAM) resource of an FPGA device. The mixed precision quantization is to achieve both a lossless accuracy and an aggressive model compression, thereby further reducing the off-chip access. A Bayesian optimization approach is used to select the best sparsity for each layer, achieving the best trade-off between the accuracy and compression. This mixing scheme allows the entire network model to be stored in BRAMs of the FPGA to aggressively reduce the off-chip access, and thereby achieves a significant performance enhancement. The model size is reduced by 22.66-28.93 times compared to that in a full-precision network with a negligible degradation of accuracy on VOC, COCO, and ImageNet datasets. Furthermore, the combination of mixed dataflow and mixed precision significantly outperforms the previous works in terms of both throughput, off-chip access, and on-chip memory requirement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · cited 1×
“In addition, 8-bit precision is also adopted by Google TPU [56].”
MethodXNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408
Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks
Chen, Y.-H., Krishna, T., Emer, J. S., & Sze, V. (2017) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2016.2616357 · Cited by 3,247 · cited 1×
“le a 32-bit data word access from DRAM (45nm CMOS technology) requires 640pJ [25]. There are a number of previous works designing a single layer accelerator for CNNs [13]-[15], [16], [18], [26]-[29], [31]. These works optimize the processing of a single CNN layer through loop optimization to increase hardware utilizat…”
MethodOptimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks
Zhang, C., Li, P., Sun, G., Guan, Y., Xiao, B., & Cong, J. (2015) · Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/2684746.2689060 · Cited by 2,158 · cited 1×
“There are several works that presents a multi-layer processor approach, in which each layer is processed by a dedicated hardware unit [10], [11], [33], [34], to maximize the utilization of computing resources.”
BackgroundA tutorial on Gaussian process regression: modelling, exploring, and exploiting functions
Schulz, E., Speekenbrink, M., & Krause, A. (2018) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2018.03.001 · Cited by 1,735
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690
DSSD : Deconvolutional Single Shot Detector
Fu, C.-Y., Liu, W., Ranga, A., Tyagi, A., & Berg, A. C. (2017) · arXiv · doi:10.48550/arxiv.1701.06659 · Cited by 1,624 · cited 1×
“Therefore, numerous promising approaches have been proposed for object detection with deep learning such as single-shot multibox detection (SSD) [1], faster R-CNN [2], RetinaNet [3], DSSD [4], and YOLO [5].”
BackgroundLearning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481 · cited 8×
“Pruning [25] reduces parameter size of convolutional layers of VGGNet by 4.”
BackgroundMethodInfluential
Research works printed but not linked: 44
- gupta, 2015, Deep learning with limited numerical precisions, Proc Int Conf Mach Learn (ICML), 1737
- 10.1145/3061639.3062244 · DOI · Paper page
- 10.1145/3173162.3173176 · DOI · Paper page
- 10.1109/TVLSI.2017.2688340 · DOI · Paper page
- 10.1109/HPCA.2017.29 · DOI · Paper page
- 10.1145/3079856.3080221 · DOI · Paper page
- 10.1109/TCSVT.2018.2864321 · DOI
- 10.1109/DAC18072.2020.9218724 · DOI
- kang, 2020, Accelerator-aware pruning for convolutional neural networks, IEEE Trans Circuits Syst Video Technol, 30, 2093
- 10.23919/DATE.2018.8342033 · DOI
- 10.1109/TC.2018.2890258 · DOI
- 10.1145/3020078.3021738 · DOI · Paper page
- liu, 2015, SSD: Single shot MultiBox detector, arXiv 1512 02325
- 10.1109/CVPR.2015.7298594 · DOI · Paper page
- redmon, 2018, YOLOv3: An incremental improvement, arXiv 1804 02767
- 10.1109/ISCAS.2018.8351021 · DOI · Paper page
- 10.1109/TCSI.2019.2962516 · DOI
- 10.1145/3079856.3080254 · DOI · Paper page
- 10.1145/3289602.3293977 · doi:10.1145/3289602.3293977
- zhou, 2017, Incremental network quantization: Towards lossless CNNs with low-precision weights, arXiv 1702 03044
- 10.1109/TCSVT.2018.2883087 · DOI
- 10.1145/3316781.3317739 · DOI · Paper page
- 10.1145/2847263.2847265 · DOI · Paper page
- han, 2015, Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding, arXiv 1510 00149 [cs]
- 10.1109/CVPR.2018.00474 · DOI · Paper page
- 10.1145/3242897 · DOI · Paper page
- 10.1145/3240765.3240801 · DOI · Paper page
- 10.1109/CVPR.2019.00881 · DOI · Paper page
- 10.1016/j.neucom.2017.09.046 · DOI · Paper page
- li, 2016, A high performance FPGA-based accelerator for large-scale convolutional neural networks, Proc Int Conf Field Program Logic Appl (FPL), 1
- 10.1109/FCCM.2017.64 · DOI · Paper page
- 10.1109/TVLSI.2018.2815603 · DOI · Paper page
- park, 2018, Value-aware quantization for training and inference of neural networks, arXiv 1804 07802
- 10.1109/ISCA.2018.00063 · DOI · Paper page
- 10.1109/HPCA.2019.00030 · DOI
- 10.1109/ICCV.2017.324 · DOI · Paper page
- 10.1145/3174243.3174266 · DOI · Paper page
- 10.23919/DATE.2018.8342121 · DOI · Paper page
- 10.1145/3289602.3293904 · DOI
- 10.1145/3007787.3001163 · DOI · Paper page
- 10.1007/s11263-015-0816-y · DOI · Paper page
- lin, 0, Microsoft COCO: Common objects in context, Proc Eur Conf Comput Vision (ECCV), 2014, 740
- snoek, 2012, Pratical Bayesian optimization of machine algorithms, Proc Adv Neural Inf Process Syst (NIPS), 2951
- 2020, BOA The Bayesian optimization algorithm
Cited by
Kim, S., & Kim, H. (2021) · IEEE Access · doi:10.1109/access.2021.3054879 · Cited by 71
“However, because a DNN uses many layers, a large number The associate editor coordinating the review of this manuscript and approving it for publication was Jun Li . of parameters are required, which significantly increases computational complexity [5]–[7].”
Self-citationImproving Extreme Low-Bit Quantization With Soft Threshold
Xu, W., Li, F., Jiang, Y., Yong, A., He, X., Wang, P., & Cheng, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3216389 · Cited by 32
Oflamaz, C. U., & Yalçın, M. E. (2025) · Electronics · doi:10.3390/electronics14183686 · Cited by 4
Metadata
- Authors
- Nguyen, D. T., Kim, H., & Lee, H.-J.
- Venue
- IEEE Transactions on Circuits and Systems for Video Technology
- Published
- 2021-06
- Volume
- 31
- Issue
- 6
- Pages
- 2450-2464
- ISSN
- 1051-8215 (ISSN-L)
- 1558-2205
- OpenAlex
- W3082058622
- Cited by
- 97
- References
- 18
- FWCI
- 4.67
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Ministry of Science and ICT (MSIT), Korea Government (2020-0-01305)
- Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2019R1A6A1A03032119)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Layer-Specific Optimization for Mixed Data Flow With Mixed Precision in FPGA Design for CNN-Based Object Detectors. https://app.cochat.ai/papers/10.1109/tcsvt.2020.3020569
- The paper
- Nguyen, D. T., Kim, H., & Lee, H.-J. (2021). Layer-Specific Optimization for Mixed Data Flow With Mixed Precision in FPGA Design for CNN-Based Object Detectors. IEEE Transactions on Circuits and Systems for Video Technology. https://doi.org/10.1109/tcsvt.2020.3020569
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.