The DOI matches its Crossref record; 181 of the 181 references it cites were found in a registry or an index.
Lightweight Deep Learning for Resource-Constrained Environments: A Survey
Open accessAbstract
Over the past decade, the dominance of deep learning has prevailed across various domains of artificial intelligence, including natural language processing, computer vision, and biomedical signal processing. While there have been remarkable improvements in model accuracy, deploying these models on lightweight devices, such as mobile phones and microcontrollers, is constrained by limited resources. In this survey, we provide comprehensive design guidance tailored for these devices, detailing the meticulous design of lightweight models, compression methods, and hardware acceleration strategies. The principal goal of this work is to explore methods and concepts for getting around hardware constraints without compromising the model’s accuracy. Additionally, we explore two notable paths for lightweight deep learning in the future: deployment techniques for TinyML and Large Language Models. Although these paths undoubtedly have potential, they also present significant challenges, encouraging research into unexplored areas.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.-C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., Pang, R., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00140 · Cited by 9,708
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Ma, N., Zhang, X., Zheng, H.-T., & Sun, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_8 · Cited by 7,356 · cited 4×
“ShuffleNetV2 [137] investigates four practical guidelines to design a memory-efficient and light-weight model that can avoid heavy MAC problems.”
BackgroundInfluentialModel Selection and Estimation in Regression with Grouped Variables
Yuan, M., & Lin, Y. (2006) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00532.x · Cited by 7,261
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00286 · Cited by 4,114
Neural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
MnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416
Swin Transformer V2: Scaling Up Capacity and Resolution
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.09883 · Cited by 2,679
Channel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · cited 1×
“He et al. [83] first implement channel pruning by focusing on eliminating redundant channels by evaluating the L1 norm.”
BackgroundLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Swin Transformer V2: Scaling Up Capacity and Resolution
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01170 · Cited by 2,484
CvT: Introducing Convolutions to Vision Transformers
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00009 · Cited by 2,444
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E., Feng, J., & Yan, S. (2021) · arXiv · doi:10.48550/arxiv.2101.11986 · Cited by 2,415
Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks
Zhang, C., Li, P., Sun, G., Guan, Y., Xiao, B., & Cong, J. (2015) · Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/2684746.2689060 · Cited by 2,158
Designing Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollar, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01044 · Cited by 2,084
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S.-N. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19827-4_41 · Cited by 1,965
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.01274 · Cited by 1,819
Self-citationZhang, Y., Xiang, T., Hospedales, T. M., & Lu, H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00454 · Cited by 1,752
AutoML: A survey of the state-of-the-art
He, X., Zhao, K., & Chu, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106622 · Cited by 1,691
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020) · Communications of the ACM · doi:10.1145/3381831 · Cited by 1,647
Gray, R. M., & Neuhoff, D. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.720541 · Cited by 1,642
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Frankle, J., & Carbin, M. (2018) · arXiv · doi:10.48550/arxiv.1803.03635 · Cited by 1,529
DaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1109/micro.2014.58 · Cited by 1,492
Bottleneck Transformers for Visual Recognition
Srinivas, A., Lin, T.-Y., Parmar, N., Shlens, J., Abbeel, P., & Vaswani, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01625 · Cited by 1,385
A data locality optimizing algorithm
Wolf, M. E., & Lam, M. S. (1991) · Proceedings of the ACM SIGPLAN 1991 conference on Programming language design and implementation - PLDI '91 · doi:10.1145/113445.113449 · Cited by 1,331
Efficient transformers: A survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
Wu, B., Keutzer, K., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., & Jia, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01099 · Cited by 1,261
Going Deeper with Embedded FPGA Platform for Convolutional Neural Network
Qiu, J., Wang, J., Yao, S., Guo, K., Li, B., Zhou, E., Yu, J., Tang, T., Xu, N., Song, S., et al. (2016) · Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/2847263.2847265 · Cited by 1,252
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
He, Y., Liu, P., Wang, Z., Hu, Z., & Yang, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00447 · Cited by 1,217
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154 · cited 1×
“Transformer models are widely used in NLP [202] and have recently obtained promising results in computer vision tasks [132, 133, 243].”
MethodSoft Filter Pruning for Accelerating Deep Convolutional Neural Networks
He, Y., Kang, G., Dong, X., Fu, Y., & Yang, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/309 · Cited by 1,020
Decoupled Knowledge Distillation
Zhao, B., Cui, Q., Song, R., Qiu, Y., & Liang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01165 · Cited by 965
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., & Ma, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00381 · Cited by 954
Pruning and quantization for deep neural network acceleration: A survey
Liang, T., Glossner, J., Wang, L., Shi, S., & Zhang, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.045 · Cited by 943
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference
Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jegou, H., & Douze, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01204 · Cited by 935
CoAtNet: Marrying Convolution and Attention for All Data Sizes
Dai, Z., Liu, H., Le, Q. V., & Tan, M. (2021) · arXiv · doi:10.48550/arxiv.2106.04803 · Cited by 925
Self-citationCondenseNet: An Efficient DenseNet Using Learned Group Convolutions
Huang, G., Liu, S., Maaten, L. V. D., & Weinberger, K. Q. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00291 · Cited by 904
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
Self-citationLearning Lightweight Lane Detection CNNs by Self Attention Distillation
Hou, Y., Ma, Z., Liu, C., & Loy, C. C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00110 · Cited by 780
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Mobile-Former: Bridging MobileNet and Transformer
Chen, Y., Dai, X., Chen, D., Liu, M., Dong, X., Yuan, L., & Liu, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00520 · Cited by 731
PACT: Parameterized Clipping Activation for Quantized Neural Networks
Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srinivasan, V., & Gopalakrishnan, K. (2018) · arXiv · doi:10.48550/arxiv.1805.06085 · Cited by 703
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1×
“Choromanski et al. [36] propose a linear self-attention mechanism based on the FAVOR+ (fast attention with positive orthogonal random features) approach to construct an approximate softmax operation.”
MethodSNIP: Single-shot Network Pruning based on Connection Sensitivity
Lee, N., Ajanthan, T., & Torr, P. H. S. (2018) · arXiv · doi:10.48550/arxiv.1810.02340 · Cited by 685
Auto-WEKA: Automatic Model Selection and Hyperparameter Optimization in WEKA
Kotthoff, L., Thornton, C., Hoos, H. H., Hutter, F., & Leyton-Brown, K. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_4 · Cited by 650
Optimal Brain Surgeon and general network pruning
Hassibi, B., Stork, D. G., & Wolff, G. J. (2002) · IEEE International Conference on Neural Networks · doi:10.1109/icnn.1993.298572 · Cited by 597
A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3447582 · Cited by 590
Distilling Knowledge via Knowledge Review
Chen, P., Liu, S., Zhao, H., & Jia, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00497 · Cited by 575
Revisiting Knowledge Distillation via Label Smoothing Regularization
Yuan, L., Tay, F. E., Li, G., Wang, T., & Feng, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00396 · Cited by 572
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
Yang, T.-J., Howard, A., Chen, B., Zhang, X., Go, A., Sandler, M., Sze, V., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_18 · Cited by 572
MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
Liu, Z., Mu, H., Zhang, X., Guo, Z., Yang, X., Cheng, K.-T., & Sun, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00339 · Cited by 563
Angel-Eye: A Complete Design Flow for Mapping CNN Onto Embedded FPGA
Guo, K., Sui, L., Qiu, J., Yu, J., Wang, J., Yao, S., Han, S., Wang, Y., & Yang, H. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2017.2705069 · Cited by 562
HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision
Dong, Z., Yao, Z., Gholami, A., Mahoney, M., & Keutzer, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00038 · Cited by 492
A review on TinyML: State-of-the-art and prospects
Ray, P. P. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.11.019 · Cited by 460
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., & Hsieh, C.-J. (2021) · arXiv · doi:10.48550/arxiv.2106.02034 · Cited by 448 · cited 3×
“This observation is apparent from Table 3, where many recent transformers, such as T2T-ViT [237] and DymViT-LVit [158], are shown to achieve higher accuracy with significantly fewer parameters and lower FLOPs than the original ViT and ResNet-based CNNs. Specifically, we split the discussion into…”
BackgroundIntriguing Properties of Vision Transformers
Naseer, M., Ranasinghe, K., Khan, S., Hayat, M., Khan, F. S., & Yang, M.-H. (2021) · arXiv · doi:10.48550/arxiv.2105.10497 · Cited by 431 · cited 2×
“Another series of efficient transformers [122, 147, 158, 232, 237] aims to prune the transformer structure efficiently via token sparsing.”
BackgroundAuto-sklearn: Efficient and Robust Automated Machine Learning
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J. T., Blum, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_6 · Cited by 430
SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
Wang, H., Zhang, Z., & Han, S. (2021) · 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca51647.2021.00018 · Cited by 427
Automated Systolic Array Architecture Synthesis for High Throughput CNN Inference on FPGAs
Wei, X., Yu, C. H., Zhang, P., Chen, Y., Wang, Y., Hu, H., Liang, Y., & Cong, J. (2017) · Proceedings of the 54th Annual Design Automation Conference 2017 · doi:10.1145/3061639.3062207 · Cited by 426
SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., Emer, J., Keckler, S. W., & Dally, W. J. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080254 · Cited by 413
Unbiased Mean Teacher for Cross-domain Object Detection
Deng, J., Li, W., Chen, Y., & Duan, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00408 · Cited by 401
Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss
Jung, S., Son, C., Lee, S., Son, J., Han, J.-J., Kwak, Y., Hwang, S. J., & Choi, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00448 · Cited by 385
ZeroQ: A Novel Zero Shot Quantization Framework
Cai, Y., Yao, Z., Dong, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01318 · Cited by 381
Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
Wu, B., Wan, A., Yue, X., Jin, P., Zhao, S., Golmant, N., Gholaminejad, A., Gonzalez, J., & Keutzer, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00951 · Cited by 381
A-ViT: Adaptive Tokens for Efficient Vision Transformer
Yin, H., Vahdat, A., Alvarez, J. M., Mallya, A., Kautz, J., & Molchanov, P. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01054 · Cited by 360
SqueezeNext: Hardware-Aware Neural Network Design
Gholami, A., Kwon, K., Wu, B., Tai, Z., Yue, X., Jin, P., Zhao, S., & Keutzer, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00215 · Cited by 350
Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
Dong, X., Chen, S., & Pan, S. J. (2017) · arXiv · doi:10.48550/arxiv.1705.07565 · Cited by 339
Early Convolutions Help Transformers See Better
Xiao, T., Singh, M., Mintun, E., Darrell, T. J., Dollár, P., & Girshick, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14881 · Cited by 339
Cross-Layer Distillation with Semantic Calibration
Chen, D., Mei, J.-P., Zhang, Y., Wang, C., Wang, Z., Feng, Y., & Chen, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16865 · Cited by 335
The Reversible Residual Network: Backpropagation Without Storing Activations
Gomez, A. N., Ren, M., Urtasun, R., & Grosse, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04585 · Cited by 335
DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · ACM SIGARCH Computer Architecture News · doi:10.1145/2654822.2541967 · Cited by 335
Online Knowledge Distillation via Collaborative Learning
Guo, Q., Wang, X., Wu, Y., Yu, Z., Liang, D., Hu, X., & Luo, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01103 · Cited by 323
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
Wan, A., Dai, X., Zhang, P., He, Z., Tian, Y., Xie, S., Wu, B., Yu, M., Xu, T., Chen, K., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01298 · Cited by 305
Rethinking Bottleneck Structure for Efficient Mobile Network Design
Zhou, D., Hou, Q., Chen, Y., Feng, J., & Yan, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58580-8_40 · Cited by 292
Sun, Y., Wang, H., Xue, B., Jin, Y., Yen, G. G., & Zhang, M. (2020) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2924461 · Cited by 290
CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs
Oliveira, A., Costa, D., Moreira, G., Martins, J., & Pinto, S. (2018) · arXiv (Cornell University) · doi:10.4230/oasics.ng-res.2025.4 · Cited by 288
Caffeine: Toward Uniformed Representation and Acceleration for Deep Convolutional Neural Networks
Zhang, C., Sun, G., Fang, Z., Zhou, P., Pan, P., & Cong, J. (2019) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2017.2785257 · Cited by 270
Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration
He, Y., Ding, Y., Liu, P., Zhu, L., Zhang, H., & Yang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00208 · Cited by 262
MCUNet: Tiny Deep Learning on IoT Devices
Lin, J., Chen, W.-M., Lin, Y., Cohn, J., Gan, C., & Han, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.10319 · Cited by 259
Knowledge Distillation with the Reused Teacher Classifier
Chen, D., Mei, J.-P., Zhang, H., Wang, C., Feng, Y., & Chen, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01163 · Cited by 254
Self-Knowledge Distillation with Progressive Refinement of Targets
Kim, K., Ji, B., Yoon, D., & Hwang, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00650 · Cited by 237
A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification
Rokh, B., Azarpeyvand, A., & Khanteymoori, A. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3623402 · Cited by 232
Capra, M., Bussolino, B., Marchisio, A., Masera, G., Martina, M., & Shafique, M. (2020) · IEEE Access · doi:10.1109/access.2020.3039858 · Cited by 222
Zhou, X., Du, Z., Guo, Q., Liu, S., Liu, C., Wang, C., Zhou, X., Li, L., Chen, T., & Chen, Y. (2018) · 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2018.00011 · Cited by 214
Snapshot Distillation: Teacher-Student Optimization in One Generation
Yang, C., Xie, L., Su, C., & Yuille, A. L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00297 · Cited by 208
A Reconfigurable Streaming Deep Convolutional Neural Network Accelerator for Internet of Things
Du, L., Du, Y., Li, Y., Su, J., Kuan, Y.-C., Liu, C.-C., & Chang, M.-C. F. (2018) · IEEE Transactions on Circuits and Systems I: Regular Papers · doi:10.1109/tcsi.2017.2735490 · Cited by 205
Self-citationAdderNet: Do We Really Need Multiplications in Deep Learning?
Chen, H., Wang, Y., Xu, C., Shi, B., Xu, C., Tian, Q., & Xu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00154 · Cited by 199
Scalable Methods for 8-bit Training of Neural Networks
Banner, R., Hubara, I., Hoffer, E., & Soudry, D. (2018) · arXiv · doi:10.48550/arxiv.1805.11046 · Cited by 193
Post-Training Quantization on Diffusion Models
Shang, Y., Yuan, Z., Xie, B., Wu, B., & Yan, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00196 · Cited by 193
Liu, B., Li, F., Wang, X., Zhang, B., & Yan, J. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10094626 · Cited by 191
Banbury, C., Zhou, C., Fedorov, I., Navarro, R. M., Thakker, U., Gope, D., Reddi, V. J., Mattina, M., & Whatmough, P. N. (2020) · arXiv · doi:10.48550/arxiv.2010.11267 · Cited by 183
A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models
Ye, J., Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., Cui, Y., Zhou, Z., Gong, C., Shen, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.10420 · Cited by 180
DaDianNao: A Neural Network Supercomputer
Luo, T., Liu, S., Li, L., Wang, Y., Zhang, S., Chen, T., Xu, Z., Temam, O., & Chen, Y. (2017) · IEEE Transactions on Computers · doi:10.1109/tc.2016.2574353 · Cited by 175
Exploring Lightweight Hierarchical Vision Transformers for Efficient Visual Tracking
Kang, B., Chen, X., Wang, D., Peng, H., & Lu, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00881 · Cited by 152
Knowledge Distillation via the Target-aware Transformer
Lin, S., Xie, H., Wang, B., Yu, K., Chang, X., Liang, X., & Wang, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01064 · Cited by 149
Self-citationFBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
Dai, X., Wan, A., Zhang, P., Wu, B., He, Z., Wei, Z., Chen, K., Tian, Y., Yu, M., Vajda, P., et al. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01601 · Cited by 147
CALIP: Zero-Shot Enhancement of CLIP with Parameter-Free Attention
Guo, Z., Zhang, R., Qiu, L., Ma, X., Miao, X., He, X., & Cui, B. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25152 · Cited by 145
CMix-NN: Mixed Low-Precision CNN Library for Memory-Constrained Edge Devices
Capotondi, A., Rusci, M., Fariselli, M., & Benini, L. (2020) · IEEE Transactions on Circuits and Systems II: Express Briefs · doi:10.1109/tcsii.2020.2983648 · Cited by 135
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
Frantar, E., & Alistarh, D. (2023) · arXiv · doi:10.48550/arxiv.2301.00774 · Cited by 131
Compression of Deep Learning Models for Text: A Survey
Gupta, M., & Agrawal, P. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3487045 · Cited by 131
Neural Architecture Search Based on a Multi-Objective Evolutionary Algorithm With Probability Stack
Xue, Y., Chen, C., & Słowik, A. (2023) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2023.3252612 · Cited by 123
DeepShift: Towards Multiplication-Less Neural Networks
Elhoushi, M., Chen, Z., Shafiq, F., Tian, Y. H., & Li, J. Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00268 · Cited by 117
HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
Dong, Z., Yao, Z., Cai, Y., Arfeen, D., Gholami, A., Mahoney, M. W., & Keutzer, K. (2019) · arXiv · doi:10.48550/arxiv.1911.03852 · Cited by 110
Embedded Intelligence on FPGA: Survey, Applications and Challenges
Seng, K. P., Lee, P. J., & Ang, L. M. (2021) · Electronics · doi:10.3390/electronics10080895 · Cited by 105 · cited 1×
“For more references on FPGA AI applications, consult References [148, 170].”
BackgroundSparse-YOLO: Hardware/Software Co-Design of an FPGA Accelerator for YOLOv2
Wang, Z., Xu, K., Wu, S., Liu, L., Liu, L., & Wang, D. (2020) · IEEE Access · doi:10.1109/access.2020.3004198 · Cited by 105
Chen, W., Xie, D., Zhang, Y., & Pu, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00741 · Cited by 97
FPGA-based Deep Learning Inference Accelerators: Where Are We Standing?
Nechi, A., Groth, L., Mulhem, S., Merchant, F., Buchty, R., & Bereković, M. (2023) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3613963 · Cited by 94
A Simple and Effective Pruning Approach for Large Language Models
Sun, M., Liu, Z., Bair, A., & Kolter, J. Z. (2023) · arXiv · doi:10.48550/arxiv.2306.11695 · Cited by 92
Fully Learnable Group Convolution for Acceleration of Deep Neural Networks
Wang, X., Kan, M., Shan, S., & Chen, X. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00926 · Cited by 87
Deep Model Compression and Architecture Optimization for Embedded Systems: A Survey
Berthelier, A., Château, T., Duffner, S., García, C., & Blanc, C. (2020) · Journal of Signal Processing Systems · doi:10.1007/s11265-020-01596-1 · Cited by 86
Li, Y., Hao, C., Zhang, X., Liu, X., Chen, Y., Xiong, J., Hwu, W., & Chen, D. (2020) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac18072.2020.9218749 · Cited by 81
Self-citationHow to Evaluate Deep Neural Network Processors: TOPS/W (Alone) Considered Harmful
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2020) · IEEE Solid-State Circuits Magazine · doi:10.1109/mssc.2020.3002140 · Cited by 80
Distilling a Powerful Student Model via Online Knowledge Distillation
Li, S., Lin, M., Wang, Y., Wu, Y., Tian, Y., Shao, L., & Ji, R. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3152732 · Cited by 78
DetNAS: Neural Architecture Search on Object Detection.Preprint
Chen, Y., Yang, T., Zhang, X., Meng, G., Pan, C., & Sun, J. (2019) · arXiv (Cornell University) · Cited by 77
Accelerator-aware Neural Network Design using AutoMLPreprint
Gupta, S., & Akin, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02838 · Cited by 76
CondenseNet V2: Sparse Feature Reactivation for Deep Networks
Yang, L., Jiang, H., Cai, R., Wang, Y., Song, S., Huang, G., & Tian, Q. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00357 · Cited by 71
MTTrans: Cross-domain Object Detection with Mean Teacher Transformer
Yu, J., Liu, J., Wei, X., Zhou, H., Nakata, Y., Gudovskiy, D., Okuno, T., Li, J., Keutzer, K., & Zhang, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20077-9_37 · Cited by 71
Chen, B., Medini, T., Farwell, J., Gobriel, S., Tai, C., & Shrivastava, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03129 · Cited by 70
Face Detection and Verification Using Lensless Cameras
Tan, J., Niu, L., Adams, J. K., Boominathan, V., Robinson, J. T., Baraniuk, R. G., & Veeraraghavan, A. (2018) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2018.2889933 · Cited by 69
Data and Hardware Efficient Design for Convolutional Neural Network
Lin, Y.-J., & Chang, T. (2017) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2017.2759803 · Cited by 68
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks
Luo, G., Zhou, Y., Sun, X., Wang, Y., Cao, L., Wu, Y., Huang, F., & Ji, R. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3139234 · Cited by 67
Efficient Semantic Segmentation via Self-Attention and Self-Distillation
An, S., Liao, Q., Lu, Z., & Xue, J. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3139001 · Cited by 66
Dynamic Group Convolution for Accelerating Convolutional Neural Networks
Su, Z., Fang, L., Kang, W., Hu, D., Pietikäinen, M., & Liu, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58539-6_9 · Cited by 64
Learnable Lookup Table for Neural Network Quantization
Wang, L., Dong, X., Wang, Y., Liu, L., An, W., & Guo, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01210 · Cited by 58
ShiftAddNet: A Hardware-Inspired Deep NetworkPreprint
You, H., Chen, X., Zhang, Y., Li, C., Li, S., Liu, Z., Wang, Z., & Lin, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12785 · Cited by 57
Neural Architecture Search and Hardware Accelerator Co-Search: A Survey
Sekanina, L. (2021) · IEEE Access · doi:10.1109/access.2021.3126685 · Cited by 53
Optimally Scheduling CNN Convolutions for Efficient Memory AccessPreprint
Stoutchinin, A., Conti, F., & Benini, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.01492 · Cited by 52
Adaptive Gradient Quantization for Data-Parallel SGDPreprint
Faghri, F., Tabrizian, I., Markov, I., Alistarh, D., Roy, D. M., & Ramezani-Kebrya, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12460 · Cited by 52
Towards efficient vision transformer inference: a first study of transformers on mobile devices
Wang, X., Zhang, L. L., Wang, Y., & Yang, M. F. (2022) · IEEE Workshop on Mobile Computing Systems and Applications (WMCSA) · doi:10.1145/3508396.3512869 · Cited by 52
Compression of Generative Pre-trained Language Models via Quantization
Tao, C., Hou, L., Zhang, W., Shang, L., Jiang, X., Liu, Q., Luo, P., & Wong, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.331 · Cited by 52
DANCE: Differentiable Accelerator/Network Co-Exploration
Choi, K., Hong, D., Yoon, H., Yu, J., Kim, Y., & Lee, J. (2021) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac18074.2021.9586121 · Cited by 50
CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization
Hu, W., Che, Z., Liu, N., Li, M., Tang, J., Zhang, C., & Wang, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3262952 · Cited by 47
Adversarial co-distillation learning for image recognition
Zhang, H., Hu, Z., Qin, W., Xu, M., & Wang, M. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107659 · Cited by 44
Post-training Quantization with Multiple Points: Mixed Precision without Mixed Precision
Liu, X., Ye, M., Zhou, D., & Liu, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17054 · Cited by 44
Scale MLPerf-0.6 models on Google TPU-v3 PodsPreprint
Kumar, S., Bittorf, V., Chen, D., Chou, C., Hechtman, B. A., Lee, H., Kumar, N., Mattson, P. R., Wang, S., Wang, T., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09756 · Cited by 42
Semi-supervised Domain Adaptation via Sample-to-Sample Self-Distillation
Yoon, J., Kang, D., & Cho, M. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00175 · Cited by 42
Differentiable Learning-to-Group Channels via Groupable Convolutional Neural Networks
Zhang, Z., Li, J., Shao, W., Peng, Z., Zhang, R., Wang, X., & Luo, P. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00364 · Cited by 38
Searching for Fast Model Families on Datacenter Accelerators
Li, S., Tan, M., Pang, R., Li, A., Cheng, L., Le, Q. V., & Jouppi, N. P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00799 · Cited by 35
AddressNet: Shift-Based Primitives for Efficient Convolutional Neural Networks
He, Y., Liu, X., Zhong, H., & Ma, Y. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00134 · Cited by 33
Yang, T.-J., Liao, Y.-L., & Sze, V. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00243 · Cited by 33
FashionMirror: Co-attention Feature-remapping Virtual Try-on with Sequential Template Poses
Chen, C.-Y., Lo, L., Huang, P.-J., Shuai, H.-H., & Cheng, W.-H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01355 · Cited by 33
Self-citationVAQF: Fully Automatic Software-hardware Co-design Framework for Low-bit Vision Transformer
Sun, M., Ma, H., Kang, G., Jiang, Y., Chen, T., Ma, X., Wang, Z., & Wang, Y. (2022) · arXiv · doi:10.48550/arxiv.2201.06618 · Cited by 32
Least squares binary quantization of neural networks
Pouransari, H., Tu, Z., & Tuzel, O. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00357 · Cited by 31
Rethinking Co-design of Neural Architectures and Hardware AcceleratorsPreprint
Zhou, Y., Dong, X., Akin, B., Tan, M., Peng, D., Meng, T., Yazdanbakhsh, A., Huang, D., Narayanaswami, R., & Laudon, J. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08619 · Cited by 30
Wu, X., Li, C., Aminabadi, R. Y., Yao, Z., & He, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.12017 · Cited by 30
Li, N., Takaki, S., Tomiokay, Y., et al. (2016) · IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI) · doi:10.1109/ssiai.2016.7459201 · Cited by 26
ESSA: An energy-Aware bit-Serial streaming deep convolutional neural network accelerator
Hsu, L.-C., Chiu, C., Lin, K.-T., Chou, H.-H., & Pu, Y.-Y. (2020) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2020.101831 · Cited by 24
Reconfigurable Binary Neural Network Accelerator with Adaptive Parallelism Scheme
Cho, J., Jung, Y., Lee, S., & Jung, Y. (2021) · Electronics · doi:10.3390/electronics10030230 · Cited by 24
Pyramidal Recurrent Unit for Language Modeling
Mehta, S., Koncel-Kedziorski, R., Rastegari, M., & Hajishirzi, H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1491 · Cited by 18
Most Important Person-guided Dual-branch Cross-Patch Attention for Group Affect Recognition
Xie, H., Lee, M.-X., Chen, T.-J., Chen, H.-J., Liu, H.-I., Shuai, H.-H., & Cheng, W.-H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01883 · Cited by 18
Self-citationDesigning Deep Learning Hardware Accelerator and Efficiency Evaluation
Qi, Z., Chen, W., Naqvi, R. A., & Siddique, K. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/1291103 · Cited by 13
Um, S., Kim, S., Kim, S., & Yoo, H. (2021) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2021.3067327 · Cited by 11
DiVIT: Algorithm and architecture co-design of differential attention in vision transformer
Li, Y., Hu, Y., Wu, F., & Li, K. (2022) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2022.102520 · Cited by 11
Self-citationBoost Transformer-based Language Models with GPU-Friendly Sparsity and Quantization
Yu, C. H., Chen, T., & Gan, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.15 · Cited by 9
Fraccaroli, M., Lamma, E., & Riguzzi, F. (2021) · Machine Learning · doi:10.1007/s10994-021-06097-1 · Cited by 7
Constructing Fast Network through Deconstruction of Convolution
Jeon, Y., & Kim, J. (2018) · arXiv · doi:10.48550/arxiv.1806.07370
Cited by
Raza, S. M., Abidi, S. M. H., Masuduzzaman, M., & Shin, S. Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131357 · Cited by 13
Metadata
- Authors
- Liu, H.-I., Galindo, M., Xie, H., Wong, L.-K., Shuai, H.-H., Li, Y.-H., & Cheng, W.-H.
- Venue
- ACM Computing Surveys
- Published
- 2024-06-24
- Volume
- 56
- Issue
- 10
- Pages
- 1-42
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4396827044
- Cited by
- 259
- References
- 181
- FWCI
- 36.58
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.