CoChat
Sign Up
PreprintCited by 2Advanced Neural Network ApplicationsIoT and Edge/Fog ComputingDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.

On Accelerating Edge AI: Optimizing Resource-Constrained Environments

Open access
Sander, Jacob, Cohen, Achraf, Dasari, Venkat R., Venable, Brent, Jalaian, Brian·2025·arXiv
DOI 10.48550/arxiv.2501.15014arXiv 2501.15014
Open this paper in CoChat — free accountOpen PDF

Abstract

Resource-constrained edge deployments demand AI solutions that balance high performance with stringent compute, memory, and energy limitations. In this survey, we present a comprehensive overview of the primary strategies for accelerating deep learning models under such constraints. First, we examine model compression techniques-pruning, quantization, tensor decomposition, and knowledge distillation-that streamline large models into smaller, faster, and more efficient variants. Next, we explore Neural Architecture Search (NAS), a class of automated methods that discover architectures inherently optimized for particular tasks and hardware budgets. We then discuss compiler and deployment frameworks, such as TVM, TensorRT, and OpenVINO, which provide hardware-tailored optimizations at inference time. By integrating these three pillars into unified pipelines, practitioners can achieve multi-objective goals, including latency reduction, memory savings, and energy efficiency-all while maintaining competitive accuracy. We also highlight emerging frontiers in hierarchical NAS, neurosymbolic approaches, and advanced distillation tailored to large language models, underscoring open challenges like pre-training pruning for massive networks. Our survey offers practical insights, identifies current research gaps, and outlines promising directions for building scalable, platform-independent frameworks to accelerate deep learning models at the edge.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2501.15014

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 3 references

Research works printed but not linked: 134

  1. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018.
  2. Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models, 2024a. URL https://arxiv.org/abs/2407. 11062.
  3. Yanshu Wang, Tong Yang, Xiyan Liang, Guoan Wang, Hanning Lu, Xu Zhe, Yaoming Li, and Li Weitao. Art and science of quantizing large-scale models: A comprehensive overview. arXiv preprint arXiv:2409.11650, 2024. · arXiv
  4. Sean I Young. Foundations of large language model compression-part 1: Weight quantization. arXiv preprint arXiv:2409.02026, 2024. · arXiv
  5. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, July 2024a. URL http://arxiv.org/abs/2306.00978. arXiv:2306.00978 [cs. · arXiv
  6. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URL https://arxiv.org/abs/2305.14314. · arXiv
  7. Yijia Zhang, Lingran Zhao, Shijie Cao, Sicheng Zhang, Wenqiang Wang, Ting Cao, Fan Yang, Mao Yang, Shang- hang Zhang, and Ningyi Xu. Integer or floating point? new outlooks for low-bit quantization on large language models. In 2024 IEEE International Conference on Multimedia and Expo (ICME), page 1-6. IEEE, July 2024. doi:10.1109/icme57554.2024.10688089. URL http://dx.doi.org/10.1109/ICME57554.2024.10688089. · DOI
  8. Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models, 2023. URL https://arxiv.org/abs/2304.01089. · arXiv
  9. Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, 2023. URL https://arxiv.org/abs/2304.09145. · arXiv
  10. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Frederick Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. ArXiv, abs/1710.03740, 2017. URL https://api.semanticscholar.org/CorpusID:3297437. · arXiv
  11. Hao Li, Yuzhu Wang, Yan Hong, Fei Li, and Xiaohui Ji. Layered mixed-precision training: A new training method for large-scale ai models. Journal of King Saud University -Computer and Information Sciences, 35(8):101656, September 2023a. ISSN 1319-1578. doi:10.1016/j.jksuci.2023.101656. URL http://dx.doi.org/10.1016/j. jksuci.2023.101656. · DOI
  12. Zihan Chen, Bike Xie, Jundong Li, and Cong Shen. Channel-wise mixed-precision quantization for large language models, 2024b. URL https://arxiv.org/abs/2410.13056. · arXiv
  13. Junhao Xu, Jianwei Yu, Shoukang Hu, Xunying Liu, and Helen Meng. Mixed precision low-bit quantization of neural network language models for speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3679-3693, 2021. ISSN 2329-9304. doi:10.1109/taslp.2021.3129357. URL http://dx.doi.org/ 10.1109/TASLP.2021.3129357. · DOI
  14. Gaurav Menghani. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Comput. Surv., 55(12), March 2023. ISSN 0360-0300. doi:10.1145/3578938. URL https://doi.org/10.1145/ 3578938. · DOI
  15. Jiedong Lang, Zhehao Guo, and Shuyu Huang. A comprehensive study on quantization techniques for large language models, 2024. URL https://arxiv.org/abs/2411.02530. · arXiv
  16. Anna Choromanska, Mikael Henaff, Michaël Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surface of multilayer networks. CoRR, abs/1412.0233, 2014. URL http://arxiv.org/abs/1412.0233. · arXiv
  17. Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: Single-shot network pruning based on connection sensitivity, 2019. URL https://arxiv.org/abs/1810.02340. · arXiv
  18. Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow, 2020a. URL https://arxiv.org/abs/2002.07376. · arXiv
  19. Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023. URL https://arxiv.org/abs/2301.00774. · arXiv
  20. Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks, 2017. URL https://arxiv.org/abs/1707.06168. · arXiv
  21. Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems, volume 32.
  22. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/ 2019/hash/b51a15f382ac914391a58850ab343b00-Abstract.html.
  23. Kyuhong Shim, Iksoo Choi, Wonyong Sung, and Jungwook Choi. Layer-wise pruning of transformer attention heads for efficient language modeling. In 2021 18th International SoC Design Conference (ISOCC), pages 357-358, 2021. doi:10.1109/ISOCC53507.2021.9613933. · DOI
  24. Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns, 2024. URL https://arxiv.org/abs/2401. 15024.
  25. Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024. URL https://arxiv.org/ abs/2403.03853. · arXiv
  26. Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models, 2023. URL https://arxiv.org/abs/2305.11627. · arXiv
  27. Kaixin Xu, Zhe Wang, Chunyun Chen, Xue Geng, Jie Lin, Xulei Yang, Min Wu, Xiaoli Li, and Weisi Lin. Lpvit: Low-power semi-structured pruning for vision transformers, 2024a. URL https://arxiv.org/abs/2407.02068. · arXiv
  28. Xiaolong Ma, Wei Niu, Tianyun Zhang, Sijia Liu, Sheng Lin, Hongjia Li, Xiang Chen, Jian Tang, Kaisheng Ma, Bin Ren, and Yanzhi Wang. An image enhancing pattern-based sparsity for real-time inference on mobile devices, 2020. URL https://arxiv.org/abs/2001.07710. · arXiv
  29. Fanxu Meng, Hao Cheng, Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruning filter in filter, 2020. URL https://arxiv.org/abs/2009.14410. · arXiv
  30. Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow, 2020. URL https://arxiv.org/abs/2006.05467. · arXiv
  31. Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip H. S. Torr. A signal propagation perspective for pruning neural networks at initialization, 2020. URL https://arxiv.org/abs/1906.06307. · arXiv
  32. Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners, 2021. URL https://arxiv.org/abs/1911.11134. · arXiv
  33. Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks, 2016. URL https://arxiv.org/abs/1608.03665. · arXiv
  34. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming, 2017. URL https://arxiv.org/abs/1708.06519. · arXiv
  35. Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. Dsa: More efficient budgeted pruning via differentiable sparsity allocation, 2020. URL https://arxiv.org/abs/2004.02164. · arXiv
  36. Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, and Song Han. APQ: Joint Search for Network Architecture, Pruning and Quantization Policy, June 2020b. URL http://arxiv.org/abs/2006.08509. arXiv:2006.08509 [cs, stat]. · arXiv
  37. Woosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers, 2022. URL https://arxiv.org/abs/2204.09656. · arXiv
  38. Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression, 2021. URL https://arxiv.org/abs/2108.00708. · arXiv
  39. Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks, 2022. URL https://arxiv.org/abs/1909.11957. · arXiv
  40. Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets, 2021. URL https://arxiv.org/abs/2101.00063. · arXiv
  41. Yongming Rao, Jiwen Lu, Ji Lin, and Jie Zhou. Runtime network routing for efficient image classification. IEEE Trans. Pattern Anal. Mach. Intell., 41(10):2291-2304, oct 2019. ISSN 0162-8828. doi:10.1109/TPAMI.2018.2878258. URL https://doi.org/10.1109/TPAMI.2018.2878258. · DOI
  42. Yehui Tang, Yunhe Wang, Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning, 2021. URL https://arxiv.org/abs/2103.05861. · arXiv
  43. Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2016. URL https://arxiv.org/abs/1510.00149. · arXiv
  44. Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets, 2017. URL https://arxiv.org/abs/1608.08710. · arXiv
  45. Chenglong Zhao, Bingbing Ni, Jian Zhang, Qiwei Zhao, Wenjun Zhang, and Qi Tian. Variational convolutional neural network pruning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2775-2784, 2019. doi:10.1109/CVPR.2019.00289. · DOI
  46. Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li. What matters in the structured pruning of generative language models?, 2023. URL https://arxiv.org/abs/2302.03773. · arXiv
  47. Lucio Dery, Steven Kolawole, Jean-François Kagy, Virginia Smith, Graham Neubig, and Ameet Talwalkar. Everybody prune now: Structured pruning of llms with only forward passes, 2024. URL https://arxiv.org/abs/2402. 05406.
  48. Mingyang Zhang, Xinyi Yu, Jingtao Rong, and Linlin Ou. Graph pruning for model compression, 2021. URL https://arxiv.org/abs/1911.09817. · arXiv
  49. Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: AutoML for Model Compression and Acceleration on Mobile Devices, page 815-832. Springer International Publishing, 2018. ISBN 9783030012342. doi:10.1007/978-3-030-01234-2_48. URL http://dx.doi.org/10.1007/978-3-030-01234-2_48. · DOI · Paper page
  50. Blanka Bencsik and Márton Szemenyei. Efficient neural network pruning using model-based reinforcement learn- ing. In 2022 International Symposium on Measurement and Control in Robotics (ISMCR), pages 1-8, 2022. doi:10.1109/ISMCR56534.2022.9950598. · DOI
  51. Zhuo Li, Hengyi Li, and Lin Meng. Model compression for deep neural networks: A survey. Computers, 12(3), 2023b. ISSN 2073-431X. doi:10.3390/computers12030060. URL https://www.mdpi.com/2073-431X/12/3/60. · DOI
  52. Habib Hajimolahoseini, Mehdi Rezagholizadeh, Vahid Partovinia, Marzieh Tahaei, Omar Mohamed Awad, and Yang Liu. Compressing pre-trained language models using progressive low rank decomposition. Advances in Neural Information Processing Systems, 2021.
  53. Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J. Goldsmith, and Mert Pilanci. Compressing large language models using low rank and low precision decomposition, 2024. URL https://arxiv.org/abs/2405.18886. · arXiv
  54. Mingxue Xu, Yao Lei Xu, and Danilo P. Mandic. Tensorgpt: Efficient compression of large language models based on tensor-train decomposition, 2023a. URL https://arxiv.org/abs/2307.00526. · arXiv
  55. Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular decomposition for large language model compression, 2024b. URL https: //arxiv.org/abs/2408.09632.
  56. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URL https: //arxiv.org/abs/1503.02531.
  57. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A Survey on Knowledge Distillation of Large Language Models, March 2024b. URL http://arxiv.org/abs/ 2402.13116. arXiv:2402.13116 [cs]. · arXiv
  58. Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. Proceed- ings of the AAAI Conference on Artificial Intelligence, 34(04):3996-4003, April 2020. ISSN 2159-5399. doi:10.1609/aaai.v34i04.5816. URL http://dx.doi.org/10.1609/aaai.v34i04.5816. · DOI
  59. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https://arxiv.org/abs/1406.2661. · arXiv
  60. Jaewon Jung, Hongsun Jang, Jaeyong Song, and Jinho Lee. Peeraid: Improving adversarial distillation from a specialized peer tutor, 2024. URL https://arxiv.org/abs/2403.06668. · arXiv
  61. Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks, 2019a. URL https://arxiv.org/abs/1904.01186. · arXiv
  62. Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning, 2017. URL https: //arxiv.org/abs/1706.00384.
  63. Meet P. Vadera, Brian Jalaian, and Benjamin M. Marlin. Generalized bayesian posterior expectation distillation for deep neural networks. CoRR, abs/2005.08110, 2020. URL https://arxiv.org/abs/2005.08110. · arXiv
  64. Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer, 2015. URL https://arxiv.org/abs/1507.00448. · arXiv
  65. Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progressive self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16430-16441, 2022.
  66. Fushuo Huo, Wenchao Xu, Jingcai Guo, Haozhao Wang, and Song Guo. C2kd: Bridging the modality gap for cross-modal knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16006-16015, 2024.
  67. Jing Liu, Tongya Zheng, Guanzheng Zhang, and Qinfen Hao. Graph-based knowledge distillation: A survey and experimental evaluation, 2023a. URL https://arxiv.org/abs/2302.14643. · arXiv
  68. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762. · arXiv
  69. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distilla- tion for task-agnostic compression of pre-trained transformers. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5776-5788.
  70. Curran Associates, Inc., 2020c. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  71. Kai Wang, Fei Yang, and Joost van de Weijer. Attention distillation: self-supervised vision transformer students need more guidance, 2022a. URL https://arxiv.org/abs/2210.00944. · arXiv
  72. Elliot J. Crowley, Gavin Gray, and Amos Storkey. Moonshine: Distilling with cheap convolutions, 2019. URL https://arxiv.org/abs/1711.02613. · arXiv
  73. Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789-1819, March 2021. ISSN 1573-1405. doi:10.1007/s11263-021-01453-z. URL http://dx.doi.org/10.1007/s11263-021-01453-z. · DOI · Paper page
  74. Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. CoRR, abs/1912.00350, 2019b. URL http://arxiv.org/abs/1912.00350. · arXiv
  75. Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3351-3361.
  76. Curran Associates, Inc., 2020. URL https://proceedings.neurips. cc/paper_files/paper/2020/file/2288f691b58edecadcc9a8691762b4fd-Paper.pdf.
  77. Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. CoRR, abs/1905.08094, 2019. URL http://arxiv.org/abs/1905.08094. · arXiv
  78. Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning. ArXiv, abs/2402.10110, 2024. URL https://api. semanticscholar.org/CorpusID:267682220. · arXiv
  79. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023. URL https://arxiv.org/abs/2306. 02707.
  80. Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data, 2023b. URL https://arxiv.org/abs/2304.01196. · arXiv
  81. Minki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks, 2023. URL https://arxiv.org/abs/2305. 18395.
  82. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. URL https://arxiv.org/abs/2310.11511. · arXiv
  83. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685. · arXiv
  84. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024. URL https://arxiv.org/abs/2404.04475. · arXiv
  85. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment, 2023. URL https://arxiv.org/abs/2310.16944. · arXiv
  86. Yuntao Bai et. al. Constitutional ai: Harmlessness from ai feedback, 2022. URL https://arxiv.org/abs/2212. 08073.
  87. Maryam Hashemzadeh, Elias Stengel-Eskin, Sarath Chandar, and Marc-Alexandre Cote. Sub-goal distillation: A method to improve small language agents, 2024. URL https://arxiv.org/abs/2405.02749. · arXiv
  88. Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents, 2024a. URL https://arxiv.org/ abs/2311.05657. · arXiv
  89. Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning, 2023. URL https://arxiv.org/abs/2310.05915. · arXiv
  90. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models, 2024. URL https://arxiv.org/abs/2306.08543. · arXiv
  91. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models, 2024. URL https://arxiv.org/abs/2402.03898. · arXiv
  92. Krishna Srinivasan, Karthik Raman, Anupam Samanta, Lingrui Liao, Luca Bertelli, and Mike Bendersky. Quill: Query intent with large language models using retrieval augmentation and multi-stage distillation, 2022. URL https://arxiv.org/abs/2210.15718. · arXiv
  93. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b. URL https: //arxiv.org/abs/2304.08485.
  94. Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration, 2023. URL https://arxiv.org/abs/2306.09093. · arXiv
  95. Lu Yin, AJAY KUMAR JAISWAL, Shiwei Liu, Souvik Kundu, and Zhangyang Wang. Junk DNA hypothesis: Pruning small pre-trained weights $\textit{Irreversibly}$ and $\textit{Monotonically}$ impairs "difficult" downstream tasks in LLMs. In Forty-first International Conference on Machine Learning, 2024b. URL https://openreview.net/ forum?id=EfUrTeuUfy.
  96. Xin He, Kaiyong Zhao, and Xiaowen Chu. AutoML: A survey of the state-of-the-art. Knowledge-Based Systems, 212: 106622, January 2021. ISSN 0950-7051. doi:10.1016/j.knosys.2020.106622. URL https://www.sciencedirect. com/science/article/pii/S0950705120307516. · DOI
  97. Barret Zoph and Quoc V. Le. Neural Architecture Search with Reinforcement Learning, February 2017. URL http://arxiv.org/abs/1611.01578. arXiv:1611.01578 [cs. · arXiv
  98. Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient Architecture Search by Network Transfor- mation, November 2017. URL http://arxiv.org/abs/1707.04873. arXiv:1707.04873 [cs. · arXiv
  99. Colin White, Willie Neiswanger, Sam Nolen, and Yash Savani. A study on encodings for neural architecture search. CoRR, abs/2007.04965, 2020. URL https://arxiv.org/abs/2007.04965. · arXiv
  100. Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers, 2023. URL https://arxiv.org/abs/2301.08727. · arXiv
  101. Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. CoRR, abs/1806.09055, 2018. URL http://arxiv.org/abs/1806.09055. · arXiv
  102. Charles Jin, Phitchaya Mangpo Phothilimthana, and Sudip Roy. Neural architecture search using property guided synthesis. Proceedings of the ACM on Programming Languages, 6(OOPSLA2):1150-1179, October 2022. ISSN 2475-1421. doi:10.1145/3563329. URL http://dx.doi.org/10.1145/3563329. · DOI
  103. Xingchen Wan, Binxin Ru, Pedro M. Esperança, and Fabio Maria Carlucci. Approximate neural architecture search via operation distribution learning. CoRR, abs/2111.04670, 2021. URL https://arxiv.org/abs/2111.04670. · arXiv
  104. Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-All: Train One Network and Specialize it for Efficient Deployment, April 2020. URL http://arxiv.org/abs/1908.09791. arXiv:1908.09791 [cs, stat]. · arXiv
  105. Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Zhengsu Chen, Lanfei Wang, An Xiao, Jianlong Chang, Xiaopeng Zhang, and Qi Tian. Weight-sharing neural architecture search: A battle to shrink the optimization gap. CoRR, abs/2008.01475, 2020. URL https://arxiv.org/abs/2008.01475. · arXiv
  106. Yawei Li, Shuhang Gu, Kai Zhang, Luc Van Gool, and Radu Timofte. Dhp: Differentiable meta pruning via hypernetworks, 2020. URL https://arxiv.org/abs/2003.13683. · arXiv
  107. Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Tim Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning, 2019. URL https://arxiv.org/abs/1903.10258. · arXiv
  108. Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. CoRR, abs/1810.05749, 2018. URL http://arxiv.org/abs/1810.05749. · arXiv
  109. Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-Level Network Transformation for Efficient Architecture Search, June 2018. URL http://arxiv.org/abs/1806.02639. arXiv:1806.02639 [cs, stat]. · arXiv
  110. Xiaofan Zhang, Zongwei Zhou, Deming Chen, and Yu Emma Wang. Autodistill: an end-to-end framework to explore and distill hardware-efficient language models. CoRR, abs/2201.08539, 2022. URL https://arxiv.org/abs/ 2201.08539. · arXiv
  111. Colin White, Arber Zela, Robin Ru, Yang Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 28454-28469.
  112. Sheng Li, Mingxing Tan, Ruoming Pang, Andrew Li, Liqun Cheng, Quoc V. Le, and Norman P. Jouppi. Searching for fast model families on datacenter accelerators. CoRR, abs/2102.05610, 2021. URL https://arxiv.org/abs/ 2102.05610. · arXiv
  113. Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile. CoRR, abs/1807.11626, 2018. URL http://arxiv.org/abs/1807.11626. · arXiv
  114. Yuxiao Zhou, Zhishan Guo, Zheng Dong, and Kecheng Yang. Multi-accelerator neural network inference via tensorrt in heterogeneous embedded systems. In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), pages 463-472, 2024. doi:10.1109/COMPSAC61105.2024.00070. · DOI
  115. Yongtao Wu, Fanghui Liu, Carl-Johann Simon-Gabriel, Grigorios G Chrysos, and Volkan Cevher. Robust nas under adversarial training: benchmark, theory, and beyond, 2024. URL https://arxiv.org/abs/2403.13134. · arXiv
  116. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121, 2024.
  117. Alex A. Freitas. The case for hybrid multi-objective optimisation in high-stakes machine learning applications. SIGKDD Explor. Newsl., 26(1):24-33, jul 2024. ISSN 1931-0145. doi:10.1145/3682112.3682116. URL https: //doi.org/10.1145/3682112.3682116. · DOI
  118. Jacob Abernethy, Robert E. Schapire, and Umar Syed. Lexicographic optimization: Algorithms and stability, 2024. URL https://arxiv.org/abs/2405.01387. · arXiv
  119. Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing, 2018. URL https://arxiv.org/abs/1802.03268. · arXiv
  120. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. CoRR, abs/1802.01548, 2018. URL http://arxiv.org/abs/1802.01548. · arXiv
  121. Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka I. Leon-Suematsu, Quoc V. Le, and Alex Kurakin. Large-scale evolution of image classifiers. CoRR, abs/1703.01041, 2017. URL http://arxiv.org/abs/1703. 01041. · arXiv
  122. Jean-Antoine Désidéri. Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathematique, 350(5):313-318, 2012. ISSN 1631-073X. doi:https://doi.org/10.1016/j.crma.2012.03.014. URL https://www.sciencedirect.com/science/article/pii/S1631073X12000738. · DOI
  123. Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Samuel Dooley, Josif Grabocka, and Frank Hutter. Multi- objective differentiable neural architecture search, 2024. URL https://arxiv.org/abs/2402.18213. · arXiv
  124. Colin White, Willie Neiswanger, and Yash Savani. BANANAS: bayesian optimization with neural architectures for neural architecture search. CoRR, abs/1910.11858, 2019. URL http://arxiv.org/abs/1910.11858. · arXiv
  125. Gresa Shala, Thomas Elsken, Frank Hutter, and Josif Grabocka. Transfer NAS with meta-learned bayesian surrogates. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/ forum?id=paGvsrl4Ntr.
  126. Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. Tvm: An automated end-to-end optimizing compiler for deep learning. arXiv preprint arXiv:1802.04799, 2018. · arXiv
  127. Yuxiao Zhou and Kecheng Yang. Exploring tensorrt to improve real-time inference for deep learning. In 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Sys- tems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), pages 2011-2018, 2022. doi:10.1109/HPCC-DSS-SmartCity- DependSys57074.2022.00299. · DOI · Paper page
  128. Andrew S. Cassidy, John V. Arthur, Filipp Akopyan, Alexander Andreopoulos, Rathinakumar Appuswamy, Pallab Datta, Michael V. Debole, Steven K. Esser, Carlos Ortega Otero, Jun Sawada, Brian Taba, Arnon Amir, Deepika Bablani, Peter J. Carlson, Myron D. Flickner, Rajamohan Gandhasri, Guillaume J. Garreau, Megumi Ito, Jennifer L. Klamo, Jeffrey A. Kusnitz, Nathaniel J. McClatchey, Jeffrey L. McKinstry, Yutaka Nakamura, Tapan K. Nayak, William P. Risk, Kai Schleupen, Ben Shaw, Jay Sivagnaname, Daniel F. Smith, Ignacio Terrizzano, Takanori Ueda, and Dharmendra Modha. 11.4 ibm northpole: An architecture for neural network inference with a 12nm chip. In 2024 IEEE International Solid-State Circuits Conference (ISSCC), volume 67, pages 214-215, 2024. doi:10.1109/ISSCC49657.2024.10454451. · DOI
  129. Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. CoRR, abs/1902.09635, 2019. URL http://arxiv.org/abs/1902. 09635. · arXiv
  130. Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. CoRR, abs/2001.00326, 2020. URL http://arxiv.org/abs/2001.00326. · arXiv
  131. Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.09777, 4:14, 2020. · arXiv
  132. Yash Mehta, Colin White, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, and Frank Hutter. Nas-bench-suite: NAS evaluation is (now) surprisingly easy. CoRR, abs/2201.13396, 2022. URL https://arxiv.org/abs/2201.13396. · arXiv
  133. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices, 2020. URL https://arxiv.org/abs/2004.02984. · arXiv
  134. Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, page 1487-1495, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874. doi:10.1145/3097983.3098043. URL https://doi.org/10.1145/ 3097983.3098043. · DOI

Cited by

Sorted by citations

See all 2

Metadata

Authors
Sander, J., Cohen, A., Dasari, V. R., Venable, B., & Jalaian, B.
Affiliations
  • University of West Florida (US)
  • DEVCOM Army Research Laboratory (US)
  • Florida Institute for Human and Machine Cognition (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2501.15014
arXiv
2501.15014
OpenAlex
W4406879999
Cited by
2
References
3
Citations by year
2025: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • U.S. Army Combat Capabilities Development Command
  • Army Research Laboratory
  • DEVCOM Army Research Laboratory
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
edge AI, deep learning acceleration, resource-constrained deployment, model compression, neural network pruning, quantization, tensor decomposition, knowledge distillation, neural architecture search, neuro-symbolic AI, inference optimization, TVM, TensorRT, OpenVINO, energy efficiency, latency reduction, memory efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: On Accelerating Edge AI: Optimizing Resource-Constrained Environments. https://app.cochat.ai/papers/10.48550/arxiv.2501.15014
The paper
Sander, J., Cohen, A., Dasari, V. R., Venable, B., & Jalaian, B. (2025). On Accelerating Edge AI: Optimizing Resource-Constrained Environments. arXiv. https://doi.org/10.48550/arxiv.2501.15014