Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.

FLASH: F ast Neura l A rchitecture S earch with H ardware Optimization

Li, Guihong, Mandal, Sumit K., Ogras, Umit Y., Marculescu, Radu · 2021 · ACM Transactions on Embedded Computing Systems

Abstract

Neural architecture search (NAS) is a promising technique to design efficient and high-performance deep neural networks (DNNs). As the performance requirements of ML applications grow continuously, the hardware accelerators start playing a central role in DNN design. This trend makes NAS even more complicated and time-consuming for most real applications. This paper proposes FLASH, a very fast NAS methodology that co-optimizes the DNN accuracy and performance on a real hardware platform. As the main theoretical contribution, we first propose the NN-Degree, an analytical metric to quantify the topological characteristics of DNNs with skip connections (e.g., DenseNets, ResNets, Wide-ResNets, and MobileNets). The newly proposed NN-Degree allows us to do training-free NAS within one second and build an accuracy predictor by training as few as 25 samples out of a vast search space with more than 63 billion configurations. Second, by performing inference on the target hardware, we fine-tune and validate our analytical models to estimate the latency, area, and energy consumption of various DNN architectures while executing standard ML datasets. Third, we construct a hierarchical algorithm based on simplicial homology global optimization (SHGO) to optimize the model-architecture co-design process, while considering the area, latency, and energy consumption of the target hardware. We demonstrate that, compared to the state-of-the-art NAS approaches, our proposed hierarchical SHGO-based algorithm enables more than four orders of magnitude speedup (specifically, the execution time of the proposed algorithm is about 0.1 seconds). Finally, our experimental evaluations show that FLASH is easily transferable to different hardware architectures, thus enabling us to do NAS on a Raspberry Pi-3B processor in less than 3 seconds.

Overview

Integrity signals

Registry

Crossref · 10.1145/3476994

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 32

  1. Mohamed S. Abdelfattah , Abhinav Mehrotra , Łukasz Dudziak , and Nicholas Donald Lane . 2021 . Zero-cost proxies for lightweight NAS . In International Conference on Learning Representations. Mohamed S. Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas Donald Lane. 2021. Zero-cost proxies for lightweight NAS. In International Conference on Learning Representations.
  2. Bowen Baker , Otkrist Gupta , Nikhil Naik , and Ramesh Raskar . 2016. Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167 ( 2016 ). Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. 2016. Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167 (2016). · arXiv
  3. Hadjer Benmeziane et al . 2021. A comprehensive survey on hardware-aware neural architecture search. arXiv preprint arXiv:2101.09336 ( 2021 ). Hadjer Benmeziane et al. 2021. A comprehensive survey on hardware-aware neural architecture search. arXiv preprint arXiv:2101.09336 (2021). · arXiv
  4. Han Cai , Ligeng Zhu , and Song Han . 2019 . ProxylessNAS: Direct neural architecture search on target task and hardware . In International Conference on Learning Representations. Han Cai, Ligeng Zhu, and Song Han. 2019. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations.
  5. Thomas Chau , Łukasz Dudziak , Mohamed S. Abdelfattah , Royson Lee , Hyeji Kim , and Nicholas D . Lane . 2020 . BRP-NAS: Prediction-based NAS using GCNs . arXiv preprint arXiv:2007.08668 (2020). Thomas Chau, Łukasz Dudziak, Mohamed S. Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D. Lane. 2020. BRP-NAS: Prediction-based NAS using GCNs. arXiv preprint arXiv:2007.08668 (2020). · arXiv
  6. Wuyang Chen , Xinyu Gong , and Zhangyang Wang . 2021 . Neural architecture search on ImageNet in four GPU hours: A theoretically inspired perspective . In International Conference on Learning Representations. Wuyang Chen, Xinyu Gong, and Zhangyang Wang. 2021. Neural architecture search on ImageNet in four GPU hours: A theoretically inspired perspective. In International Conference on Learning Representations.
  7. Matthieu Courbariaux , Itay Hubara , Daniel Soudry , Ran El-Yaniv , and Yoshua Bengio . 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 ( 2016 ). Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 (2016). · arXiv
  8. Xuanyi Dong and Yi Yang . 2020. NAS-Bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326 ( 2020 ). Xuanyi Dong and Yi Yang. 2020. NAS-Bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326 (2020). · arXiv
  9. Boris Grot and Stephen W. Keckler . 2008. Scalable on-chip interconnect topologies . In 2nd Workshop on Chip Multiprocessor Memory Systems and Interconnects. Boris Grot and Stephen W. Keckler. 2008. Scalable on-chip interconnect topologies. In 2nd Workshop on Chip Multiprocessor Memory Systems and Interconnects.
  10. Song Han , Huizi Mao , and William J . Dally . 2015 . Deep compression: Compressing de ep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015). Song Han, Huizi Mao, and William J. Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015). · arXiv
  11. Geoffrey Hinton , Oriol Vinyals , and Jeff Dean . 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 ( 2015 ). Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015). · arXiv
  12. Chi-Hung Hsu et al . 2018 . Monas: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806.10332 (2018). Chi-Hung Hsu et al. 2018. Monas: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806.10332 (2018). · arXiv
  13. Gokul Krishnan et al . 2021 . Interconnect-centric benchmarking of in-memory acceleration for DNNS . In 2021 China Semiconductor Technology International Conference (CSTIC). IEEE, 1–4. Gokul Krishnan et al. 2021. Interconnect-centric benchmarking of in-memory acceleration for DNNS. In 2021 China Semiconductor Technology International Conference (CSTIC). IEEE, 1–4.
  14. 10.5555/3437539.3437669 · doi:10.5555/3437539.3437669
  15. Hanxiao Liu , Karen Simonyan , and Yiming Yang . 2018 . Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018). Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018). · arXiv
  16. Jovita Lukasik , David Friede , Heiner Stuckenschmidt , and Margret Keuper . 2020. Neural architecture performance prediction using graph neural networks. arXiv preprint arXiv:2010.10024 ( 2020 ). Jovita Lukasik, David Friede, Heiner Stuckenschmidt, and Margret Keuper. 2020. Neural architecture performance prediction using graph neural networks. arXiv preprint arXiv:2010.10024 (2020). · arXiv
  17. 10.5555/311445 · doi:10.5555/311445
  18. Volodymyr Mnih et al . 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 ( 2013 ). Volodymyr Mnih et al. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013). · arXiv
  19. 10.5555/1137830 · doi:10.5555/1137830
  20. Hieu Pham , Melody Guan , Barret Zoph , Quoc Le , and Jeff Dean . 2018 . Efficient neural architecture search via parameters sharing . In International Conference on Machine Learning. PMLR, 4095–4104 . Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. 2018. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning. PMLR, 4095–4104.
  21. 10.5555/3305890.3305981 · doi:10.5555/3305890.3305981
  22. Karen Simonyan and Andrew Zisserman . 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 ( 2014 ). Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). · arXiv
  23. Dimitrios Stamoulis et al . 2019. Single-path NAS: Designing hardware-efficient convnets in less than 4 hours. arXiv preprint arXiv:1904.02877 ( 2019 ). Dimitrios Stamoulis et al. 2019. Single-path NAS: Designing hardware-efficient convnets in less than 4 hours. arXiv preprint arXiv:1904.02877 (2019). · arXiv
  24. Mingxing Tan et al . 2019 . Mnasnet: Platform-aware neural architecture search for mobile . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2820–2828 . Mingxing Tan et al. 2019. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2820–2828.
  25. 10.5555/3157096.3157158 · doi:10.5555/3157096.3157158
  26. Duncan J. Watts and Steven H . Strogatz . 1998 . Collective dynamics of ‘small-world’ networks. Nature 393, 6684 (1998), 440–442. Duncan J. Watts and Steven H. Strogatz. 1998. Collective dynamics of ‘small-world’ networks. Nature 393, 6684 (1998), 440–442.
  27. Bichen Wu et al . 2019 . Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10734–10742 . Bichen Wu et al. 2019. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10734–10742.
  28. Sirui Xie , Hehui Zheng , Chunxiao Liu , and Liang Lin . 2019 . SNAS: Stochastic neural architecture search . In International Conference on Learning Representations. Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. 2019. SNAS: Stochastic neural architecture search. In International Conference on Learning Representations.
  29. Jiahui Yu et al . 2020. BigNAS: Scaling up neural architecture search with big single-stage models . In Computer Vision – ECCV 2020 . 702–717. Jiahui Yu et al. 2020. BigNAS: Scaling up neural architecture search with big single-stage models. In Computer Vision – ECCV 2020. 702–717.
  30. Sergey Zagoruyko and Nikos Komodakis . 2016. Wide residual networks. arXiv preprint arXiv:1605.07146 ( 2016 ). Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. arXiv preprint arXiv:1605.07146 (2016). · arXiv
  31. Barret Zoph and Quoc V . Le . 2016 . Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016). Barret Zoph and Quoc V. Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016). · arXiv
  32. Barret Zoph , Vijay Vasudevan , Jonathon Shlens , and Quoc V. Le . 2018. Learning transferable architectures for scalable image recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 8697–8710 . Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 8697–8710.

Cited by

See all 15

Metadata

Authors
Li, G., Mandal, S. K., Ogras, U. Y., & Marculescu, R.
Affiliations
  • The University of Texas at Austin (US)
  • University of Wisconsin–Madison (US)
  • The University of Texas at Austin, Austin, Texas, USA
Venue
ACM Transactions on Embedded Computing Systems
Published
2021-09-17
Volume
20
Issue
5s
Pages
1-26
ISSN
  • 1539-9087 (ISSN-L)
  • 1558-3465
OpenAlex
W3200067639
Cited by
15
References
24
Citations by year
2026: 1 · 2025: 2 · 2024: 4 · 2023: 5 · 2022: 2 · 2021: 1
FWCI
1.08
Type
journal article
Language
en
Open access
closed
Locations
ACM Transactions on Embedded Computing Systems (publishedVersion)
Funders
  • US National Science Foundation (CNS-2007284)
  • Semiconductor Research Corporation (GRC 2939.001 and 3012.001)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
Keywords
neural architecture search, hardware-aware neural network design, load forecasting, hierarchical optimization, skip connections, Raspberry Pi

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.