The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
FLASH: F ast Neura l A rchitecture S earch with H ardware Optimization
Abstract
Neural architecture search (NAS) is a promising technique to design efficient and high-performance deep neural networks (DNNs). As the performance requirements of ML applications grow continuously, the hardware accelerators start playing a central role in DNN design. This trend makes NAS even more complicated and time-consuming for most real applications. This paper proposes FLASH, a very fast NAS methodology that co-optimizes the DNN accuracy and performance on a real hardware platform. As the main theoretical contribution, we first propose the NN-Degree, an analytical metric to quantify the topological characteristics of DNNs with skip connections (e.g., DenseNets, ResNets, Wide-ResNets, and MobileNets). The newly proposed NN-Degree allows us to do training-free NAS within one second and build an accuracy predictor by training as few as 25 samples out of a vast search space with more than 63 billion configurations. Second, by performing inference on the target hardware, we fine-tune and validate our analytical models to estimate the latency, area, and energy consumption of various DNN architectures while executing standard ML datasets. Third, we construct a hierarchical algorithm based on simplicial homology global optimization (SHGO) to optimize the model-architecture co-design process, while considering the area, latency, and energy consumption of the target hardware. We demonstrate that, compared to the state-of-the-art NAS approaches, our proposed hierarchical SHGO-based algorithm enables more than four orders of magnitude speedup (specifically, the execution time of the proposed algorithm is about 0.1 seconds). Finally, our experimental evaluations show that FLASH is easily transferable to different hardware architectures, thus enabling us to do NAS on a Raspberry Pi-3B processor in less than 3 seconds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such optimization lies within the domain of hardware-aware neural architecture search, which optimizes the design parameters of the neural network considering hardware feedback261–264, or in some cases, searches for the optimum hardware parameters265,266.”
From Hardware implementation of memristor-based artificial neural networks (2024) · cites this paper as [262]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1993.341010 · Cited by 14,265
Barabási, A.-L., & Bonabeau, E. (2003) · Scientific American · doi:10.1038/scientificamerican0503-60 · Cited by 2,494
1.1 Computing's energy problem (and what we can do about it)
Horowitz, M. (2014) · 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) · doi:10.1109/isscc.2014.6757323 · Cited by 2,069
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
ISAAC: a convolutional neural network accelerator with in-situ analog arithmetic in crossbars
Shafiee, A., Nag, A., Muralimanohar, N., Balasubramonian, R., Strachan, J. P., Hu, M., Williams, R. S., & Srikumar, V. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001139 · Cited by 1,536
Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
Chiang, W.-L., Liu, X., Si, S., Li, Y., Bengio, S., & Hsieh, C.-J. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330925 · Cited by 1,331
Once-for-All: Train One Network and Specialize it for Efficient Deployment
Cai, H., Gan, C., Wang, T., Zhang, Z., & Han, S. (2019) · arXiv · doi:10.48550/arxiv.1908.09791 · Cited by 845 · cited 2×
“Several optimization techniques have been proposed to fill the accuracy gap between the shared weights and stand-alone training [7, 58].”
BackgroundPipeLayer: A Pipelined ReRAM-Based Accelerator for Deep Learning
Song, L., Qian, X., Li, H., & Chen, Y. (2017) · 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2017.55 · Cited by 833
A detailed and flexible cycle-accurate Network-on-Chip simulator
Jiang, N., Balfour, J., Becker, D. U., Towles, B., Dally, W. J., Michelogiannakis, G., & Kim, J. (2013) · 2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass.2013.6557149 · Cited by 790
Learning scheduling algorithms for data processing clusters
Mao, H., Schwarzkopf, M., Venkatakrishnan, S. B., Meng, Z., & Alizadeh, M. (2019) · Proceedings of the ACM Special Interest Group on Data Communication · doi:10.1145/3341302.3342080 · Cited by 723
Chen, P.-Y., Peng, X., & Yu, S. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2018.2789723 · Cited by 652
A simplicial homology algorithm for Lipschitz optimisation
Endres, S. C., Sandrock, C., & Focke, W. W. (2018) · Journal of Global Optimization · doi:10.1007/s10898-018-0645-y · Cited by 192
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2017) · arXiv · doi:10.48550/arxiv.1712.00559 · Cited by 171 · cited 2×
“neural architecture search (NAS) has been proposed to automatically design neural architectures with reduced model sizes [2, 17, 31, 32, 60].”
BackgroundNeural Predictor for Neural Architecture Search
Wen, W., Liu, H., Chen, Y., Li, H., Bender, G., & Kindermans, P.-J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58526-6_39 · Cited by 167
A Generic Graph-Based Neural Architecture Encoding Scheme for Predictor-Based NAS
Ning, X., Zheng, Y., Zhao, T., Wang, Y., & Yang, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_12 · Cited by 131
AtomLayer: A Universal ReRAM-Based CNN Accelerator with Atomic Layer Computation
Qiao, X., Cao, X., Yang, H., Song, L., & Li, H. H. (2018) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3195970.3195998 · Cited by 122
ElskenThomas, Metzen, J. H., & HutterFrank (2019) · Journal of Machine Learning Research · doi:10.5555/3322706.3361996 · Cited by 105
Device-Circuit-Architecture Co-Exploration for Computing-in-Memory Neural Accelerators
Jiang, W., Lou, Q., Yan, Z., Yang, L., Hu, J., Hu, X. S., & Shi, Y. (2021) · IEEE Transactions on Computers · doi:10.1109/tc.2020.2991575 · Cited by 96
Neural Network Inference on Mobile SoCs
Wang, S., Pathania, A., & Mitra, T. (2020) · IEEE Design and Test · doi:10.1109/mdat.2020.2968258 · Cited by 80
Hardware-aware machine learning: modeling and optimization
Marculescu, D., Stamoulis, D. S., & Cai, E. (2018) · doi:10.1145/3240765.3243479 · Cited by 55
A Latency-Optimized Reconfigurable NoC for In-Memory Acceleration of DNNs
Mandal, S. K., Krishnan, G., V, Chakrabarti, C., Seo, J.-S., Cao, Y. K., & Ogras, U. Y. (2020) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2020.3015509 · Cited by 40
Self-citationInterconnect-Aware Area and Energy Optimization for In-Memory Acceleration of DNNs
Krishnan, G., V, Mandal, S. K., Chakrabarti, C., Seo, J.-S., Ogras, U. Y., & Cao, Y. K. (2020) · IEEE Design and Test · doi:10.1109/mdat.2020.3001559 · Cited by 37
Self-citationBhardwaj, K., Li, G.-H., & Mărculescu, R. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr46437.2021.01329 · Cited by 26
Self-citationInference engine benchmarking across technological platforms from CMOS to RRAM
Peng, X., Kim, M., Sun, X., Yin, S., Rakshit, T., Hatcher, R., Kittl, J. A., Seo, J.-S., & Yu, S. (2019) · International Symposium on Memory Systems (MEMSYS) · doi:10.1145/3357526.3357566 · Cited by 13
Research works printed but not linked: 32
- Mohamed S. Abdelfattah , Abhinav Mehrotra , Łukasz Dudziak , and Nicholas Donald Lane . 2021 . Zero-cost proxies for lightweight NAS . In International Conference on Learning Representations. Mohamed S. Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas Donald Lane. 2021. Zero-cost proxies for lightweight NAS. In International Conference on Learning Representations.
- Bowen Baker , Otkrist Gupta , Nikhil Naik , and Ramesh Raskar . 2016. Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167 ( 2016 ). Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. 2016. Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167 (2016). · arXiv
- Hadjer Benmeziane et al . 2021. A comprehensive survey on hardware-aware neural architecture search. arXiv preprint arXiv:2101.09336 ( 2021 ). Hadjer Benmeziane et al. 2021. A comprehensive survey on hardware-aware neural architecture search. arXiv preprint arXiv:2101.09336 (2021). · arXiv
- Han Cai , Ligeng Zhu , and Song Han . 2019 . ProxylessNAS: Direct neural architecture search on target task and hardware . In International Conference on Learning Representations. Han Cai, Ligeng Zhu, and Song Han. 2019. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations.
- Thomas Chau , Łukasz Dudziak , Mohamed S. Abdelfattah , Royson Lee , Hyeji Kim , and Nicholas D . Lane . 2020 . BRP-NAS: Prediction-based NAS using GCNs . arXiv preprint arXiv:2007.08668 (2020). Thomas Chau, Łukasz Dudziak, Mohamed S. Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D. Lane. 2020. BRP-NAS: Prediction-based NAS using GCNs. arXiv preprint arXiv:2007.08668 (2020). · arXiv
- Wuyang Chen , Xinyu Gong , and Zhangyang Wang . 2021 . Neural architecture search on ImageNet in four GPU hours: A theoretically inspired perspective . In International Conference on Learning Representations. Wuyang Chen, Xinyu Gong, and Zhangyang Wang. 2021. Neural architecture search on ImageNet in four GPU hours: A theoretically inspired perspective. In International Conference on Learning Representations.
- Matthieu Courbariaux , Itay Hubara , Daniel Soudry , Ran El-Yaniv , and Yoshua Bengio . 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 ( 2016 ). Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 (2016). · arXiv
- Xuanyi Dong and Yi Yang . 2020. NAS-Bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326 ( 2020 ). Xuanyi Dong and Yi Yang. 2020. NAS-Bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326 (2020). · arXiv
- Boris Grot and Stephen W. Keckler . 2008. Scalable on-chip interconnect topologies . In 2nd Workshop on Chip Multiprocessor Memory Systems and Interconnects. Boris Grot and Stephen W. Keckler. 2008. Scalable on-chip interconnect topologies. In 2nd Workshop on Chip Multiprocessor Memory Systems and Interconnects.
- Song Han , Huizi Mao , and William J . Dally . 2015 . Deep compression: Compressing de ep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015). Song Han, Huizi Mao, and William J. Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015). · arXiv
- Geoffrey Hinton , Oriol Vinyals , and Jeff Dean . 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 ( 2015 ). Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015). · arXiv
- Chi-Hung Hsu et al . 2018 . Monas: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806.10332 (2018). Chi-Hung Hsu et al. 2018. Monas: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806.10332 (2018). · arXiv
- Gokul Krishnan et al . 2021 . Interconnect-centric benchmarking of in-memory acceleration for DNNS . In 2021 China Semiconductor Technology International Conference (CSTIC). IEEE, 1–4. Gokul Krishnan et al. 2021. Interconnect-centric benchmarking of in-memory acceleration for DNNS. In 2021 China Semiconductor Technology International Conference (CSTIC). IEEE, 1–4.
- 10.5555/3437539.3437669 · doi:10.5555/3437539.3437669
- Hanxiao Liu , Karen Simonyan , and Yiming Yang . 2018 . Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018). Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018). · arXiv
- Jovita Lukasik , David Friede , Heiner Stuckenschmidt , and Margret Keuper . 2020. Neural architecture performance prediction using graph neural networks. arXiv preprint arXiv:2010.10024 ( 2020 ). Jovita Lukasik, David Friede, Heiner Stuckenschmidt, and Margret Keuper. 2020. Neural architecture performance prediction using graph neural networks. arXiv preprint arXiv:2010.10024 (2020). · arXiv
- 10.5555/311445 · doi:10.5555/311445
- Volodymyr Mnih et al . 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 ( 2013 ). Volodymyr Mnih et al. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013). · arXiv
- 10.5555/1137830 · doi:10.5555/1137830
- Hieu Pham , Melody Guan , Barret Zoph , Quoc Le , and Jeff Dean . 2018 . Efficient neural architecture search via parameters sharing . In International Conference on Machine Learning. PMLR, 4095–4104 . Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. 2018. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning. PMLR, 4095–4104.
- 10.5555/3305890.3305981 · doi:10.5555/3305890.3305981
- Karen Simonyan and Andrew Zisserman . 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 ( 2014 ). Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). · arXiv
- Dimitrios Stamoulis et al . 2019. Single-path NAS: Designing hardware-efficient convnets in less than 4 hours. arXiv preprint arXiv:1904.02877 ( 2019 ). Dimitrios Stamoulis et al. 2019. Single-path NAS: Designing hardware-efficient convnets in less than 4 hours. arXiv preprint arXiv:1904.02877 (2019). · arXiv
- Mingxing Tan et al . 2019 . Mnasnet: Platform-aware neural architecture search for mobile . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2820–2828 . Mingxing Tan et al. 2019. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2820–2828.
- 10.5555/3157096.3157158 · doi:10.5555/3157096.3157158
- Duncan J. Watts and Steven H . Strogatz . 1998 . Collective dynamics of ‘small-world’ networks. Nature 393, 6684 (1998), 440–442. Duncan J. Watts and Steven H. Strogatz. 1998. Collective dynamics of ‘small-world’ networks. Nature 393, 6684 (1998), 440–442.
- Bichen Wu et al . 2019 . Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10734–10742 . Bichen Wu et al. 2019. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10734–10742.
- Sirui Xie , Hehui Zheng , Chunxiao Liu , and Liang Lin . 2019 . SNAS: Stochastic neural architecture search . In International Conference on Learning Representations. Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. 2019. SNAS: Stochastic neural architecture search. In International Conference on Learning Representations.
- Jiahui Yu et al . 2020. BigNAS: Scaling up neural architecture search with big single-stage models . In Computer Vision – ECCV 2020 . 702–717. Jiahui Yu et al. 2020. BigNAS: Scaling up neural architecture search with big single-stage models. In Computer Vision – ECCV 2020. 702–717.
- Sergey Zagoruyko and Nikos Komodakis . 2016. Wide residual networks. arXiv preprint arXiv:1605.07146 ( 2016 ). Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. arXiv preprint arXiv:1605.07146 (2016). · arXiv
- Barret Zoph and Quoc V . Le . 2016 . Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016). Barret Zoph and Quoc V. Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016). · arXiv
- Barret Zoph , Vijay Vasudevan , Jonathon Shlens , and Quoc V. Le . 2018. Learning transferable architectures for scalable image recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 8697–8710 . Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 8697–8710.
Cited by
Hardware implementation of memristor-based artificial neural networks
Aguirre, F., Sebastian, A., Le Gallo, M., Song, W., Wang, T., Yang, J. J., Lu, W., Chang, M.-F., Ielmini, D., Yang, Y., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-45670-9 · Cited by 564
Neural Architecture Search Survey: A Hardware Perspective
Chitty-Venkata, K. T., & Somani, A. K. (2023) · ACM Computing Surveys · doi:10.1145/3524500 · Cited by 106
Systematic review on neural architecture search
Avval, S. S. P., Eskue, N., Groves, R. M., & Yaghoubi, V. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11058-w · Cited by 141
Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities
Li, G.-H., Hoang, D.-T., Bhardwaj, K., Lin, M., Wang, Z.-Y., & Mărculescu, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3395423 · Cited by 69
Self-citationGeneric Neural Architecture Search via RegressionPreprint
Li, Y., Hao, C., Li, P., Xiong, J., & Chen, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01899 · Cited by 24
Metadata
- Authors
- Li, G., Mandal, S. K., Ogras, U. Y., & Marculescu, R.
- Affiliations
- The University of Texas at Austin (US)
- University of Wisconsin–Madison (US)
- The University of Texas at Austin, Austin, Texas, USA
- Venue
- ACM Transactions on Embedded Computing Systems
- Published
- 2021-09-17
- Volume
- 20
- Issue
- 5s
- Pages
- 1-26
- ISSN
- 1539-9087 (ISSN-L)
- 1558-3465
- OpenAlex
- W3200067639
- Cited by
- 15
- References
- 24
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 4 · 2023: 5 · 2022: 2 · 2021: 1
- FWCI
- 1.08
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Embedded Computing Systems (publishedVersion)
- Funders
- US National Science Foundation (CNS-2007284)
- Semiconductor Research Corporation (GRC 2939.001 and 3012.001)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- neural architecture search, hardware-aware neural network design, load forecasting, hierarchical optimization, skip connections, Raspberry Pi
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.