Once-for-All: Train One Network and Specialize it for Efficient Deployment
Open accessAbstract
We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally prohibitive (causing $CO_2$ emission as much as 5 cars' lifetime) thus unscalable. In this work, we propose to train a once-for-all (OFA) network that supports diverse architectural settings by decoupling training and search, to reduce the cost. We can quickly get a specialized sub-network by selecting from the OFA network without additional training. To efficiently train OFA networks, we also propose a novel progressive shrinking algorithm, a generalized pruning method that reduces the model size across many more dimensions than pruning (depth, width, kernel size, and resolution). It can obtain a surprisingly large number of sub-networks ($> 10^{19}$) that can fit different hardware platforms and latency constraints while maintaining the same level of accuracy as training independently. On diverse edge devices, OFA consistently outperforms state-of-the-art (SOTA) NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3, or same accuracy but 1.5x faster than MobileNetV3, 2.6x faster than EfficientNet w.r.t measured latency) while reducing many orders of magnitude GPU hours and $CO_2$ emission. In particular, OFA achieves a new SOTA 80.0% ImageNet top-1 accuracy under the mobile setting ($
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
OSA maintains the same accuracy level independent of the training phase of the network.
“While doing such model setting, the OSA maintains the same accuracy level which is completely independent of the training phase of the network.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
The Conclusion (p. 11) says progressive shrinking enables many OFA sub-networks to achieve the same level of accuracy as if trained independently. But the paper does not say accuracy is independent of the training phase: Section 3.3 (p. 5) describes a staged training process, and Figure 7 (p. 6) says progressive shrinking improves sub-network accuracy. (LLM, conf=0.86)
Experimental — accuracy not yet measuredPSA outperforms the generalized pruning approach for model size minimization.
“It is seen that this newly developed PSA outperforms generalized pruning approach for model size minimization.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
The Abstract (p. 1) says progressive shrinking reduces model size across more dimensions than pruning, and Section 3.3 (pp. 4–5) says it offers greater flexibility than conventional network pruning. The paper does not show that it achieves a smaller minimum model size, and it describes progressive shrinking itself as a generalized pruning method, not as a method outperforming a separate generalized-pruning approach. (LLM, conf=0.85)
Experimental — accuracy not yet measuredOSA can outperform the accuracy of MobileNetV3 or similar networks with 1.5x fast response time.
“The OSA can outperform the accuracy of MobileNetV3 or similar networks with 1.5x fast response time.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
The Abstract (p. 1) says OFA—not OSA—achieves up to 4.0% higher ImageNet top-1 accuracy than MobileNetV3, or the same accuracy at 1.5× faster measured latency. The claim combines those alternative comparisons into a simultaneous accuracy and speed improvement, which the passage does not establish. (LLM, conf=0.88)
Experimental — accuracy not yet measuredOFA is assisted by the progressive shrinking algorithm (PSA) that reduces model size in terms of kernel size, resolution, depth, and width.
“The OFA is assisted by the newly developed progressive shrinking algorithm (PSA) that reduces the model size in terms of kernel size, resolution, depth, and width.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
Cai et al. (2019) investigated a one-for-all (OFA) network for leveraging a variety of architectural settings where complete decoupling between training and search is performed.
“A new method is investigated in (Cai et al., 2019) that applies a one-for-all (OFA) network for leveraging a variety of architectural settings where a complete decoupling between training and search is performed.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
The OFA method allows direct selection of a specialized subnetwork for the FA network without requiring training.
“The OFA method allows to directly select specialized subnetwork for the FA network without requiring training.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
OFA can reduce deployment cost from O(n) to O(1).
“It can reduce the deployment cost from O(n) to O(1).”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
Improved PSA can help obtain more than 10^9 sub networks that can fit a very large set of various hardware platforms while considering different latency constraints.
“The improved PSA can help to obtain a greater number of sub networks (>10 9 ) which can fit a very large set of various hardware platforms while considering different latency constraints.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
It is computationally infeasible to enumerate all sub-nets for getting appropriate gradient during an update step.
“It is computationally infeasible to enumerate all sun-net for getting appropriate gradient during an update step.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
Training an OFA network is a non-trivial task for maintaining accuracy for the huge number of subnets based on joint optimization of weights.
“A challenge is noticed during the training of OFA network i.e., a non-trivial task for maintaining accuracy for the huge number of subnets based on the joint optimization of weights.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
PSA is a generalized network pruning that aims to shrink depth, kernel size, width, and resolution of the full network while preserving the accuracy of all subnets.
“The PSA is a generalized network pruning that aims to shrink depth, kernel size, width, and resolution of the full network while preserving the accuracy of all subnets (in contrast, the pruning helps to shrink width of network and accuracy for a single network).”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
The main reason behind severe accuracy drops from random sampling is the difference between various subnets that leads to inefficient training of the OFA network.
“The main reason behind such behavior is the difference between various subnets that leads to inefficient training of the OFA network.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
Random sampling of sub-nets may incur severe accuracy drops.
“On other hand, random sampling of sub-nets may incur severe accuracy drops.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
OFA achieves 80% ImageNet top1 with 595 M MACs.
“Results show that OFA achieves 80% ImageNet top1 with 595 M MACs.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
Evaluation of OFA shows significant improvement of state-of-the-art hardware-NAS while minimizing CO2 emission, cost, and GPU hours.
“The evaluation of OFA shows significant improvement of state-of-the-art hardware-NAS while minimizing the CO 2 emission, cost, and GPU hours.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Cai et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 42 · Cited by 845
Research works printed but not linked: 39
- Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, and Kris M Kitani. N2n learning: Network to network compression via policy gradient reinforcement learning. In ICLR , 2018.
- Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI , 2018a.
- Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformation for efficient architecture search. In ICML , 2018b.
- Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In ICLR , 2019. URL https://arxiv.org/pdf/1812.00332.pdf . · arXiv
- Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. In NeurIPS , 2019.
- Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS , 2015.
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.
- Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420 , 2019. · arXiv
- Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurIPS , 2015.
- Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR , 2016.
- Cong Hao, Xiaofan Zhang, Yuhong Li, Sitao Huang, Jinjun Xiong, Kyle Rupnow, Wen-mei Hwu, and Deming Chen. Fpga/dnn co-design: An efficient design methodology for 1ot intelligence on the edge. In 2019 56th ACM/IEEE Design Automation Conference (DAC) , pp. 1-6. IEEE, 2019.
- Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In ECCV , 2018.
- Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. · arXiv
- Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017. · arXiv
- Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR , 2017.
- Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. In ICLR , 2018.
- Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size. arXiv preprint arXiv:1602.07360 , 2016. · arXiv
- Weiwen Jiang, Lei Yang, Edwin Sha, Qingfeng Zhuge, Shouzhen Gu, Yiyu Shi, and Jingtong Hu. Hardware/software co-exploration of neural architectures. arXiv preprint arXiv:1907.04650 , 2019a. · arXiv
- Weiwen Jiang, Xinyi Zhang, Edwin H-M Sha, Lei Yang, Qingfeng Zhuge, Yiyu Shi, and Jingtong Hu. Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search. In Proceedings of the 56th Annual Design Automation Conference 2019 , pp. 1-6, 2019b.
- Jason Kuen, Xiangfei Kong, Zhe Lin, Gang Wang, Jianxiong Yin, Simon See, and Yap-Peng Tan. Stochastic downsampling for cost-adjustable inference and improved regularization in convolutional networks. In CVPR , 2018.
- Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NeurIPS , 2017.
- Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV , 2018.
- Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR , 2019.
- Lanlan Liu and Jia Deng. Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution. In AAAI , 2018.
- Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV , 2017.
- Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016. · arXiv
- Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In AAAI , 2019.
- Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR , 2018.
- Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In ACL , 2019.
- Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In ECCV , 2018.
- Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR , 2019.
- Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR , 2018.
- Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11728 , 2019a. · arXiv
- Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In ICCV , 2019b.
- Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In ICLR , 2019.
- Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR , 2018.
- Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In ICLR , 2017.
- Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In ICLR , 2017.
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR , 2018.
- Authors
- Cai, H., Gan, C., Wang, T., Zhang, Z., & Han, S.
- Affiliations
- Massachusetts Institute of Technology (US)
- IBM (United States) (US)
- Shanghai Jiao Tong University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1908.09791
- OpenAlex
- W2969797940
- Cited by
- 845
- References
- 42
- Citations by year
- 2026: 8 · 2025: 49 · 2024: 156 · 2023: 208 · 2022: 129 · 2021: 215 · 2020: 75 · 2019: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- neural architecture search, network pruning, edge devices, inference efficiency, CO2 emission reduction, model compression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
