ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
Open accessThe DOI matches its OpenAlex record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Neural architecture search (NAS) has a great impact by automatically designing effective neural network architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. $10^4$ GPU hours) makes it difficult to \emph{directly} search the architectures on large-scale tasks (e.g. ImageNet). Differentiable NAS can reduce the cost of GPU hours via a continuous representation of network architecture but suffers from the high GPU memory consumption issue (grow linearly w.r.t. candidate set size). As a result, they need to utilize~\emph{proxy} tasks, such as training on a smaller dataset, or learning with only a few blocks, or training just for a few epochs. These architectures optimized on proxy tasks are not guaranteed to be optimal on the target task. In this paper, we present \emph{ProxylessNAS} that can \emph{directly} learn the architectures for large-scale target tasks and target hardware platforms. We address the high memory consumption issue of differentiable NAS and reduce the computational cost (GPU hours and GPU memory) to the same level of regular training while still allowing a large candidate set. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of directness and specialization. On CIFAR-10, our model achieves 2.08\% test error with only 5.7M parameters, better than the previous state-of-the-art architecture AmoebaNet-B, while using 6$\times$ fewer parameters. On ImageNet, our model achieves 3.1\% better top-1 accuracy than MobileNetV2, while being 1.2$\times$ faster with measured GPU latency. We also apply ProxylessNAS to specialize neural architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN architecture design.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · cited 2×
“2 EXPERIMENTS ON IMAGENET For ImageNet experiments, we focus on learning efficient CNN architectures (Iandola et al., 2016; Howard et al., 2017; Sandler et al., 2018; Zhu et al., 2018) that have not only high accuracy but also low latency on specific hardware platforms.”
MethodLearning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00907 · Cited by 7,831
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Ma, N., Zhang, X., Zheng, H.-T., & Sun, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_8 · Cited by 7,356 · cited 1×
“5) (Ma et al., 2018) 72.”
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · cited 1×
“For ImageNet experiments, we focus on learning efficient CNN architectures (Iandola et al., 2016; Howard et al., 2017; Sandler et al., 2018; Zhu et al., 2018) that have not only high accuracy but also low latency on specific hardware platforms.”
Self-citationHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Self-citationNeural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974
MnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416 · cited 4×
“As illustrated in Figure 5, we observe a strong correlation between the predicted latency and real measured latency on the test set, suggesting that the latency prediction model can be used to replace the expensive mobile farm infrastructure (Tan et al., 2018) with little error introduced.”
BackgroundMethodInfluentialSimple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248
Improved regularization of convolutional neural networks with cutout
DeVries, T., & Taylor, G. W. (2017) · arXiv · doi:10.48550/arxiv.1708.04552 · Cited by 2,776
Learning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180
Deep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_2 · Cited by 1,833
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
Learning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481
Self-citationAMC: AutoML for Model Compression and Acceleration on Mobile Devices
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., & Han, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_48 · Cited by 1,400
Self-citationUnderstanding and Simplifying One-Shot Architecture Search
Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., & Le, Q. V. (2018) · International Conference on Machine Learning · Cited by 834
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J. (1992) · Machine Learning · doi:10.1023/a:1022672621406 · Cited by 817
Deep Pyramidal Residual Networks
Han, D., Kim, J., & Kim, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.668 · Cited by 715 · cited 5×
“DenseNet-BC (Huang et al., 2017) 25.6M 3.46 PyramidNet (Han et al., 2017) 26.0M 3.31 Shake-Shake + c/o (DeVries & Taylor, 2017) 26.2M 2.56 PyramidNet + SD (Yamada et al., 2018) 26.0M 2.31 ENAS + c/o (Pham et al., 2018) 4.6M 2.89 DARTS + c/o (Liu et al., 2018c) 3.4M 2.83 NASNet-A + c/o (Zoph et al.,…”
MethodInfluentialEfficient Architecture Search by Network Transformation
Cai, H., Chen, T., Zhang, W., Yu, Y., & Wang, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11709 · Cited by 557
Self-citationPractical Block-Wise Neural Network Architecture Generation
Zhong, Z., Yan, J., Wu, W., Shao, J., & Liu, C.-L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00257 · Cited by 522
Efficient Neural Architecture Search via Parameter Sharing
Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., & Dean, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03268 · Cited by 517
Neural Architecture Optimization
Luo, R., Tian, F., Qin, T., Chen, E., & Liu, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07233 · Cited by 427
Regularized Evolution for Image Classifier Architecture Search
Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014780 · Cited by 328
Efficient multi-objective neural architecture search via Lamarckian evolution
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1804.09081 · Cited by 311
SMASH: One-Shot Model Architecture Search through HyperNetworks
Brock, A., Lim, T., Ritchie, J. M., & Weston, N. J. (2018) · International Conference on Learning Representations · Cited by 217
Hierarchical Representations for Efficient Architecture Search
Liu, H., Simonyan, K., Vinyals, O., Fernando, C., & Kavukcuoglu, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00436 · Cited by 194
DPP-Net: Device-Aware Progressive Search for Pareto-Optimal Neural Architectures
Dong, J.-D., Cheng, A.-C., Juan, D.-C., Wei, W., & Sun, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_32 · Cited by 189
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2017) · arXiv · doi:10.48550/arxiv.1712.00559 · Cited by 171 · cited 13×
“Normal Train NAS DARTS & One-shot Proxyless (Ours) However, naively including all the candidate paths leads to GPU memory explosion (Liu et al., 2018c; Bender et al., 2018), as the memory consumption grows linearly w.r.t. the number of choices.”
BackgroundMethodInfluentialShakedrop Regularization for Deep Residual Learning
Yamada, Y., Iwamura, M., Akiba, T., & Kise, K. (2019) · IEEE Access · doi:10.1109/access.2019.2960566 · Cited by 171
Simple And Efficient Architecture Search for Convolutional Neural Networks
Elsken, T., Metzen, J. H., & Hutter, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04528 · Cited by 170
MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
Hsu, C.-H., Chang, S.-H., Liang, J.-H., Chou, H.-P., Liu, C.-H., Chang, S.-C., Pan, T., Chen, Y., Wei, W., & Juan, D.-C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10332 · Cited by 137
Path-Level Network Transformation for Efficient Architecture Search
Cai, H., Yang, J., Zhang, W., Han, S., & Yu, Y. H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.02639 · Cited by 135
Self-citationSparsely Aggregated Convolutional Networks
Zhu, L., Deng, R., Maire, M., Deng, Z., Mori, G., & Tan, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01258-8_12 · Cited by 67
Self-citationMulti-objective Architecture Search for CNNs.Preprint
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv (Cornell University) · Cited by 25
Fast Neural Architecture Construction using EnvelopeNetsPreprint
Kamath, P., Singh, A., & Dutta, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.06744 · Cited by 12
Research works printed but not linked: 3
- Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization. arXiv, 2018.
- Yoshihiro Yamada, Masakazu Iwamura, and Koichi Kise. Shakedrop regularization. arXiv preprint arXiv:1802.02375, 2018. · arXiv
- Ligeng Zhu, Ruizhi Deng, Michael Maire, Zhiwei Deng, Greg Mori, and Ping Tan. Sparsely aggre- gated convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 186-201, 2018.
Cited by
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
“Unlike (Tan et al., 2019; Cai et al., 2019), here we optimize FLOPS rather than latency since we are not targeting any specific hardware device.”
BackgroundMethodHoward, A., Sandler, M., Chen, B., Wang, W., Chen, L.-C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., Pang, R., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00140 · Cited by 9,708
“As can be seen on figure 1 our models outperform the current state of the art such as MnasNet [43], ProxylessNas [5] and MobileNetV2 [39].”
MethodCoordinate Attention for Efficient Mobile Network Design
Hou, Q., Zhou, D., & Feng, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01350 · Cited by 6,961
A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects
Li, Z., Liu, F., Yang, W., Peng, S., & Zhou, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3084827 · Cited by 5,273
GhostNet: More Features From Cheap Operations
Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., & Xu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00165 · Cited by 5,222
Metadata
- Authors
- Cai, H., Zhu, L., & Han, S.
- Venue
- arXiv (Cornell University)
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1812.00332
- OpenAlex
- W2902251695
- Cited by
- 267
- References
- 40
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.