CoChat
Sign Up
PreprintCited by 10Advanced Neural Network ApplicationsMachine Learning and Data ClassificationDomain Adaptation and Few-Shot Learning

AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling

Open access
Wang, Dilin, Li, Meng, Gong, Chengyue, Chandra, Vikas·2020·arXiv
DOI 10.48550/arxiv.2011.09011arXiv 2011.09011
Open this paper in CoChat — free accountOpen PDF

Abstract

Neural architecture search (NAS) has shown great promise in designing state-of-the-art (SOTA) models that are both accurate and efficient. Recently, two-stage NAS, e.g. BigNAS, decouples the model training and searching process and achieves remarkable search efficiency and accuracy. Two-stage NAS requires sampling from the search space during training, which directly impacts the accuracy of the final searched models. While uniform sampling has been widely used for its simplicity, it is agnostic of the model performance Pareto front, which is the main focus in the search process, and thus, misses opportunities to further improve the model accuracy. In this work, we propose AttentiveNAS that focuses on improving the sampling strategy to achieve better performance Pareto. We also propose algorithms to efficiently and effectively identify the networks on the Pareto during training. Without extra re-training or post-processing, we can simultaneously obtain a large number of networks across a wide range of FLOPs. Our discovered model family, AttentiveNAS models, achieves top-1 accuracy from 77.3% to 80.7% on ImageNet, and outperforms SOTA models, including BigNAS and Once-for-All networks. We also achieve ImageNet accuracy of 80.1% with only 491 MFLOPs. Our training code and pretrained models are available at https://github.com/facebookresearch/AttentiveNAS.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2011.09011

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Work [246] uses a reinforcement learning agent to select hardware-aware mixed-precision settings based on a latency look-up table for layers and bit widths.

    “In particular, the work [246] uses a reinforcement learning agent to determine the hardware-aware mixed-precision setting for quantization, based on a look-up table of latency with respect to different layers with different bitwidth.”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [246]

    Section 4.2 (page unavailable) describes AttentiveNAS sampling architectures under FLOPs constraints using an empirical distribution and rejection sampling, not a reinforcement-learning agent. The search-space description in Additional information (page unavailable) lists widths, depths, kernel sizes, expansion ratios, and input resolution rather than quantization bit widths; the copy does not describe a layer-and-bit-width latency look-up table. (LLM, conf=0.86)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • The approach in [246] relies on simulated hardware latency.

    “However, this approach uses simulated hardware latency.”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [246]

    Section 4.2 describes AttentiveNAS as sampling architectures under FLOPs constraints, not simulated hardware-latency constraints. The Additional information section says latency is another metric the method could be adapted to and reports GPU latency comparisons on 2080 Ti and V100 GPUs; it does not describe reliance on simulated hardware latency. (LLM, conf=0.86)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Exploration-based mixed-precision methods often need large computational resources and are sensitive to hyperparameters and initialization.

    “One disadvantage of these exploration-based methods [246,254] is that they often require large computational resources, and their performance is typically sensitive to hyperparameters and even initialization.”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [246]

    The Introduction and Related Work describe conventional neural architecture search as computationally expensive, and Related Work says some weight-sharing search methods are sensitive to hyperparameter choices, including random seeds. The paper does not establish these points specifically for mixed-precision methods, and it does not show that the expensive methods are also sensitive to initialization. Page numbers are unavailable in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • HAWQv2 extends this to mixed-precision activation quantization and is over 100 times faster than RL-based mixed-precision methods.

    “In HAWQv2, this method was extended to mixedprecision activation quantization [50], and was shown to be more than 100x faster than RL based mixed-precision methods [246].”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [246]

    The citation to this paper is attached to the claim that HAWQv2 is more than 100 times faster than RL-based mixed-precision methods. Our copy's Additional information section compares AttentiveNAS training costs with RL-based neural architecture search, but does not address HAWQv2 or mixed-precision quantization. Searches for HAWQv2, activation quantization, and the claimed speed comparison found no supporting passage; page numbers were not available. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Mixed-precision quantization can address this by using different bit precision for each layer.

    “It is possible to address this with mixed-precision quantization [51,82,102,162,187,199,211,239,246,249,263,282,286].”

    From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [246]

    The Search Space section and Table 2 in Additional information describe architectural choices such as channel width, depth, and kernel size, not quantization precision. Searches for mixed-precision quantization and layer-specific bit widths found no passage in our copy addressing the claim; page numbers were not available. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A conver- gence theory for deep learning via over-parameterization. In International Conference on Machine Learning, pages 242- 252. PMLR, 2019. 3
  2. Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In European conference on computer vision, pages 446-461. Springer, 2014. 13
  3. Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791, 2019. 1, 3, 8, 12, 13 · arXiv
  4. Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332, 2018. 2, 8, 13 · arXiv
  5. Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural net- works. In Advances in Neural Information Processing Sys- tems, pages 10836-10846, 2019. 3
  6. An-Chieh Cheng, Jin-Dong Dong, Chi-Hung Hsu, Shu- Huan Chang, Min Sun, Shih-Chieh Chang, Jia-Yu Pan, Yu- Ting Chen, Wei Wei, and Da-Cheng Juan. Searching to- ward pareto-optimal device-aware neural architectures. In Proceedings of the International Conference on Computer- Aided Design, pages 1-7, 2018. 2
  7. Ting-Wu Chin, Ari S Morcos, and Diana Marculescu. Pareco: Pareto-aware channel optimization for slimmable neural networks. arXiv preprint arXiv:2007.11752, 2020. 2 · arXiv
  8. Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fair- nas: Rethinking evaluation fairness of weight sharing neural architecture search. arXiv preprint arXiv:1907.01845, 2019. 1, 2, 3, 6, 8, 11 · arXiv
  9. Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasude- van, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501, 2018. 11 · arXiv
  10. Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zi- jian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. Fbnetv3: Joint architecture-recipe search using neural acquisition function. arXiv preprint arXiv:2006.02049, 2020. 1, 5, 8, 11 · arXiv
  11. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009. 3
  12. Xuanyi Dong and Yi Yang. Nas-bench-102: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326, 2020. 2 · arXiv
  13. Chengyue Gong, Tongzheng Ren, Mao Ye, and Qiang Liu. Maxup: A simple way to improve generalization of neural network training. arXiv preprint arXiv:2002.09024, 2020. 2 · arXiv
  14. Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one- shot neural architecture search with uniform sampling. In European Conference on Computer Vision, pages 544-560. Springer, 2020. 1, 2, 3
  15. Song Han, Huizi Mao, and William J Dally. Deep com- pression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015. 1 · arXiv
  16. Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mo- bilenetv3. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 1314-1324, 2019. 8, 12
  17. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An- dreetto, and Hartwig Adam. Mobilenets: Efficient convolu- tional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. 1 · arXiv
  18. Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018. 12
  19. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Fi- rat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in Neural Information Processing Systems, 2018. 13
  20. SouYoung Jin, Aruni RoyChowdhury, Huaizu Jiang, Ashish Singh, Aditya Prasad, Deep Chakraborty, and Erik Learned- Miller. Unsupervised hard example mining from videos for improved object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 307-324, 2018. 4
  21. Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine-grained cars. Second Workshop on Fine-Grained Visual Categorization, 2013. 13
  22. Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018. 2, 8, 13 · arXiv
  23. Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008. 13
  24. Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268, 2018. 2 · arXiv
  25. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780-4789, 2019. 2
  26. Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alex Kurakin. Large-scale evolution of image classifiers. arXiv preprint arXiv:1703.01041, 2017. 2 · arXiv
  27. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4510-4520, 2018. 8, 12
  28. Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard ex- ample mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761-769, 2016. 4
  29. Evgeny Smirnov, Aleksandr Melnikov, Andrei Oleinik, Elizaveta Ivanova, Ilya Kalinovskiy, and Eugene Luck- yanets. Hard example mining with auxiliary embeddings. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 37-46, 2018. 2, 4
  30. Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Mar- culescu. Single-path nas: Designing hardware-efficient con- vnets in less than 4 hours. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 481-497. Springer, 2019. 2
  31. Masanori Suganuma, Mete Ozay, and Takayuki Okatani. Ex- ploiting the potential of standard convolutional autoencoders for image restoration by evolutionary search. arXiv preprint arXiv:1803.00370, 2018. 2 · arXiv
  32. Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee. Stochastic class-based hard example mining for deep metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7251- 7259, 2019. 4
  33. Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnas- net: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2820-2828, 2019. 1, 2, 8, 12, 13
  34. Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946, 2019. 1, 8, 12, 13 · arXiv
  35. Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuan- dong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12965-12974, 2020. 1, 8
  36. Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. arXiv preprint arXiv:2005.14187, 2020. 1, 2, 3, 6 · arXiv
  37. Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient con- vnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10734-10742, 2019. 1, 2
  38. Lingxi Xie and Alan Yuille. Genetic cnn. In Proceedings of the IEEE international conference on computer vision, pages 1379-1388, 2017. 2
  39. Antoine Yang, Pedro M Esperanc ¸a, and Fabio M Car- lucci. Nas evaluation is frustratingly hard. arXiv preprint arXiv:1912.12522, 2019. 2, 6 · arXiv
  40. Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pages 7105-7114, 2019. 2
  41. Jiahui Yu and Thomas S Huang. Universally slimmable net- works and improved training techniques. In Proceedings of the IEEE International Conference on Computer Vision, pages 1803-1811, 2019. 6
  42. Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xi- aodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. arXiv preprint arXiv:2003.11142, 2020. 1, 2, 3, 5, 8, 11 · arXiv
  43. Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture gener- ation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2423-2432, 2018. 2
  44. Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016. 1, 2, 12 · arXiv
  45. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018. 2, 8

Cited by

Sorted by citations

Authors
Wang, D., Li, M., Gong, C., & Chandra, V.
Affiliations
The University of Texas at Austin (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2011.09011
arXiv
2011.09011
OpenAlex
W3098072209
Cited by
10
References
1
Citations by year
2024: 2 · 2023: 2 · 2022: 1 · 2021: 5
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
neural architecture search, Pareto front, ImageNet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review