The DOI matches its DataCite record; 46 of the 51 references it cites were found in a registry or an index.
Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
Open accessAbstract
Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and largely reduces the search cost. However, its stability has been challenged for yielding deteriorating architectures as the search proceeds. We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. Based on this observation, we propose a perturbation-based regularization - SmoothDARTS (SDARTS), to smooth the loss landscape and improve the generalizability of DARTS-based methods. In particular, our new formulations stabilize DARTS-based methods by either random smoothing or adversarial attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the effectiveness of our approach and due to the improved stability, we achieve performance gain across various search spaces on 4 datasets. Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The convergence to a smooth landscape has been shown to benefit the generalization of deep neural networks (Chen and Hsieh, 2020; Chen et al., 2022; Foret et al., 2021; Keskar et al., 2017).”
From Symbolic Discovery of Optimization Algorithms (2023) · cites this paper as Chen and Hsieh, 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Ma, N., Zhang, X., Zheng, H.-T., & Sun, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_8 · Cited by 7,356 · cited 1×
“2 ShuffleNet (v2) (Ma et al., 2018) 25.”
Learning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00907 · Cited by 6,875
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,705
MnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416
Evolving Neural Networks through Augmenting Topologies
Stanley, K. O., & Miikkulainen, R. P. (2002) · Evolutionary Computation · doi:10.1162/106365602320169811 · Cited by 3,269
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248
Improved regularization of convolutional neural networks with cutout
DeVries, T., & Taylor, G. W. (2017) · arXiv · doi:10.48550/arxiv.1708.04552 · Cited by 2,776
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Wei, H., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_2 · Cited by 1,975
ProxylessNAS: Direct Neural Architecture Search on Target Task and HardwarePreprint
Cai, H., Zhu, L., & Han, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00332 · Cited by 1,821
(2018) · International Conference on Machine Learning · Cited by 1,465
Large-Scale Evolution of Image ClassifiersPreprint
Real, E., Moore, S., Selle, A. P., Saxena, S., Suematsu, Y. I. L., Tan, J., Le, Q. V., & Kurakin, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01041 · Cited by 1,431
Certified Adversarial Robustness via Randomized SmoothingPreprint
Cohen, J. M., Rosenfeld, E., & Kolter, J. Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02918 · Cited by 1,232
Miikkulainen, R. P., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., et al. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-815480-9.00015-3 · Cited by 871
Hierarchical Representations for Efficient Architecture SearchPreprint
Liu, H., Simonyan, K., Vinyals, O., Fernando, C., & Kavukcuoglu, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00436 · Cited by 851
Certified Robustness to Adversarial Examples with Differential Privacy
Lecuyer, M., Atlidakis, V., Geambasu, R., Hsu, D., & Jana, S. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00044 · Cited by 750
Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation
Chen, X., Xie, L., Wu, J. H., & Tian, Q. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00138 · Cited by 731
Self-citationSNAS: Stochastic Neural Architecture Search
Xie, S., Zheng, H., Liu, C., & Lin, L. (2018) · arXiv · doi:10.48550/arxiv.1812.09926 · Cited by 731
Deep Pyramidal Residual Networks
Han, D., Kim, J., & Kim, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.668 · Cited by 715
Searching for a Robust Neural Architecture in Four GPU Hours
Dong, X., & Yang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00186 · Cited by 675
PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture SearchPreprint
Xu, Y., Xie, L., Zhang, X., Chen, X., Qi, G.-J., Tian, Q. C., & Xiong, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05737 · Cited by 617
Self-citationEfficient Architecture Search by Network Transformation
Cai, H., Chen, T., Zhang, W., Yu, Y. H., & Wang, Y.-X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11709 · Cited by 616
Random Search and Reproducibility for Neural Architecture SearchPreprint
Li, L., & Talwalkar, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07638 · Cited by 582
Practical Block-Wise Neural Network Architecture Generation
Zhong, Z., Yan, J., Wu, W., Shao, J., & Liu, C. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00257 · Cited by 581
Neural Architecture OptimizationPreprint
Luo, R., Tian, F., Qin, T., Chen, E., & Liu, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07233 · Cited by 541
Adversarial Examples Improve Image Recognition
Xie, C., Tan, M., Gong, B., Wang, J., Yuille, A., & Le, Q. V. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00090 · Cited by 510
Towards Robust Neural Networks via Random Self-ensemble
Liu, X., Cheng, M., Zhang, H., & Hsieh, C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_23 · Cited by 408
Self-citationFair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search
Chu, X., Zhou, T., Zhang, B., & Li, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58555-6_28 · Cited by 356
DARTS+: Improved Differentiable Architecture Search with Early StoppingPreprint
Liang, H., Zhang, S., Sun, J., He, X., Huang, W., Zhuang, K., & Li, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06035 · Cited by 349
Breaking the Softmax Bottleneck: A High-Rank RNN Language ModelPreprint
Yang, Z., Dai, Z., Salakhutdinov, R., & Cohen, W. W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.03953 · Cited by 348
Regularized Evolution for Image Classifier Architecture Search
Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014780 · Cited by 328
Understanding and Robustifying Differentiable Architecture SearchPreprint
Zela, A., Elsken, T., Saikia, T., Marrakchi, Y., Brox, T., & Hutter, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09656 · Cited by 279
Certified Adversarial Robustness with Additive NoisePreprint
Li, B., Chen, C., Wang, W., & Carin, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03113 · Cited by 250
SMASH: One-Shot Model Architecture Search through HyperNetworks
Brock, A., Lim, T., Ritchie, J. M., & Weston, N. J. (2018) · International Conference on Learning Representations · Cited by 217
Learning Curve Prediction with Bayesian Neural Networks
Klein, A., Falkner, S., Springenberg, J. T., & Hutter, F. (2017) · International Conference on Learning Representations · Cited by 176
BayesNAS: A Bayesian Approach for Neural Architecture SearchPreprint
Zhou, H., Yang, M., Wang, J., & Pan, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.04919 · Cited by 176
Rob-GAN: Generator, Discriminator, and Adversarial Attacker
Liu, X., & Hsieh, C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01149 · Cited by 139
Self-citationEvaluating The Search Phase of Neural Architecture Search
Yu, K., Sciuto, C., Jaggi, M., Musat, C., & Salzmann, M. (2020) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 136
Efficient Neural Architecture Search via Proximal Iterations
Yao, Q., Xu, J., Tu, W.-W., & Zhu, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6143 · Cited by 104
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture SearchPreprint
Zela, A., Siems, J., & Hutter, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.10422 · Cited by 74
Understanding Architectures Learnt by Cell-based Neural Architecture SearchPreprint
Shu, Y., Wang, W., & Cai, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09569 · Cited by 70
How Does Noise Help Robustness? Explanation and Exploration under the Neural SDE Framework
Liu, X., Xiao, T., Si, S., Cao, Q., Kumar, S., & Hsieh, C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00036 · Cited by 54
Self-citationStabilizing Differentiable Architecture Search via Perturbation-based Regularization
Chen, X. (2025) · TIB Data Manager · doi:10.57702/ahufmtof · Cited by 36
Self-citation
Research works printed but not linked: 11
- Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., and Le, Q. Understanding and simplifying oneshot architecture search. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp. 550-559, Stockholmsm¨ assan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://proceedings.mlr. press/v80/bender18a.html .
- Cai, H., Yang, J., Zhang, W., Han, S., and Yu, Y. Pathlevel network transformation for efficient architecture search. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp. 678-687, Stockholmsm¨ assan, Stockholm Sweden, 1015 Jul 2018b. PMLR. URL http://proceedings. mlr.press/v80/cai18a.html .
- International Conference on Learning Representations , 2019. URL https://openreview.net/forum? id=HylVB3AqYm .
- Elsken, T., Metzen, J. H., and Hutter, F. Efficient multiobjective neural architecture search via lamarckian evolution. In International Conference on Learning Representations , 2019. URL https://openreview.net/ forum?id=ByME42AqK7 .
- Goodfellow, I., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In International Conference on Learning Representations , 2015. URL http://arxiv.org/abs/1412.6572 . · arXiv
- Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018a.
- Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018b. URL https:// openreview.net/forum?id=rJzIBfZAb .
- Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. In International Conference on Learning Representations , 2014. URL http: //arxiv.org/abs/1312.6199 .
- Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. , 8(3-4):229-256, May 1992. ISSN 0885-6125. doi: 10.1007/BF00992696. URL https://doi.org/ 10.1007/BF00992696 . · DOI
- Yao, Q., Chen, X., Kwok, J. T., Li, Y., and Hsieh, C.-J. Efficient neural interaction function search for collaborative filtering. In Proceedings of The Web Conference 2020 , WWW '20, pp. 1660-1670, New York, NY, USA, 2020a. Association for Computing Machinery. ISBN 9781450370233. doi: 10.1145/ 3366423.3380237. URL https://doi.org/10. 1145/3366423.3380237 .
- Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., and Hutter, F. NAS-bench-101: Towards reproducible neural architecture search. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 7105-7114, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http://proceedings.mlr. press/v97/ying19a.html .
Cited by
Sorted by citations
Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning
Mendieta, M., Yang, T., Wang, P., Lee, M., Ding, Z., & Chen, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00821 · Cited by 204
AutoML: A systematic review on automated machine learning with neural architecture search
Salehin, I., Islam, M. S., Saha, P., Noman, S. M., Tuni, A., Hasan, M. M., & Baten, M. A. (2023) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2023.10.002 · Cited by 213
Symbolic Discovery of Optimization Algorithms
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.06675 · Cited by 303
“, 2022), neural architecture search (Chen and Hsieh, 2020; Chen et al., 2021; Liu et al., 2019a; Pham et al., 2018; Real et al., 2019; So et al., 2019; Wang et al., 2021; Yang et al., 2022; Zoph and Le, 2017) and hyperparameter optimization (Dong et al.”
BackgroundSelf-citationDifferentiable quantum architecture search
Zhang, S., Hsieh, C., Zhang, S., & Yao, H. (2022) · Quantum Science and Technology · doi:10.1088/2058-9565/ac87cd · Cited by 206
Self-citation$\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture Search
Ye, P., Li, B., Li, Y., Chen, T., Fan, J., & Ouyang, W. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01060 · Cited by 140
Metadata
- Authors
- Chen, X., & Hsieh, C.-J.
- Affiliations
- University of California, Los Angeles (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2002.05283
- OpenAlex
- W3006500080
- Cited by
- 126
- References
- 51
- Citations by year
- 2025: 17 · 2024: 22 · 2023: 33 · 2022: 23 · 2021: 23 · 2020: 8
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- differentiable architecture search, DART, adversarial attacks, generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
