CoChat
Sign Up
PreprintCited by 127Advanced Neural Network ApplicationsMachine Learning and Data ClassificationDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 46 of the 51 references it cites were found in a registry or an index.

Stabilizing Differentiable Architecture Search via Perturbation-based Regularization

Open access
Chen, Xiangning, Hsieh, Cho-Jui·2020·arXiv
DOI 10.48550/arxiv.2002.05283arXiv 2002.05283
Open this paper in CoChat — free accountOpen PDF

Abstract

Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and largely reduces the search cost. However, its stability has been challenged for yielding deteriorating architectures as the search proceeds. We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. Based on this observation, we propose a perturbation-based regularization - SmoothDARTS (SDARTS), to smooth the loss landscape and improve the generalizability of DARTS-based methods. In particular, our new formulations stabilize DARTS-based methods by either random smoothing or adversarial attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the effectiveness of our approach and due to the improved stability, we achieve performance gain across various search spaces on 4 datasets. Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2002.05283

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 51 references

5 of the 51 references couldn’t be matched to a known paper.

Research works printed but not linked: 11

  1. Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., and Le, Q. Understanding and simplifying oneshot architecture search. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp. 550-559, Stockholmsm¨ assan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://proceedings.mlr. press/v80/bender18a.html .
  2. Cai, H., Yang, J., Zhang, W., Han, S., and Yu, Y. Pathlevel network transformation for efficient architecture search. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp. 678-687, Stockholmsm¨ assan, Stockholm Sweden, 1015 Jul 2018b. PMLR. URL http://proceedings. mlr.press/v80/cai18a.html .
  3. International Conference on Learning Representations , 2019. URL https://openreview.net/forum? id=HylVB3AqYm .
  4. Elsken, T., Metzen, J. H., and Hutter, F. Efficient multiobjective neural architecture search via lamarckian evolution. In International Conference on Learning Representations , 2019. URL https://openreview.net/ forum?id=ByME42AqK7 .
  5. Goodfellow, I., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In International Conference on Learning Representations , 2015. URL http://arxiv.org/abs/1412.6572 . · arXiv
  6. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018a.
  7. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018b. URL https:// openreview.net/forum?id=rJzIBfZAb .
  8. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. In International Conference on Learning Representations , 2014. URL http: //arxiv.org/abs/1312.6199 .
  9. Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. , 8(3-4):229-256, May 1992. ISSN 0885-6125. doi: 10.1007/BF00992696. URL https://doi.org/ 10.1007/BF00992696 . · DOI
  10. Yao, Q., Chen, X., Kwok, J. T., Li, Y., and Hsieh, C.-J. Efficient neural interaction function search for collaborative filtering. In Proceedings of The Web Conference 2020 , WWW '20, pp. 1660-1670, New York, NY, USA, 2020a. Association for Computing Machinery. ISBN 9781450370233. doi: 10.1145/ 3366423.3380237. URL https://doi.org/10. 1145/3366423.3380237 .
  11. Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., and Hutter, F. NAS-bench-101: Towards reproducible neural architecture search. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 7105-7114, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http://proceedings.mlr. press/v97/ying19a.html .

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Chen, X., & Hsieh, C.-J.
Affiliations
University of California, Los Angeles (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2002.05283
arXiv
2002.05283
OpenAlex
W3006500080
Cited by
126
References
51
Citations by year
2025: 17 · 2024: 22 · 2023: 33 · 2022: 23 · 2021: 23 · 2020: 8
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
differentiable architecture search, DART, adversarial attacks, generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review