CoChat
Sign Up
PreprintCited by 143Evolutionary Algorithms and ApplicationsMachine Learning and Data ClassificationStochastic Gradient Optimization Techniques

AutoML-Zero: Evolving Machine Learning Algorithms From Scratch

Open access
Real, Esteban, Liang, Chen, So, David R., Le, Quoc V.·2020·arXiv
DOI 10.48550/arxiv.2003.03384arXiv 2003.03384
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2003.03384

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AutoML-Zero attempts to search every component of a machine learning pipeline while evaluating on toy tasks.

    “AutoML-Zero (Real et al., 2020) is an ambitious effort that attempts to search every component of a machine learning pipeline while evaluating on toy tasks.”

    From Symbolic Discovery of Optimization Algorithms (2023) · cites this paper as Real et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 81

  1. Alba, E. and Tomassini, M. Parallelism and evolutionary algorithms. IEEE transactions on evolutionary computation , 2002.
  2. Alet, F., Schneider, M. F., Lozano-Perez, T., and Kaelbling, L. P. Meta-learning curiosity algorithms. In International Conference on Learning Representations , 2019.
  3. Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., and de Freitas, N. Learning to learn by gradient descent by gradient descent. In NIPS , 2016.
  4. Angeline, P. J., Saunders, G. M., and Pollack, J. B. An evolutionary algorithm that constructs recurrent neural networks. IEEE transactions on Neural Networks , 1994.
  5. Baker, B., Gupta, O., Naik, N., and Raskar, R. Designing neural network architectures using reinforcement learning. In ICLR , 2017.
  6. Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., and Tarlow, D. Deepcoder: Learning to write programs. ICLR , 2017.
  7. Bello, I., Zoph, B., Vasudevan, V., and Le, Q. V. Neural optimizer search with reinforcement learning. ICML , 2017.
  8. Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint , 2013.
  9. Bergstra, J. and Bengio, Y. Random search for hyperparameter optimization. JMLR , 2012.
  10. Chalmers, D. J. The evolution of learning: An experiment in genetic connectionism. In Connectionist Models . Elsevier, 1991.
  11. Chen, X., Liu, C., and Song, D. Towards synthesizing complex programs from input-output examples. arXiv preprint arXiv:1706.01284 , 2017. · arXiv
  12. Chrabaszcz, P., Loshchilov, I., and Hutter, F. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819 , 2017. · arXiv
  13. Collins, M. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of the ACL-02 conference on Empirical methods in natural language processingVolume 10 , pp. 1-8. Association for Computational Linguistics, 2002.
  14. Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation policies from data. CVPR , 2019a.
  15. Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmentation with a reduced search space. arXiv preprint arXiv:1909.13719 , 2019b. · arXiv
  16. Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., rahman Mohamed, A., and Kohli, P. Robustfill: Neural program learning under noisy i/o. In ICML , 2017.
  17. Elsken, T., Metzen, J. H., and Hutter, F. Efficient multiobjective neural architecture search via lamarckian evolution. In ICLR , 2019a.
  18. Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search. In Automated Machine Learning . Springer, 2019b.
  19. Fahlman, S. E. and Lebiere, C. The cascade-correlation learning architecture. In NIPS , 1990.
  20. Finn, C., Abbeel, P., and Levine, S. Model-agnostic metalearning for fast adaptation of deep networks. In ICML , 2017.
  21. Forsyth, R. et al. Beagle-a darwinian approach to pattern recognition. Kybernetes , 10(3):159-166, 1981.
  22. Gaier, A. and Ha, D. Weight agnostic neural networks. In NeurIPS , 2019.
  23. Ghiasi, G., Lin, T.-Y., and Le, Q. V. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In CVPR , 2019.
  24. Goodfellow, I., Bengio, Y., and Courville, A. Deep learning . MIT press, 2016.
  25. Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. arXiv preprint arXiv:1410.5401 , 2014. · arXiv
  26. Hazan, E., Levy, K., and Shalev-Shwartz, S. Beyond convexity: Stochastic quasi-convex optimization. In Advances in Neural Information Processing Systems , pp. 1594-1602, 2015.
  27. He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV , 2015.
  28. Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., et al. Population based training of neural networks. arXiv , 2017.
  29. Jayakumar, S. M., Menick, J., Czarnecki, W. M., Schwarz, J., Rae, J., Osindero, S., Teh, Y. W., Harley, T., and Pascanu, R. Multiplicative interactions and where to find them. In ICLR , 2020.
  30. Kim, M. and Rigazio, L. Deep clustered convolutional kernels. arXiv , 2015.
  31. Koza, J. R. and Koza, J. R. Genetic programming: on the programming of computers by means of natural selection . MIT press, 1992.
  32. Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Master's thesis, Dept. of Computer Science, U. of Toronto , 2009.
  33. Lenat, D. B. Eurisko: a program that learns new heuristics and domain concepts: the nature of heuristics iii: program design and results. Artificial intelligence , 21(1-2):61-98, 1983.
  34. Levy, K. Y. The power of normalization: Faster evasion of saddle points. arXiv preprint arXiv:1611.04831 , 2016. · arXiv
  35. Li, K. and Malik, J. Learning to optimize. ICLR , 2017.
  36. Li, L. and Talwalkar, A. Random search and reproducibility for neural architecture search. CoRR , abs/1902.07638, 2019. URL http://arxiv.org/ abs/1902.07638 . · arXiv
  37. Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. Hyperband: A novel bandit-based approach to hyperparameter optimization. JMLR , 2018.
  38. Liang, C., Berant, J., Le, Q. V., Forbus, K. D., and Lao, N. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In ACL , 2016.
  39. Liang, C., Norouzi, M., Berant, J., Le, Q. V., and Lao, N. Memory augmented policy optimization for program synthesis and semantic parsing. In NeurIPS , 2018.
  40. Liu, C., Zoph, B., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., and Murphy, K. Progressive neural architecture search. ECCV , 2018.
  41. Liu, C., Chen, L.-C., Schroff, F., Adam, H., Hua, W., Yuille, A. L., and Fei-Fei, L. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In CVPR , 2019a.
  42. Liu, H., Simonyan, K., and Yang, Y. Darts: Differentiable architecture search. ICLR , 2019b.
  43. Loshchilov, I. and Hutter, F. Cma-es for hyperparameter optimization of deep neural networks. arXiv preprint arXiv:1604.07269 , 2016. · arXiv
  44. Mei, J., Li, Y., Lian, X., Jin, X., Yang, L., Yuille, A., and Yang, J. Atomnas: Fine-grained end-to-end neural architecture search. ICLR , 2020.
  45. Mendoza, H., Klein, A., Feurer, M., Springenberg, J. T., and Hutter, F. Towards automatically-tuned neural networks. In Workshop on Automatic Machine Learning , 2016.
  46. Metz, L., Maheswaranathan, N., Cheung, B., and SohlDickstein, J. Meta-learning update rules for unsupervised representation learning. In ICLR , 2019.
  47. Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., et al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing . Elsevier, 2019.
  48. Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In ICML , 2010.
  49. Neelakantan, A., Le, Q. V., and Sutskever, I. Neural programmer: Inducing latent programs with gradient descent. CoRR , abs/1511.04834, 2015. · arXiv
  50. Negrinho, R., Gormley, M., Gordon, G. J., Patil, D., Le, N., and Ferreira, D. Towards modular and programmable architecture search. In NeurIPS , 2019.
  51. Noy, A., Nayman, N., Ridnik, T., Zamir, N., Doveh, S., Friedman, I., Giryes, R., and Zelnik-Manor, L. Asap: Architecture search, anneal and prune. arXiv , 2019.
  52. Parisotto, E., rahman Mohamed, A., Singh, R., Li, L., Zhou, D., and Kohli, P. Neuro-symbolic program synthesis. ArXiv , abs/1611.01855, 2016. · arXiv
  53. Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., and Le, Q. V. Specaugment: A simple data augmentation method for automatic speech recognition. Proc. Interspeech , 2019.
  54. Pitrat, J. Implementation of a reflective system. Future Generation Computer Systems , 12(2-3):235-242, 1996.
  55. Polozov, O. and Gulwani, S. Flashmeta: a framework for inductive program synthesis. In OOPSLA 2015 , 2015.
  56. Polyak, B. T. and Juditsky, A. B. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization , 30(4):838-855, 1992.
  57. Ramachandran, P., Zoph, B., and Le, Q. Searching for activation functions. ICLR Workshop , 2017.
  58. Ravi, S. and Larochelle, H. Optimization as a model for few-shot learning. ICLR , 2017.
  59. Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y. L., Le, Q., and Kurakin, A. Large-scale evolution of image classifiers. In ICML , 2017.
  60. Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. Regularized evolution for image classifier architecture search. AAAI , 2019.
  61. Reed, S. E. and de Freitas, N. Neural programmerinterpreters. CoRR , abs/1511.06279, 2015. · arXiv
  62. Risi, S. and Stanley, K. O. Indirectly encoding neural plasticity as a pattern of local rules. In International Conference on Simulation of Adaptive Behavior , 2010.
  63. Schmidhuber, J. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta... hook . PhD thesis, Technische Universität München, 1987.
  64. Schmidhuber, J. Optimal ordered problem solver. Machine Learning , 54(3):211-254, 2004.
  65. So, D. R., Liang, C., and Le, Q. V. The evolved transformer. In ICML , 2019.
  66. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. JMLR , 2014.
  67. Sun, Y., Xue, B., Zhang, M., and Yen, G. G. Evolving deep convolutional neural networks for image classification. IEEE Transactions on Evolutionary Computation , 2019.
  68. Valkov, L., Chaudhari, D., Srivastava, A., Sutton, C. A., and Chaudhuri, S. Houdini: Lifelong learning as program synthesis. In NeurIPS , 2018.
  69. Wichrowska, O., Maheswaranathan, N., Hoffman, M. W., Colmenarejo, S. G., Denil, M., de Freitas, N., and SohlDickstein, J. Learned optimizers that scale and generalize. ICML , 2017.
  70. Wilson, D. G., Cussat-Blanc, S., Luga, H., and Miller, J. F. Evolving simple programs for playing atari games. In Proceedings of the Genetic and Evolutionary Computation Conference , pp. 229-236, 2018.
  71. Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017. · arXiv
  72. Xie, L. and Yuille, A. Genetic CNN. In ICCV , 2017.
  73. Xie, S., Kirillov, A., Girshick, R., and He, K. Exploring randomly wired neural networks for image recognition. In Proceedings of the IEEE International Conference on Computer Vision , pp. 1284-1293, 2019.
  74. Yang, A., Esperança, P. M., and Carlucci, F. M. Nas evaluation is frustratingly hard. ICLR , 2020.
  75. Yao, Q., Wang, M., Chen, Y., Dai, W., Yi-Qi, H., Yu-Feng, L., Wei-Wei, T., Qiang, Y., and Yang, Y. Taking human out of learning applications: A survey on automated machine learning. arXiv , 2018.
  76. Yao, X. Evolving artificial neural networks. IEEE , 1999.
  77. Ying, C., Klein, A., Real, E., Christiansen, E., Murphy, K., and Hutter, F. Nas-bench-101: Towards reproducible neural architecture search. ICML , 2019.
  78. Zela, A., Klein, A., Falkner, S., and Hutter, F. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. ICML AutoML Workshop , 2018.
  79. Zhong, Z., Yan, J., Wu, W., Shao, J., and Liu, C.-L. Practical block-wise neural network architecture generation. In CVPR , 2018.
  80. Zoph, B. and Le, Q. V. Neural architecture search with reinforcement learning. In ICLR , 2016.
  81. Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. Learning transferable architectures for scalable image recognition. In CVPR , 2018.

Cited by

Sorted by citations

Authors
Real, E., Liang, C., So, D. R., & Le, Q. V.
ORCID
Le, Quoc V.
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
8007-8019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2003.03384
arXiv
2003.03384
OpenAlex
W3010604601
Cited by
143
References
6
Citations by year
2025: 10 · 2024: 25 · 2023: 32 · 2022: 20 · 2021: 38 · 2020: 18
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords
automated machine learning, algorithm discovery, evolutionary search, neural architecture search, two-layer neural networks, bilinear interaction, weight averaging, dropout, CIFAR-10

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review