CoChat
Sign Up
PreprintCited by 33Machine Learning and Data ClassificationAdvanced Neural Network ApplicationsAdvanced Bandit Algorithms Research

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search

Open access
White, Colin, Neiswanger, Willie, Savani, Yash·2019·arXiv
DOI 10.48550/arxiv.1910.11858arXiv 1910.11858
Open this paper in CoChat — free accountOpen PDF

Abstract

Over the past half-decade, many methods have been considered for neural architecture search (NAS). Bayesian optimization (BO), which has long had success in hyperparameter optimization, has recently emerged as a very promising strategy for NAS when it is coupled with a neural predictor. Recent work has proposed different instantiations of this framework, for example, using Bayesian neural networks or graph convolutional networks as the predictive model within BO. However, the analyses in these papers often focus on the full-fledged NAS algorithm, so it is difficult to tell which individual components of the framework lead to the best performance. In this work, we give a thorough analysis of the "BO + neural predictor" framework by identifying five main components: the architecture encoding, neural predictor, uncertainty calibration method, acquisition function, and acquisition optimization strategy. We test several different methods for each component and also develop a novel path-based encoding scheme for neural architectures, which we show theoretically and empirically scales better than other encodings. Using all of our analyses, we develop a final algorithm called BANANAS, which achieves state-of-the-art performance on NAS search spaces. We adhere to the NAS research checklist (Lindauer and Hutter 2019) to facilitate best practices, and our code is available at https://github.com/naszilla/naszilla.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1910.11858

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 76

  1. [1][1] Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprint arXiv:1705.10823 , 2017. - · arXiv
  2. [2][2] Maximilian Balandat, Brian Karrer, Daniel R Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. Botorch: Programmable bayesian optimization in pytorch. arXiv preprint arXiv:1910.06403 , 2019. - · arXiv
  3. [3][3] William H Beluch, Tim Genewein, Andreas N¨ urnberger, and Jan M K¨ ohler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. -
  4. [4][4] James S Bergstra, R´ emi Bardenet, Yoshua Bengio, and Bal´ azs K´ egl. Algorithms for hyperparameter optimization. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS) , 2011. -
  5. [5][5] Youngwon Choi, Yongchan Kwon, Hanbyul Lee, Beom Joon Kim, Myunghee Cho Paik, and Joong-Ho Won. Ensemble of deep convolutional neural networks for prognosis of ischemic stroke. In International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries , 2016. -
  6. [6][6] Boyang Deng, Junjie Yan, and Dahua Lin. Peephole: Predicting network performance before training. arXiv preprint arXiv:1712.03351 , 2017. - · arXiv
  7. [7][7] Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-Fourth International Joint Conference on Artificial Intelligence , 2015. -
  8. [8][8] Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In Proceedings of the IEEE Conference on computer vision and pattern recognition , pages 1761-1770, 2019. -
  9. [9][9] Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020. -
  10. [10][10] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. arXiv preprint arXiv:1808.05377 , 2018. - · arXiv
  11. [11][11] Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. In Proceedings of the International Conference on Machine Learning (ICML) , 2018. -
  12. [12][12] Dario Floreano, Peter D¨ urr, and Claudio Mattiussi. Neuroevolution: from architectures to learning. Evolutionary intelligence , 1(1):47-62, 2008. -
  13. [13][13] Peter I Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018. - · arXiv
  14. [14][14] Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1487-1495. ACM, 2017. -
  15. [15][15] Javier Gonz´ alez, Zhenwen Dai, Philipp Hennig, and Neil Lawrence. Batch bayesian optimization via local penalization. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , 2016. -
  16. [16][16] Brent Hecht, Lauren Wilcox, Jeffrey P Bigham, Johannes Sch¨ oning, Ehsan Hoque, Jason Ernst, Yonatan Bisk, Luigi De Russis, Lana Yarosh, Bushra Anjum, Danish Contractor, and Cathy Wu. It's time to do something: Mitigating the negative impacts of computing through a change to the peer review process. ACM Future of Computing Blog , 2018. -
  17. [17][17] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017. -
  18. [18][18] Roxana Istrate, Florian Scheidegger, Giovanni Mariani, Dimitrios Nikolopoulos, Costas Bekas, and A Cristiano I Malossi. Tapas: Train-less accuracy predictor for architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence , 2019. -
  19. [19][19] Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: Efficient neural architecture search with network morphism. arXiv preprint arXiv:1806.10282 , 2018. - · arXiv
  20. [20][20] Kirthevasan Kandasamy, Akshay Krishnamurthy, Jeff Schneider, and Barnab´ as P´ oczos. Parallelised bayesian optimisation via thompson sampling. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , 2018. -
  21. [21][21] Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems , pages 2016-2025, 2018. -
  22. [22][22] Hiroaki Kitano. Designing neural networks using genetic algorithms with graph generation system. Complex systems , 4(4):461-476, 1990. -
  23. [23][23] Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. ICLR 2017 , 2017. -
  24. [24][24] Chepuri Shri Krishna, Ashish Gupta, Himanshu Rai, and Swarnim Narayan. Neural architecture search with reinforce and masked attention autoregressive density estimators. arXiv preprint arXiv:2006.00939 , 2020. - · arXiv
  25. [25][25] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS) , 2012. -
  26. [26][26] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. arXiv preprint arXiv:1807.00263 , 2018. - · arXiv
  27. [27][27] Harold J Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering , 86(1):97-106, 1964. -
  28. [28][28] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems , pages 6402-6413, 2017. -
  29. [29][29] Kevin Alexander Laube and Andreas Zell. Prune and replace nas. arXiv preprint arXiv:1906.07528 , 2019. - · arXiv
  30. [30][30] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. arXiv preprint arXiv:1902.07638 , 2019. - · arXiv
  31. [31][31] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv:1603.06560 , 2016. - · arXiv
  32. [32][32] Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. arXiv preprint arXiv:1909.06035 , 2019. - · arXiv
  33. [33][33] Marius Lindauer and Frank Hutter. Best practices for scientific research on neural architecture search. arXiv preprint arXiv:1909.02453 , 2019. - · arXiv
  34. [34][34] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 19-34, 2018. -
  35. [35][35] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 , 2018. - · arXiv
  36. [36][36] Lizheng Ma, Jiaxu Cui, and Bo Yang. Deep neural architecture search with deep graph bayesian optimization. In 2019 IEEE/WIC/ACM International Conference on Web Intelligence (WI) , pages 500-507. IEEE, 2019. -
  37. [37][37] Krzysztof Maziarz, Andrey Khorlin, Quentin de Laroussilhe, and Andrea Gesmundo. Evolutionary-neural hybrid agents for architecture search. arXiv preprint arXiv:1811.09828 , 2018. - · arXiv
  38. [38][38] Jonas Moˇ ckus. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference , pages 400-404. Springer, 1975. -
  39. [39][39] Willie Neiswanger, Kirthevasan Kandasamy, Barnabas Poczos, Jeff Schneider, and Eric Xing. Probo: a framework for using probabilistic programming in bayesian optimization. arXiv preprint arXiv:1901.11515 , 2019. - · arXiv
  40. [40][40] Vu Nguyen, Tam Le, Makoto Yamada, and Michael A Osborne. Optimal transport kernels for sequential and parallel neural architecture search. arXiv preprint arXiv:2006.07593 , 2020. - · arXiv
  41. [41][41] Jiˇ r´ ı Oˇ cen´ aˇ sek and Josef Schwarz. The parallel bayesian optimization algorithm. In The State of the Art in Computational Intelligence . Springer, 2000. -
  42. [42][42] Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 , 2018. - · arXiv
  43. [43][43] Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning , pages 63-71. Springer, 2003. -
  44. [44][44] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence , volume 33, pages 4780-4789, 2019. -
  45. [45][45] Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne. Neural architecture search using bayesian optimisation with weisfeiler-lehman kernel. arXiv preprint arXiv:2006.07556 , 2020. - · arXiv
  46. [46][46] Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. arXiv preprint arXiv:1902.08142 , 2019. - · arXiv
  47. [47][47] Syed Asif Raza Shah, Wenji Wu, Qiming Lu, Liang Zhang, Sajith Sasidharan, Phil DeMar, Chin Guok, John Macauley, Eric Pouyoul, Jin Kim, et al. Amoebanet: An sdn-enabled network service for big data science. Journal of Network and Computer Applications , 119:70-82, 2018. -
  48. [48][48] Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James T Kwok, and Tong Zhang. Multi-objective neural architecture search via predictive network performance optimization. arXiv preprint arXiv:1911.09336 , 2019. - · arXiv
  49. [49][49] Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.09777 , 2020. - · arXiv
  50. [50][50] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS) , 2012. -
  51. [51][51] Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D Sculley, Joshua Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS) , 2019. -
  52. [52][52] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning , pages 2171-2180, 2015. -
  53. [53][53] Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In Advances in Neural Information Processing Systems , pages 4134-4142, 2016. -
  54. [54][54] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995 , 2009. - · arXiv
  55. [55][55] Pantelimon Stanica. Good lower and upper bounds on binomial coefficients. Journal of Inequalities in Pure and Applied Mathematics , 2001. -
  56. [56][56] Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation , 10(2):99-127, 2002. -
  57. [57][57] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI Conference on Artificial Intelligence , 2017. -
  58. [58][58] Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946 , 2019. - · arXiv
  59. [60][60] Kevin Tran, Willie Neiswanger, Junwoong Yoon, Qingyang Zhang, Eric Xing, and Zachary W Ulissi. Methods for comparing uncertainty quantifications for material property predictions. Machine Learning: Science and Technology , 1(2):025006, 2020. -
  60. [61][61] Linnan Wang, Saining Xie, Teng Li, Rodrigo Fonseca, and Yuandong Tian. Sample-efficient neural architecture search by learning action space. arXiv preprint arXiv:1906.06832 , 2019. - · arXiv
  61. [62][62] Linnan Wang, Yiyang Zhao, Yuu Jinnai, and Rodrigo Fonseca. Alphax: exploring neural architectures with deep neural networks and monte carlo tree search. arXiv preprint arXiv:1805.07440 , 2018. - · arXiv
  62. [63][63] Chen Wei, Chuang Niu, Yiping Tang, and Jimin Liang. Npenas: Neural predictor guided evolution for neural architecture search. arXiv preprint arXiv:2003.12857 , 2020. - · arXiv
  63. [64][64] Wei Wen, Hanxiao Liu, Hai Li, Yiran Chen, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. arXiv preprint arXiv:1912.00848 , 2019. - · arXiv
  64. [65][65] Colin White, Willie Neiswanger, Sam Nolen, and Yash Savani. A study on encodings for neural architecture search. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS) , 2020. -
  65. [66][66] Colin White, Sam Nolen, and Yash Savani. Local search is state of the art for nas benchmarks. arXiv preprint arXiv:2005.02960 , 2020. - · arXiv
  66. [67][67] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , pages 229-256, 1992. -
  67. [68][68] James Wilson, Frank Hutter, and Marc Deisenroth. Maximizing acquisition functions for bayesian optimization. In Advances in Neural Information Processing Systems , pages 98849895, 2018. -
  68. [69][69] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search. In International Conference on Learning Representations , 2019. -
  69. [70][70] Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? arXiv preprint arXiv:2006.06936 , 2020. - · arXiv
  70. [71][71] Antoine Yang, Pedro M Esperan¸ ca, and Fabio M Carlucci. Nas evaluation is frustratingly hard. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020. -
  71. [73][73] Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, and Yee Whye Teh. Neural ensemble search for performant and calibrated predictions. arXiv preprint arXiv:2006.08573 , 2020. - · arXiv
  72. [74][74] Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018. -
  73. [75][75] Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-vae: A variational autoencoder for directed acyclic graphs. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS) , 2019. -
  74. [77][77] Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for neural architecture search. arXiv preprint arXiv:1905.04919 , 2019. - · arXiv
  75. [78][78] Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph neural networks: A review of methods and applications. arXiv preprint arXiv:1812.08434 , 2018. - · arXiv
  76. [79][79] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR) , 2017.

Cited by

See all 33

Metadata

Authors
White, C., Neiswanger, W., & Savani, Y.
Affiliations
  • Carnegie Mellon University (US)
  • Stanford University (US)
ORCID
Neiswanger, Willie
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
10293-10301
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1910.11858
arXiv
1910.11858
OpenAlex
W3176084580
Cited by
33
References
1
Citations by year
2026: 1 · 2025: 1 · 2023: 3 · 2022: 4 · 2021: 17 · 2020: 7
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • TIB Data Manager
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
Keywords
neural architecture search, Bayesian optimization, uncertainty calibration, acquisition functions, acquisition optimization, Bayesian neural networks, graph convolutional networks, banana

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review