LLMatic: Neural Architecture Search Via Large Language Models And Quality Diversity Optimization
Open accessThe DOI matches its Crossref record; 37 of the 42 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have emerged as powerful tools capable of accomplishing a broad spectrum of tasks. Their abilities span numerous areas, and one area where they have made a significant impact is in the domain of code generation. Here, we propose using the coding abilities of LLMs to introduce meaningful variations to code defining neural networks. Meanwhile, Quality-Diversity (QD) algorithms are known to discover diverse and robust solutions. By merging the code-generating abilities of LLMs with the diversity and robustness of QD solutions, we introduce LLMatic, a Neural Architecture Search (NAS) algorithm. While LLMs struggle to conduct NAS directly through prompts, LLMatic uses a procedural approach, leveraging QD for prompts and network architecture to create diverse and high-performing networks. We test LLMatic on the CIFAR-10 and NAS-bench-201 benchmarks, demonstrating that it can produce competitive networks while evaluating just 2, 000 candidates, even without prior knowledge of the benchmark domain or exposure to any previous top-performing models for the benchmark. The open-sourced code is available at https://github.com/umair-nasir14/LLMatic.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“The intermediate residual block is the basic residual block with a stride of 2 (He et al., 2016), which serves to downsample the spatial size and double the channels of an input feature map.”
BackgroundVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Transformers [6] facilitated the explosion of LLMs [7, 8].”
BackgroundImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“The specified cell within the search domain is depicted as a densely connected directed acyclic graph with four nodes and six ImageNet16-120 is a variant of ImageNet dataset [38] which is down-sampled to 16x16 image sizes and contains the first 120 classes.”
Multidimensional binary search trees used for associative searching
Bentley, J. L. (1975) · Communications of the ACM · doi:10.1145/361002.361007 · Cited by 7,520
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795 · cited 1×
“The networks are optimized by stochastic gradient descent [21] with the learning rate set to 0.”
BackgroundMnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416 · cited 2×
“…a fixed number of nodes/layers and edges/operations) and the corresponding test accuracy of each architecture after training on a given dataset (e.g. CIFAR-10), allowing researchers to explore the tradeoffs of various NAS algorithms without needing to retrain each candidate network during search.”
BackgroundMethodEvolving Neural Networks through Augmenting Topologies
Stanley, K. O., & Miikkulainen, R. (2002) · Evolutionary Computation · doi:10.1162/106365602320169811 · Cited by 3,258 · cited 1×
“LLMatic begins its search with a very basic neural network, inspired by the work of Stanley [15] which suggests that neuroevolution tends to perform better when starting with a small network.”
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248 · cited 1×
“We show empirically that LLMatic can find more than 20 competitive networks in CIFAR-10 and near-to-optimal networks in NAS-bench-201, using only 2000 evaluations.”
MethodTorrey, L., & Shavlik, J. (2010) · Handbook of Research on Machine Learning Applications and Trends · doi:10.4018/978-1-60566-766-9.ch011 · Cited by 1,503
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“Apart from creating state-of-the-art models in core natural language processing tasks, they led to creating models for a wide variety of other tasks, such as generating video game levels and code (Chen et al. 2021; Todd et al. 2023).”
BackgroundReflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Robots that can adapt like animals
Cully, A., Clune, J., Tarapore, D., & Mouret, J.-B. (2015) · Nature · doi:10.1038/nature14422 · Cited by 979 · cited 1×
“QD methods are valuable in domains such as robot control, where it is useful to learn diverse high-quality trajectories, in case one solution should become unavailable during deployment because of a physical obstruction or mechanical malfunction [9].”
MethodNAS-Bench-201: Extending the Scope of Reproducible Neural Architecture SearchPreprint
Dong, X., & Yang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00326 · Cited by 798
Designing neural networks using genetic algorithms
Miller, G. F., Todd, P. M., & Hegde, S. S. (1989) · international conference on Genetic algorithms · Cited by 737
A Survey on Evolutionary Neural Architecture Search
Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G. G., & Tan, K. C. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3100554 · Cited by 644 · cited 2×
“Evolutionary methods can also be used to search directly through the space of possible architectures (Liu et al., 2021).”
BackgroundMethodEuclidean Distance Matrices: Essential theory, algorithms, and applications
Dokmanic, I., Parhizkar, R., Ranieri, J., & Vetterli, M. (2015) · IEEE Signal Processing Magazine · doi:10.1109/msp.2015.2398954 · Cited by 611 · cited 1×
“We use k-d tree (Bentley, 1975) to create and write centroids to the archive and find the nearest neighbors using a Euclidean distance metric (Dokmanic et al., 2015).”
MethodQuality Diversity: A New Frontier for Evolutionary Computation
Pugh, J. K., Soros, L. B., & Stanley, K. O. (2016) · Frontiers in Robotics and AI · doi:10.3389/frobt.2016.00040 · Cited by 514 · cited 2×
“To enable this, the solution we propose here leverages quality-diversity search [36], specifically a version of the MAP-Elites algorithm [28].”
MethodThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 1×
“1 Billion parameter variant of CodeGen-Mono due to computational constraints.”
BackgroundNeural Architecture Search: Insights from 1000 PapersPreprint
White, C., Safari, M., Sukthanker, R. S., Ru, B., Elsken, T., Zela, A., Dey, D., & Hutter, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.08727 · Cited by 389 · cited 1×
“Neural Architecture Search (NAS) is the generic term for various approaches to automating this design process [50].”
BackgroundIlluminating search spaces by mapping elites
Mouret, J.-B., & Clune, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.04909 · Cited by 386
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
Chrabąszcz, P., Loshchilov, I., & Hutter, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.08819 · Cited by 336 · cited 1×
“After we generate the network, we mutate the temperature by adding 0 .”
MethodCodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · cited 1×
“Mutate the temperature for the next generation .”
BackgroundQuality and Diversity Optimization: A Unifying Modular Framework
Cully, A., & Demiris, Y. (2018) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2017.2704781 · Cited by 258
Reinforcement learning for neural architecture search: A review
Jaafra, Y., Luc Laurent, J., Deruyver, A., & Saber Naceur, M. (2019) · Image and Vision Computing · doi:10.1016/j.imavis.2019.06.005 · Cited by 183 · cited 2×
“Alternatively, reinforcement learning can be used to train an agent (usually another neural network) to output candidate architectures for a given task, with the performance after training of the candidate architecture acting as a reward signal [20].”
Vassiliades, V., Chatzilygeroudis, K., & Mouret, J.-B. (2018) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2017.2735550 · Cited by 151 · cited 1×
“Mutate the temperature for the next generation .”
BackgroundElskenThomas, Metzen, J. H., & HutterFrank (2019) · Journal of Machine Learning Research · doi:10.5555/3322706.3361996 · Cited by 95
Level Generation Through Large Language Models
Todd, G., Earle, S., Nasir, M. U., Green, M. C., & Togelius, J. (2023) · Proceedings of the 18th International Conference on the Foundations of Digital Games · doi:10.1145/3582437.3587211 · Cited by 90 · cited 1×
“2022; Nasir and Mchechesi 2022), they led to creating models for a wide variety of other tasks, such as generating video game levels and code (Chen et al. 2021; Todd et al. 2023; Nasir and Togelius 2023).”
Self-citationEvolution through Large Models
Lehman, J., Gordon, J., Jain, S., Ndousse, K., Yeh, C., & Stanley, K. O. (2022) · arXiv · doi:10.48550/arxiv.2206.08896 · Cited by 74 · cited 2×
“Evolution through Large Models (ELM) (Lehman et al., 2022) has introduced evolutionary operators through LLMs and MAP-Elites (Mouret & Clune, 2015) to evolve robots morphology at a code level.”
BackgroundNAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly EasyPreprint
Mehta, Y., White, C., Zela, A., Krishnakumar, A., Zabergja, G., Moradian, S., Safari, M., Yu, K., & Hutter, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.13396 · Cited by 61 · cited 1×
“13 Get the network and prompt individuals and store them.”
BackgroundAre quality diversity algorithms better at generating stepping stones than objective-based search?
Gaier, A., Asteroth, A., & Mouret, J.-B. (2019) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3319619.3321897 · Cited by 49 · cited 1×
“Here, maintaining a diversity of fit individuals may protect the population from falling into local optima [15].”
EvoPrompting: Language Models for Code-Level Neural Architecture Search
Chen, A., Dohan, D. M., & So, D. R. (2023) · arXiv · doi:10.48550/arxiv.2302.14838 · Cited by 45 · cited 1×
“EvoPrompting [11] is a method that similarly uses code-LLMs as mutation and crossover operators in order to perform NAS for the MNIST-1D classification task [12] and the CLRS algorithmic reasoning benchmark [13].”
MethodΛ-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among CellsPreprint
Movahedi, S., Adabinejad, M., Imani, A., Keshavarz, A., Dehghani, M., Shakery, A., & Araabi, B. N. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.07998 · Cited by 35 · cited 2×
“, 2023) as an LLM baseline, Λ-DARTS (Movahedi et al., 2022) as it achieves close to optimal result, where optimal is the maximum test accuracy, and Random Search.”
MethodAdelani, D. I., Alabi, J. O., Fan, A., Kreutzer, J., Shen, X., Reid, M., Ruiter, D., Klakow, D., Nabende, P., Chang, E., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.02022 · Cited by 32 · cited 1×
“Apart from creating state-of-the-art models in core natural language processing tasks (Adelani et al., 2022; Nasir & Mchechesi, 2022), they led to creating models for a wide variety of other tasks, such as generating video game levels and code (Chen et al.”
BackgroundSelf-citationScaling Down Deep Learning with MNIST-1DPreprint
Greydanus, S., & Kobak, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.14439 · Cited by 31
Ayala, H. V. H., Muñoz, D. M., Llanos, C. H., & Coelho, L. D. S. (2017) · Control Engineering Practice · doi:10.1016/j.conengprac.2016.12.004 · Cited by 19
Can GPT-4 Perform Neural Architecture Search?
Zheng, M., Su, X., You, S., Wang, F., Qian, C., Xu, C., & Albanie, S. (2023) · arXiv · doi:10.48550/arxiv.2304.10970 · Cited by 15
Ullah, S., Salam, M. A., & Masood, M. (2022) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v25.i2.pp1208-1216 · Cited by 7 · cited 1×
“…a fixed number of nodes/layers and edges/operations) and the corresponding test accuracy of each architecture after training on a given dataset (e.g. CIFAR-10), allowing researchers to explore the tradeoffs of various NAS algorithms without needing to retrain each candidate network during search.”
BackgroundPractical PCG Through Large Language ModelsPreprint
Nasir, M. U., & Togelius, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.18243 · Cited by 1 · cited 1×
“Evolution through Large Models (ELM) [23] casts LLMs as evolutionary operators within a MAP-Elites [28] algorithm tasked with evolving a robot’s morphology at code-level.”
MethodSelf-citationNasir, M. U., & Mchechesi, I. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.08621
Self-citationAugmentative Topology Agents For Open-Ended LearningPreprint
Nasir, M. U., Beukman, M., James, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.11442 · cited 1×
“With this, LLMatic can be used in tasks like Open-ended Learning as well (Nasir et al., 2022).”
MethodSelf-citationFinding Competitive Network Architectures Within a Day Using UCT
Wistuba, M. (2017) · arXiv · doi:10.48550/arxiv.1712.07420
Research works printed but not linked: 10
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
- Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. pmlr, 448-456.
- Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. 2018. Neural architecture search with bayesian optimisation and optimal transport. Advances in neural information processing systems 31 (2018).
- Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve re- stricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10). 807-814. · doi:10.5555/3104322.3104425
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
- Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. PMLR, 6105-6114.
- Manoel Tenorio and Wei-Tsih Lee. 1988. Self organizing neural networks for the identification problem. Advances in Neural Information Processing Systems 1 (1988).
- Petar Veličković, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. 2022. The CLRS algorithmic reasoning benchmark. In International Conference on Machine Learning. PMLR, 22084-22102.
- Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. 2019. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning. PMLR, 7105-7114.
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544
“To surpass the ‘nominal’ capabilities of LLMs, recent studies 3 have combined them with evolutionary algorithms 14,15 , leading to important improvements on diverse synthetic problems 16 , searching for neural network architectures 17–19 and solving puzzles 20 .”
BackgroundWhen large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367
“LLMatic [177] and EvoPrompting [178] use codeLLMs as mutation and crossover operators for a genetic NAS algorithm.”
Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap
Wu, X., Wu, S.-H., Wu, J., Feng, L., & Tan, K. C. (2024) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2024.3506731 · Cited by 330
“2) With Code Generation Capability of General LLM: Nasir et al. [126] leveraged the code generation ability of large models to help NAS.”
Algorithm Evolution Using Large Language ModelPreprint
Liu, F., Tong, X., Yuan, M., & Zhang, Q. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.18894.43847 · Cited by 164
Metadata
- Authors
- Nasir, M. U., Earle, S., Togelius, J., James, S., & Cleghorn, C.
- Affiliations
- University of the Witwatersrand (ZA)
- Amazon (United States) (US)
- Venue
- Proceedings of the Genetic and Evolutionary Computation Conference
- Publisher
- ACM
- Published
- 2024-07-14
- Pages
- 1110-1118
- OpenAlex
- W4400413686
- Cited by
- 52
- References
- 18
- Citations by year
- 2026: 21 · 2025: 18 · 2024: 11
- FWCI
- 14.76
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Genetic and Evolutionary Computation Conference (GECCO) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Center for High Performance Computing
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- neural architecture search, large language models, code generation, procedural content generation, robustness, NAS-Bench-201, CIFAR-10
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.