LLMatic: Neural Architecture Search Via Large Language Models And Quality Diversity Optimization
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have emerged as powerful tools capable of accomplishing a broad spectrum of tasks. Their abilities span numerous areas, and one area where they have made a significant impact is in the domain of code generation. Here, we propose using the coding abilities of LLMs to introduce meaningful variations to code defining neural networks. Meanwhile, Quality-Diversity (QD) algorithms are known to discover diverse and robust solutions. By merging the code-generating abilities of LLMs with the diversity and robustness of QD solutions, we introduce LLMatic, a Neural Architecture Search (NAS) algorithm. While LLMs struggle to conduct NAS directly through prompts, LLMatic uses a procedural approach, leveraging QD for prompts and network architecture to create diverse and high-performing networks. We test LLMatic on the CIFAR-10 and NAS-bench-201 benchmarks, demonstrating that it can produce competitive networks while evaluating just 2, 000 candidates, even without prior knowledge of the benchmark domain or exposure to any previous top-performing models for the benchmark. The open-sourced code is available at https://github.com/umair-nasir14/LLMatic.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“The intermediate residual block is the basic residual block with a stride of 2 (He et al., 2016), which serves to downsample the spatial size and double the channels of an input feature map.”
BackgroundVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Transformers [6] facilitated the explosion of LLMs [7, 8].”
BackgroundImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“The specified cell within the search domain is depicted as a densely connected directed acyclic graph with four nodes and six ImageNet16-120 is a variant of ImageNet dataset [38] which is down-sampled to 16x16 image sizes and contains the first 120 classes.”
Multidimensional binary search trees used for associative searching
Bentley, J. L. (1975) · Communications of the ACM · doi:10.1145/361002.361007 · Cited by 7,520
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795 · cited 1×
“The networks are optimized by stochastic gradient descent [21] with the learning rate set to 0.”
BackgroundMnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416 · cited 2×
“…a fixed number of nodes/layers and edges/operations) and the corresponding test accuracy of each architecture after training on a given dataset (e.g. CIFAR-10), allowing researchers to explore the tradeoffs of various NAS algorithms without needing to retrain each candidate network during search.”
BackgroundMethodEvolving Neural Networks through Augmenting Topologies
Stanley, K. O., & Miikkulainen, R. (2002) · Evolutionary Computation · doi:10.1162/106365602320169811 · Cited by 3,258 · cited 1×
“LLMatic begins its search with a very basic neural network, inspired by the work of Stanley [15] which suggests that neuroevolution tends to perform better when starting with a small network.”
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248 · cited 1×
“We show empirically that LLMatic can find more than 20 competitive networks in CIFAR-10 and near-to-optimal networks in NAS-bench-201, using only 2000 evaluations.”
MethodTorrey, L., & Shavlik, J. (2010) · Handbook of Research on Machine Learning Applications and Trends · doi:10.4018/978-1-60566-766-9.ch011 · Cited by 1,503
Research works printed but not linked: 9
- Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901.
- Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. pmlr, 448--456.
- Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. 2018. Neural architecture search with bayesian optimisation and optimal transport. Advances in neural information processing systems 31 (2018).
- 10.5555/3104322.3104425 · doi:10.5555/3104322.3104425
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
- Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. PMLR, 6105--6114.
- Manoel Tenorio and Wei-Tsih Lee. 1988. Self organizing neural networks for the identification problem. Advances in Neural Information Processing Systems 1 (1988).
- Petar Veličković, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. 2022. The CLRS algorithmic reasoning benchmark. In International Conference on Machine Learning. PMLR, 22084--22102.
- Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. 2019. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning. PMLR, 7105--7114.
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544
“To surpass the ‘nominal’ capabilities of LLMs, recent studies 3 have combined them with evolutionary algorithms 14,15 , leading to important improvements on diverse synthetic problems 16 , searching for neural network architectures 17–19 and solving puzzles 20 .”
BackgroundWhen large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367
“LLMatic [177] and EvoPrompting [178] use codeLLMs as mutation and crossover operators for a genetic NAS algorithm.”
Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap
Wu, X., Wu, S.-H., Wu, J., Feng, L., & Tan, K. C. (2024) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2024.3506731 · Cited by 330
“2) With Code Generation Capability of General LLM: Nasir et al. [126] leveraged the code generation ability of large models to help NAS.”
Algorithm Evolution Using Large Language ModelPreprint
Liu, F., Tong, X., Yuan, M., & Zhang, Q. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.18894.43847 · Cited by 164
Metadata
- Authors
- Nasir, M. U., Earle, S., Togelius, J., James, S., & Cleghorn, C.
- Affiliations
- University of the Witwatersrand (ZA)
- Amazon (United States) (US)
- Venue
- Proceedings of the Genetic and Evolutionary Computation Conference
- Publisher
- ACM
- Published
- 2024-07-14
- Pages
- 1110-1118
- OpenAlex
- W4400413686
- Cited by
- 52
- References
- 18
- Citations by year
- 2026: 21 · 2025: 18 · 2024: 11
- FWCI
- 14.76
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Genetic and Evolutionary Computation Conference (GECCO) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Center for High Performance Computing
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- neural architecture search, large language models, code generation, procedural content generation, robustness, NAS-Bench-201, CIFAR-10
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: LLMatic: Neural Architecture Search Via Large Language Models And Quality Diversity Optimization. https://app.cochat.ai/papers/10.1145/3638529.3654017
- The paper
- Nasir, M. U., Earle, S., Togelius, J., James, S., & Cleghorn, C. (2024). LLMatic: Neural Architecture Search Via Large Language Models And Quality Diversity Optimization. Proceedings of the Genetic and Evolutionary Computation Conference. https://doi.org/10.1145/3638529.3654017
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.