CoChat

LLMatic: Neural Architecture Search Via Large Language Models And Quality Diversity Optimization

Open access
Nasir, Muhammad Umair, Earle, Sam, Togelius, Julian, James, Steven, Cleghorn, Christopher · 2024 · Proceedings of the Genetic and Evolutionary Computation Conference

The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.

Abstract

Large language models (LLMs) have emerged as powerful tools capable of accomplishing a broad spectrum of tasks. Their abilities span numerous areas, and one area where they have made a significant impact is in the domain of code generation. Here, we propose using the coding abilities of LLMs to introduce meaningful variations to code defining neural networks. Meanwhile, Quality-Diversity (QD) algorithms are known to discover diverse and robust solutions. By merging the code-generating abilities of LLMs with the diversity and robustness of QD solutions, we introduce LLMatic, a Neural Architecture Search (NAS) algorithm. While LLMs struggle to conduct NAS directly through prompts, LLMatic uses a procedural approach, leveraging QD for prompts and network architecture to create diverse and high-performing networks. We test LLMatic on the CIFAR-10 and NAS-bench-201 benchmarks, demonstrating that it can produce competitive networks while evaluating just 2, 000 candidates, even without prior knowledge of the benchmark domain or exposure to any previous top-performing models for the benchmark. The open-sourced code is available at https://github.com/umair-nasir14/LLMatic.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3638529.3654017

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Illustrated in the figure is the flow of LLMatic. In the initial round of evolution, an initial network with a random prompt goes through a mutation operation. Network individual and prompt individual are then evaluated to be stored in separate archives. During the evolutionary loop, the selected prompt and network go through an evolutionary operation (the prompt is fixed if the operation is crossover) to create more networks and prompt individuals to fill and illuminate the archives.
Figure 1 Illustrated in the figure is the flow of LLMatic. In the initial round of evolution, an initial network with a random prompt goes through a mutation operation. Network individual and prompt individual are then evaluated to be stored in separate archives. During the evolutionary loop, the selected prompt and network go through an evolutionary operation (the prompt is fixed if the operation is crossover) to create more networks and prompt individuals to fill and illuminate the archives.CC-BY · From the PDF
Image 2
CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Image 4
CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Figure 4 The illustration of how many trainable networks are created in a generation. The total number of networks created is 100 per generation. This illustration is calculated over 10 runs. The shaded region is the standard deviation.
Figure 4 The illustration of how many trainable networks are created in a generation. The total number of networks created is 100 per generation. This illustration is calculated over 10 runs. The shaded region is the standard deviation.CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Image 8
CC-BY · From the PDF
Image 9
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 42 references

Research works printed but not linked: 9

  1. Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901.
  2. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. pmlr, 448--456.
  3. Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. 2018. Neural architecture search with bayesian optimisation and optimal transport. Advances in neural information processing systems 31 (2018).
  4. 10.5555/3104322.3104425 · doi:10.5555/3104322.3104425
  5. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
  6. Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. PMLR, 6105--6114.
  7. Manoel Tenorio and Wei-Tsih Lee. 1988. Self organizing neural networks for the identification problem. Advances in Neural Information Processing Systems 1 (1988).
  8. Petar Veličković, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. 2022. The CLRS algorithmic reasoning benchmark. In International Conference on Machine Learning. PMLR, 22084--22102.
  9. Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. 2019. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning. PMLR, 7105--7114.

Cited by

See all 103

Metadata

Authors
Nasir, M. U., Earle, S., Togelius, J., James, S., & Cleghorn, C.
Affiliations
  • University of the Witwatersrand (ZA)
  • Amazon (United States) (US)
Venue
Proceedings of the Genetic and Evolutionary Computation Conference
Publisher
ACM
Published
2024-07-14
Pages
1110-1118
OpenAlex
W4400413686
Cited by
52
References
18
Citations by year
2026: 21 · 2025: 18 · 2024: 11
FWCI
14.76
Type
conference paper
Language
en
Open access
Locations
  • Genetic and Evolutionary Computation Conference (GECCO) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Center for High Performance Computing
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
neural architecture search, large language models, code generation, procedural content generation, robustness, NAS-Bench-201, CIFAR-10

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: LLMatic: Neural Architecture Search Via Large Language Models And Quality Diversity Optimization. https://app.cochat.ai/papers/10.1145/3638529.3654017
The paper
Nasir, M. U., Earle, S., Togelius, J., James, S., & Cleghorn, C. (2024). LLMatic: Neural Architecture Search Via Large Language Models And Quality Diversity Optimization. Proceedings of the Genetic and Evolutionary Computation Conference. https://doi.org/10.1145/3638529.3654017

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.