CoChat
Sign Up
PreprintCited by 74Evolutionary Algorithms and ApplicationsSoftware Engineering ResearchTeaching and Learning Programming

Evolution through Large Models

Open access
Lehman, Joel, Gordon, Jonathan, Jain, Shawn, Ndousse, Kamal, Yeh, Cathy, Stanley, Kenneth O.·2022·arXiv
DOI 10.48550/arxiv.2206.08896arXiv 2206.08896
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper pursues the insight that large language models (LLMs) trained to generate code can vastly improve the effectiveness of mutation operators applied to programs in genetic programming (GP). Because such LLMs benefit from training data that includes sequential changes and modifications, they can approximate likely changes that humans would make. To highlight the breadth of implications of such evolution through large models (ELM), in the main experiment ELM combined with MAP-Elites generates hundreds of thousands of functional examples of Python programs that output working ambulating robots in the Sodarace domain, which the original LLM had never seen in pre-training. These examples then help to bootstrap training a new conditional language model that can output the right walker for a particular terrain. The ability to bootstrap new models that can output appropriate artifacts for a given context in a domain where zero training data was previously available carries implications for open-endedness, deep learning, and reinforcement learning. These implications are explored here in depth in the hope of inspiring new directions of research now opened up by ELM.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.08896

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Promptbreeder makes use of the observation that LLMs can be used to generate variations of input text.

    “Promptbreeder makes use of the observation that LLMs can be used to generate variations of input text (Lehman et al., 2022;Meyerson et al., 2023;Chen et al., 2023).”

    From Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution (2023) · cites this paper as Lehman et al., 2022

  • LLMs are effective at generating mutations from examples.

    “Promptbreeder makes use of the observation by Lehman et al. (2022), Meyerson et al. (2023) and Chen et al. (2023) that LLMs are effective at generating mutations from examples.”

    From Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution (2023) · cites this paper as Lehman et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 64

  1. [1][1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021. - · arXiv
  2. [2][2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020. - · arXiv
  3. [3][3] J. R. Koza. Genetic Programming: On the Programming of Computers by Means of Natural Selection . MIT Press, Cambridge, MA, 1992. -
  4. [4][4] Wolfgang Banzhaf, Peter Nordin, Robert E Keller, and Frank D Francone. Genetic programming: an introduction: on the automatic evolution of computer programs and its applications . Morgan Kaufmann Publishers Inc., 1998. -
  5. [5][5] K.O. Stanley, J Lehman, and L Soros. Open-endedness: The last grand challenge you've never heard of. O'Reilly Online , December 19, 2017. -
  6. [6][6] Russell K Standish. Open-ended artificial evolution. International Journal of Computational Intelligence and Applications , 3(02):167-175, 2003. -
  7. [8][8] Michael O'Neill, Leonardo Vanneschi, Steven Gustafson, and Wolfgang Banzhaf. Open issues in genetic programming. Genetic Programming and Evolvable Machines , 11(3):339-363, 2010. -
  8. [9][9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021. - · arXiv
  9. [10][10] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R´ emi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814 , 2022. - · arXiv
  10. [12][12] Paul Szerlip and Kenneth O. Stanley. Indirectly Encoding Running and Jumping Sodarace Creatures for Artificial Life. Artificial Life , 21(4):432444, 11 2015. ISSN 1064-5462. doi: 10.1162/ARTL-a-00185. URL https: //doi.org/10.1162/ARTL a 00185. - · DOI
  11. [13][13] Mark A Bedau, John S McCaskill, Norman H Packard, Steen Rasmussen, Chris Adami, David G Green, Takashi Ikegami, Kunihiko Kaneko, and Thomas S Ray. Open problems in artificial life. Artificial life , 6(4):363376, 2000. -
  12. [14][14] Kenneth O Stanley, Joel Lehman, and Lisa Soros. Open-endedness: The last grand challenge you've never heard of. O'Reilly Radar Online Article , December 2017. -
  13. [15][15] William B Langdon and Riccardo Poli. Foundations of genetic programming . Springer Science & Business Media, 2013. -
  14. [16][16] John R Koza, Martin A Keane, Matthew J Streeter, William Mydlowec, Jessen Yu, and Guido Lanza. Genetic programming IV: Routine humancompetitive machine intelligence , volume 5. Springer Science & Business Media, 2006. -
  15. [17][17] Markus Brameier and Wolfgang Banzhaf. A comparison of linear genetic programming and neural networks in medical data mining. IEEE Transactions on Evolutionary Computation , 5(1):17-26, 2001. -
  16. [18][18] Istvan Jonyer and Akiko Himes. Improving modularity in genetic programming using graph-based data mining. In FLAIRS Conference , pages 556-561, 2006. -
  17. [19][19] Gregory Seront. External concepts reuse in genetic programming. In working notes for the AAAI Symposium on Genetic programming , pages 94-98. MIT/AAAI Cambridge, 1995. -
  18. [20][20] Leo Gugerty and Gary Olson. Debugging by skilled and novice programmers. In Proceedings of the SIGCHI conference on human factors in computing systems , pages 171-174, 1986. -
  19. [21][21] Paul Luo Li, Andrew J Ko, and Jiamin Zhu. What makes a great software engineer? In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , volume 1, pages 700-710. IEEE, 2015. -
  20. [22][22] Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O Stanley. Safe mutations for deep and recurrent neural networks through output gradients. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 117-124. ACM, 2018. -
  21. [23][23] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning , pages 1889-1897, 2015. -
  22. [24][24] Edgar Galv´ an-L´ opez, James McDermott, Michael O'Neill, and Anthony Brabazon. Towards an understanding of locality in genetic programming. In Proceedings of the 12th annual conference on Genetic and evolutionary computation , pages 901-908, 2010. -
  23. [25][25] Rafal Salustowicz and J¨ urgen Schmidhuber. Probabilistic incremental program evolution. Evolutionary computation , 5(2):123-141, 1997. -
  24. [26][26] Lee Spector and Alan Robinson. Genetic programming and autoconstructive evolution with the push programming language. Genetic Programming and Evolvable Machines , 3(1):7-40, 2002. -
  25. [27][27] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. - · arXiv
  26. [28][28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, glyph[suppress] Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. -
  27. [29][29] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. - · arXiv
  28. [30][30] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 , 2021. - · arXiv
  29. [32][32] Tim Taylor. Exploring the concept of open-ended evolution. In Artificial Life 13 (Proceedings of the Thirteenth International Conference on the Simulation and Synthesis of Living Systems) , pages 540-541, Cambridge, MA, 2012. MIT Press. -
  30. [33][33] Tim Taylor, Mark Bedau, Alastair Channon, David Ackley, Wolfgang Banzhaf, Guillaume Beslon, Emily Dolson, Tom Froese, Simon Hickinbotham, Takashi Ikegami, et al. Open-ended evolution: Perspectives from the oee workshop in york. Artificial life , 22(3):408-423, 2016. -
  31. [34][34] Joel Lehman and Kenneth O. Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation , 19(2): 189-223, 2011. URL http://eplex.cs.ucf.edu/papers/lehman ecj11.pdf. -
  32. [35][35] J.-B. Mouret and St´ ephane Doncieux. Encouraging behavioral diversity in evolutionary robotics: An empirical study. Evolutionary computation , 20 (1):91-133, 2012. -
  33. [36][36] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 2778-2787. PMLR, 2017. -
  34. [37][37] Christopher Stanton and Jeff Clune. Curiosity search: producing generalists by encouraging individuals to continually explore and acquire skills throughout their lifetime. PloS one , 11(9):e0162235, 2016. -
  35. [38][38] Djordje Grbic, Rasmus Berg Palm, Elias Najarro, Claire Glanois, and Sebastian Risi. Evocraft: A new challenge for open-endedness. In International Conference on the Applications of Evolutionary Computation (Part of EvoStar) , pages 325-340. Springer, 2021. -
  36. [39][39] Sam Earle, Julian Togelius, and LB Soros. Video games as a testbed for open-ended phenomena. In 2021 IEEE Conference on Games (CoG) , pages 1-9. IEEE, 2021. -
  37. [40][40] Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth O. Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning , pages 9940-9951. PMLR, 2020. -
  38. [41][41] L. B. Soros and Kenneth O. Stanley. Identifying minimal conditions for open-ended evolution through the artificial life world of chromaria. In Proceedings of the Fourteenth International Conference on the Synthesis and Simulation of Living Systems , pages 793-800, Cambridge, MA, 2014. MIT Press. -
  39. [42][42] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in Neural Information Processing Systems , 33:13049-13061, 2020. -
  40. [43][43] Jonathan C Brant and Kenneth O Stanley. Minimal criterion coevolution: a new approach to open-ended search. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 67-74. ACM, 2017. -
  41. [46][46] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022. - · arXiv
  42. [48][48] Melanie Mitchell. An introduction to genetic algorithms . MIT Press, 1999. -
  43. [49][49] Kenneth A. De Jong. Evolutionary Computation: A unified approach . MIT Press, Cambridge, MA, 2006. -
  44. [50][50] Justin K Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality diversity: A new frontier for evolutionary computation. 3(40), 2016. ISSN 2296-9144. URL http://www.frontiersin.org/evolutionary-robotics/ 10.3389/frobt.2016.00040/abstract. - · DOI
  45. [51][51] Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. ArXiv e-prints , abs/1504.04909, 2015. URL http://arxiv.org/ abs/1504.04909. - · arXiv
  46. [52][52] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Poet: Open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference , GECCO '19, page 142-151, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361118. doi: 10.1145/3321707. 3321799. URL https://doi.org/10.1145/3321707.3321799. - · DOI · Paper page
  47. [53][53] Jonathan C. Brant and Kenneth O. Stanley. Diversity preservation in minimal criterion coevolution through resource limitation. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference , GECCO '20, page 58-66, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450371285. doi: 10.1145/3377930.3389809. URL https://doi.org/10.1145/3377930.3389809. - · DOI
  48. [54][54] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528 , 2019. - · arXiv
  49. [55][55] Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature , 521(7553):503-507, 2015. -
  50. [56][56] Silja Meyer-Nieberg and Hans-Georg Beyer. Self-adaptation in evolutionary algorithms. In Parameter setting in evolutionary algorithms , pages 47-75. Springer, 2007. -
  51. [57][57] Oliver Kramer. Evolutionary self-adaptation: a survey of operators and strategy parameters. Evolutionary Intelligence , 3(2):51-65, 2010. -
  52. [58][58] Nikolaus Hansen. The cma evolution strategy: a comparing review. Towards a new evolutionary computation , pages 75-102, 2006. -
  53. [59][59] Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Natural evolution strategies. In Evolutionary Computation, 2008. CEC 2008.(IEEE World Congress on Computational Intelligence). IEEE Congress on , pages 3381-3387. IEEE, 2008. -
  54. [60][60] Kenneth O. Stanley and Risto Miikkulainen. A taxonomy for artificial embryogeny. Artificial Life , 9(2):93-130, 2003. URL http://nn.cs.utexas. edu/keyword?stanley:alife03. -
  55. [61][61] Kenneth O. Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic Programming and Evolvable Machines Special Issue on Developmental Systems , 8(2):131-162, 2007. -
  56. [62][62] Josh C. Bongard and Rolf Pfeifer. Repeated structure and dissociation of genotypic and phenotypic complexity in artificial ontogeny. In Lee Spector, Erik D. Goodman, Annie Wu, W. B. Langdon, Hans-Michael Voigt, Mitsuo Gen, Sandip Sen, Marco Dorigo, Shahram Pezeshk, Max H. Garzon, and Edmund Burke, editors, Genetic and Evolutionary Computation Conference , pages 829-836, 2001. ISBN 1-55860-774-9. URL http://www-illigal.ge.uiuc.edu:8080/gecco-2001/. -
  57. [63][63] Petet J. Bentley and S. Kumar. Three ways to grow designs: A comparison of embryogenies for an evolutionary design problem. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-1999) , pages 35-43, 1999. -
  58. [65][65] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. CTRL: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858 , 2019. - · arXiv
  59. [66][66] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 , 2015. - · arXiv
  60. [67][67] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. - · arXiv
  61. [68][68] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015. - · arXiv
  62. [69][69] Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv preprint arXiv:1706.03741 , 2017. - · arXiv
  63. [70][70] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325 , 2020. - · arXiv
  64. [71][71] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01345 , 2021. - · arXiv

Cited by

Sorted by citations

Authors
Lehman, J., Gordon, J., Jain, S., Ndousse, K., Yeh, C., & Stanley, K. O.
Affiliations
  • OpenAI (United States) (US)
  • Anthropic (United States) (US)
Corresponding author
Lehman, Joel
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
331-366
ISSN
1932-0167 (ISSN-L)
DOI
10.48550/arxiv.2206.08896
arXiv
2206.08896
OpenAlex
W4388139328
Cited by
74
References
0
Citations by year
2026: 17 · 2025: 23 · 2024: 22 · 2023: 10 · 2022: 2
FWCI
39.94
Type
book chapter
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Genetic and evolutionary computation (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
large language models, code generation, mutation operator, genetic programming, MAP-Elites, bootstrapping, open-endedness, zero-shot learning, Python

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review