CoChat
Sign Up
Cited by 6Ethics and Social Impacts of AIEvolutionary Algorithms and ApplicationsPsychiatry, Mental Health, Neuroscience

Open Questions in Creating Safe Open-ended AI: Tensions Between Control and Creativity

Open access
Ecoffet, Adrien, Clune, Jeff, Lehman, Joel·2020·The 2020 Conference on Artificial Life
DOI 10.1162/isal_a_00323
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial life originated and has long studied the topic of open-ended evolution, which seeks the principles underlying artificial systems that innovate continually, inspired by biological evolution. Recently, interest has grown within the broader field of AI in a generalization of open-ended evolution, here called open-ended search, wherein such questions of open-endedness are explored for advancing AI, whatever the nature of the underlying search algorithm (e.g. evolutionary or gradient-based). For example, open-ended search might design new architectures for neural networks, new reinforcement learning algorithms, or most ambitiously, aim at designing artificial general intelligence. This paper proposes that open-ended evolution and artificial life have much to contribute towards the understanding of open-ended AI, focusing here in particular on the safety of open-ended search. The idea is that AI systems are increasingly applied in the real world, often producing unintended harms in the process, which motivates the growing field of AI safety. This paper argues that open-ended AI has its own safety challenges, in particular, whether the creativity of open-ended systems can be productively and predictably controlled. This paper explains how unique safety problems manifest in open-ended search, and suggests concrete contributions and research questions to explore them. The hope is to inspire progress towards creative, useful, and safe open-ended search algorithms.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/isal_a_00323

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Any advancement that increases the autonomous capabilities of AI systems introduces its own set of safety considerations, especially for systems that improve in an open-ended way.

    “Any advancement that increases the autonomous capabilities of AI systems introduces its own set of safety considerations (Bengio et al., 2024), especially for systems that improve in an open-ended way (Ecoffet et al., 2020; Clune, 2019).”

    From Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents (2025) · cites this paper as Ecoffet et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. Akkaya, I.; Andrychowicz, M.; Chociej, M.; Litwin, M.; McGrew, B.; Petron, A.; Paino, A.; Plappert, M.; et al. 2019. Solving rubik's cube with a robot hand. arXiv preprint .
  2. Allen, C.; Smit, I.; and Wallach, W. 2005. Artificial morality: Top-down, bottom-up, and hybrid approaches. Ethics and information technology 7(3):149-155.
  3. Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P.; Schulman, J.; and Mané, D. 2016. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 . · arXiv
  4. Armstrong, S. 2013. General purpose intelligence: arguing the orthogonality thesis. Analysis and Metaphysics (12):68-84.
  5. Brant, J. C., and Stanley, K. O. 2017. Minimal criterion coevolution: a new approach to open-ended search. In Proceedings of the Genetic and Evolutionary Computation Conference , 6774. ACM.
  6. Cavusoglu, H.; Cavusoglu, H.; and Raghunathan, S. 2007. Efficiency of vulnerability disclosure mechanisms to disseminate vulnerability knowledge. IEEE Transactions on Software Engineering 33(3):171-185.
  7. Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , 4299-4307.
  8. Danielson, P. 2002. Artificial morality: Virtuous robots for virtual games . Routledge.
  9. Everitt, T.; Lea, G.; and Hutter, M. 2018. Agi safety literature review. arXiv preprint arXiv:1805.01109 . · arXiv
  10. Finn, C.; Abbeel, P.; and Levine, S. 2017. Model-agnostic metalearning for fast adaptation of deep networks. In Proc. of the Intl.Conf. on Machine Learning , 1126-1135. JMLR.
  11. Guttenberg, N.; Virgo, N.; and Penn, A. 2019. On the potential for open-endedness in neural networks. Artificial life 25(2):145167.
  12. Hadfield-Menell, D.; Milli, S.; Abbeel, P.; Russell, S. J.; and Dragan, A. 2017. Inverse reward design. In Advances in neural information processing systems , 6765-6774.
  13. Houthooft, R.; Chen, Y.; Isola, P.; Stadie, B.; Wolski, F.; Ho, O. J.; and Abbeel, P. 2018. Evolved policy gradients. In Advances in Neural Information Processing Systems , 5400-5409.
  14. Hubinger, E.; van Merwijk, C.; Mikulik, V.; Skalse, J.; and Garrabrant, S. 2019. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820 . · arXiv
  15. Kahn, L. H. 2004. Biodefense research: can secrecy and safety coexist? Biosecurity and bioterrorism: biodefense strategy, practice, and science 2(2):81-85.
  16. Lehman, J., and Stanley, K. O. 2011. Novelty search and the problem with objectives. In Genetic programming theory and practice IX . Springer. 37-56.
  17. Lehman, J.; Clune, J.; Misevic, D.; Adami, C.; Altenberg, L.; Beaulieu, J.; Bentley, P. J.; Bernard, S.; Beslon, G.; Bryson, D. M.; et al. 2018. The surprising creativity of digital evolution. arXiv preprint arXiv:1803.03453 . · arXiv
  18. Leike, J.; Krueger, D.; Everitt, T.; Martic, M.; Maini, V .; and Legg, S. 2018. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871 . · arXiv
  19. Mitchell, M. 2009. Complexity: A guided tour . Oxford University Press.
  20. Nguyen, A. M.; Yosinski, J.; and Clune, J. 2015. Innovation engines: Automated creativity and improved stochastic optimization via deep learning. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation , 959-966.
  21. Nguyen, A.; Yosinski, J.; and Clune, J. 2019. Understanding neural networks via feature visualization: A survey. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning . Springer. 55-76.
  22. Ofria, C., and Wilke, C. O. 2004. Avida: A software platform for research in computational evolutionary biology. Artificial life 10(2):191-229.
  23. Olah, C.; Satyanarayan, A.; Johnson, I.; Carter, S.; Schubert, L.; Ye, K.; and Mordvintsev, A. 2018. The building blocks of interpretability. Distill 3(3):e10.
  24. Packard, N.; Bedau, M. A.; Channon, A.; Ikegami, T.; Rasmussen, S.; Stanley, K. O.; and Taylor, T. 2019. An overview of openended evolution: Editorial introduction to the open-ended evolution ii special issue. Artificial life 25(2):93-103.
  25. Packard, N. H. 1997. A comparison of evolutionary activity in artificial evolving systems and in the biosphere. In Fourth European conference on artificial life , volume 4, 125. MIT Press.
  26. Saunders, W.; Sastry, G.; Stuhlmueller, A.; and Evans, O. 2018. Trial without error: Towards safe reinforcement learning via human intervention. In Proc. of AAMAS 2018 , 2067-2069.
  27. Sayre-McCord, G. 2017. Moral realism. In Zalta, E. N., ed., The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, fall 2017 edition.
  28. Shine, R. 2010. The ecological impact of invasive cane toads (bufo marinus) in australia. The Quarterly review of biology 85(3):253-291.
  29. Soltoggio, A.; Bullinaria, J. A.; Mattiussi, C.; Dürr, P.; and Floreano, D. 2008. Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios. In Proceedings of the 11th international conference on artificial life (Alife XI) , number CONF, 569-576. MIT Press.
  30. Standish, R. K. 2003. Open-ended artificial evolution. International Journal of Computational Intelligence and Applications 3(02):167-175.
  31. Stanley, K. O., and Lehman, J. 2015. Why greatness cannot be planned: The myth of the objective . Springer.
  32. Stanley, K. O.; Lehman, J.; and Soros, L. 2017. Open-endedness: The last grand challenge you've never heard of. O'Reilly Online .
  33. Taylor, T., and Hallam, J. 1998. An investigation into the role of contingency in evolution. In Artificial life VI: Proceedings of the Sixth International Conference on Artificial Life , volume 6, 256. MIT Press.
  34. Taylor, T.; Bedau, M.; Channon, A.; Ackley, D.; Banzhaf, W.; Beslon, G.; Dolson, E.; Froese, T.; Hickinbotham, S.; Ikegami, T.; et al. 2016b. Open-ended evolution: Perspectives from the oee workshop in york. Artificial life 22(3):408-423.
  35. Trut, L. N. 1999. Early canid domestication: The farm-fox experiment: Foxes bred for tamability in a 40-year experiment exhibit remarkable transformations that suggest an interplay between behavioral genetics and development. American Scientist 87(2):160-169.
  36. Vilalta, R., and Drissi, Y . 2002. A perspective view and survey of meta-learning. Artificial intelligence review 18(2):77-95.
  37. Wagenaar, D. A., and Adami, C. 2004. Influence of chance, history, and adaptation on digital evolution. Artificial life 10(2):181190.
  38. Winston, M. L. 1992. The biology and management of africanized honey bees. Annual review of entomology 37(1):173-193.
  39. Woolley, B. G., and Stanley, K. O. 2014. A novel human-computer collaboration: combining novelty search with interactive evolution. In Proceedings of the 2014 annual conference on genetic and evolutionary computation , 233-240.

Cited by

Sorted by citations

Authors
Ecoffet, A., Clune, J., & Lehman, J.
ORCID
Venue
The 2020 Conference on Artificial Life
Publisher
MIT Press
Published
2020
Pages
27-35
DOI
10.1162/isal_a_00323
OpenAlex
W3035352369
Cited by
6
References
3
Citations by year
2026: 1 · 2025: 1 · 2024: 2 · 2023: 1 · 2021: 1
FWCI
0.52
Type
conference paper
Language
en
Open access
Locations
  • Conference on Artificial Life (ALIFE) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
Keywords
open-ended evolution, AI safety, artificial life, artificial general intelligence, unintended harm, neural architecture search

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review