CoChat
Sign Up
PreprintCited by 2Ethics and Social Impacts of AISafety Systems Engineering in AutonomyAdversarial Robustness in Machine Learning

The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.

Safety Must Precede the Deployment of Open-Ended AI

Open access
Sheth, Ivaxi, Wehner, Jan, Abdelnabi, Sahar, Binkyte, Ruta, Fritz, Mario·2025·arXiv
DOI 10.48550/arxiv.2502.04512arXiv 2502.04512
Open this paper in CoChat — free accountOpen PDF

Abstract

AI advancements have been significantly driven by a combination of foundation models and curiosity-driven learning aimed at increasing capability and adaptability. Within this landscape, open-endedness, where AI agents autonomously and indefinitely generate novel behaviors, representations, or solutions, has gained increasing interest. This has become relevant in the context of self-evolving agents and long-horizon discovery. This position paper argues that the defining properties of open-ended AI systems introduce a distinct and underexplored class of safety challenges, including loss of predictability, emergent misalignment, and difficulties in maintaining effective control as systems evolve beyond their initial design assumptions, that must be addressed preemptively. These challenges differ qualitatively from those associated with task-bounded or static models and are unlikely to be addressed by existing safety frameworks alone, which is why these risks must be examined proactively, before large-scale deployment. The paper proposes a taxonomy for key challenges, discusses research opportunities, and calls for coordinated action to support the safe and responsible development of open-ended AI.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.04512

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Iterative self-modification could lead to increasingly complex and uninterpretable internal logic, hindering human understanding, oversight, and control.

    “Iterative self-modification could also lead to increasingly complex and uninterpretable internal logic, hindering human understanding, oversight, and control (Sheth et al., 2025; Anwar et al., 2024; Greenblatt et al., 2024; Ganguli et al., 2022).”

    From Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents (2025) · cites this paper as [120]

    The Human-in-the-Loop Oversight and Hierarchical Oversight sections say complex artifacts can be difficult for humans to judge, and call for interpretable decisions to enable oversight. The paper does not say that iterative self-modification makes the system’s internal logic increasingly complex or uninterpretable; its stated complexity concern is principally about generated artifacts. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Iterative self-modification could lead to increasingly complex and uninterpretable internal logic, hindering human understanding, oversight, and control.

    “Iterative self-modification could also lead to increasingly complex and uninterpretable internal logic, hindering human understanding, oversight, and control (Sheth et al., 2025; Anwar et al., 2024; Greenblatt et al., 2024; Ganguli et al., 2022).”

    From Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents (2025) · cites this paper as Sheth et al., 2025

    The Human-in-the-Loop Oversight and Hierarchical Oversight sections say that complex artifacts can exceed humans’ ability to evaluate them and call for interpretable decisions to enable oversight. The paper does not say that iterative self-modification makes an AI’s internal logic increasingly complex or uninterpretable; its complexity concern is about generated artifacts. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 77

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv, 2023.
  2. Aki, F., Ikeda, R., Saito, T., Regan, C., and Oka, M. Llm- poet: Evolving complex environments using large lan- guage models. In the Genetic and Evolutionary Compu- tation Conference Companion, 2024.
  3. Akiba, T., Shing, M., Tang, Y., Sun, Q., and Ha, D. Evolu- tionary optimization of model merging recipes. Nature Machine Intelligence, pp. 1-10, 2025.
  4. Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. arXiv, 2018.
  5. Betley, J., Bao, X., Soto, M., Sztyber-Betley, A., Chua, J., and Evans, O. Tell me about yourself: Llms are aware of their learned behaviors. arXiv, 2025.
  6. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv, 2021.
  7. Bradley, H., Dai, A., Teufel, H. B., Zhang, J., Oostermeijer, K., Bellagente, M., Clune, J., Stanley, K., Schott, G., and Lehman, J. Quality-diversity through AI feedback. In Second Agent Learning in Open-Endedness Workshop, 2023.
  8. Brant, J. C. and Stanley, K. O. Minimal criterion coevolu- tion: a new approach to open-ended search. In the Genetic and Evolutionary Computation Conference, 2017.
  9. Bruce, J., Dennis, M. D., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steiger- wald, R., Apps, C., et al. Genie: Generative interactive environments. In ICML, 2024.
  10. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv, 2023.
  11. Burton, J. W., Lopez-Lopez, E., Hechtlinger, S., Rah- wan, Z., Aeschbach, S., Bakker, M. A., Becker, J. A., Berditchevskaia, A., Berger, J., Brinkmann, L., et al. How large language models can reshape collective intelligence. Nature human behaviour, 8(9):1643-1655, 2024.
  12. Cartoni, E., Montella, D., Triesch, J., and Baldassarre, G. Real-x-robot open-ended autonomous learning ar- chitectures: Achieving truly end-to-end sensorimotor au- tonomous learning systems. arXiv, 2020.
  13. Cartoni, E., Montella, D., Triesch, J., and Baldassarre, G. Real-x-robot open-ended autonomous learning architec- ture: Building truly end-to-end sensorimotor autonomous learning systems. Transactions on Cognitive and Devel- opmental Systems, 15(4):2014-2030, 2023.
  14. Chavan, P. and Chavan, P. Automation of ad-ohc dashbord and monitoring of cloud resources using genrative ai to reduce costing and enhance performance. In the IEEE International Conference on Innovations and Challenges in Emerging Technologies (ICICET), 2024.
  15. Che, H., He, X., Liu, Q., Jin, C., and Chen, H. Gamegen-x: Interactive open-world game video generation. arXiv, 2024.
  16. Christiano, P., Shlegeris, B., and Amodei, D. Supervising strong learners by amplifying weak experts. arXiv, 2018.
  17. Clune, J. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv, 2019.
  18. Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. arXiv, 2023.
  19. Dawood, M., Shokry, A., and Bennewitz, M. A dynamic safety shield for safe and efficient reinforcement learning of navigation tasks. arXiv, 2024.
  20. Deep Pala, T., Toh, V. Y., Bhardwaj, R., and Poria, S. Ferret: Faster and effective automated red teaming with reward- based scoring technique. arXiv, 2024.
  21. Di Langosco, L. L., Koch, J., Sharkey, L. D., Pfau, J., and Krueger, D. Goal misgeneralization in deep reinforce- ment learning. In ICML, 2022.
  22. Ecoffet, A., Clune, J., and Lehman, J. Open questions in creating safe open-ended ai: tensions between control and creativity. In Artificial Life Conference Proceedings 32, pp. 27-35. MIT Press, 2020.
  23. Fang, F., Zhu, L., Feng, X., Hou, J., Zhao, Q., Li, C., Hu, X., Xu, R., and Yang, M. Clha: A simple yet effec- tive contrastive learning framework for human alignment. In the Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC- COLING), 2024.
  24. Fisac, J. F., Akametalu, A. K., Zeilinger, M. N., Kaynama, S., Gillula, J., and Tomlin, C. J. A general safety frame- work for learning-based control in uncertain robotic sys- tems. Transactions on Automatic Control, 64(7):2737- 2752, 2018.
  25. Flageat, M. and Cully, A. Uncertain quality-diversity: evalu- ation methodology and new methods for quality-diversity in uncertain domains. Transactions on Evolutionary Com- putation, 2023.
  26. Flageat, M., Janmohamed, H., Lim, B., and Cully, A. Exploring the performance-reproducibility trade-off in quality-diversity. arXiv, 2024.
  27. Garcıa, J. and Fernández, F. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
  28. Han, V. T. Y., Bhardwaj, R., and Poria, S. Ruby team- ing: Improving quality diversity search with memory for automated red teaming. arXiv, 2024.
  29. Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. Unsolved problems in ml safety. arXiv, 2021.
  30. Hendrycks, D., Mazeika, M., and Woodside, T. An overview of catastrophic ai risks. arXiv, 2023.
  31. Hu, S. and Clune, J. Thought cloning: Learning to think while acting by imitating human thinking. NeurIPS, 2024.
  32. Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language models cannot self-correct reasoning yet. In ICLR, 2024.
  33. Hughes, E., Dennis, M., Parker-Holder, J., Behbahani, F., Mavalankar, A., Shi, Y., Schaul, T., and Rocktaschel, T. Open-endedness is essential for artificial superhuman intelligence. ICML, 2024.
  34. Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. In NeurIPS, 2018.
  35. Irving, G., Christiano, P., and Amodei, D. Ai safety via debate. arXiv, 2018.
  36. Jiang, M., Rocktäschel, T., and Grefenstette, E. General in- telligence requires rethinking exploration. Royal Society Open Science, 10(6):230539, 2023.
  37. Kamoi, R., Zhang, Y., Zhang, N., Han, J., and Zhang, R. When can llms actually correct their own mistakes? a critical survey of self-correction of llms. Transactions of the Association for Computational Linguistics, 12:1417- 1440, 2024.
  38. Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., and Perez, E. Debating with more per- suasive llms leads to more truthful answers. In ICML, 2024.
  39. Kıcıman, E., Ness, R., Sharma, A., and Tan, C. Causal reasoning and large language models: Opening a new frontier for causality. TMLR, 2024.
  40. Krakovna, V., Orseau, L., Kumar, R., Martic, M., and Legg, S. Penalizing side effects using stepwise relative reacha- bility. arXiv, 2018.
  41. Lehman, J. and Stanley, K. O. Revising the evolutionary computation abstraction: minimal criteria novelty search. In the 12th annual conference on Genetic and evolution- ary computation, 2010.
  42. Lehman, J. and Stanley, K. O. Abandoning objectives: Evo- lution through the search for novelty alone. Evolutionary computation, 19(2):189-223, 2011.
  43. Lehman, J., Gordon, J., Jain, S., Ndousse, K., Yeh, C., and Stanley, K. O. Evolution through large models. In Handbook of Evolutionary Machine Learning, pp. 331- 366. Springer, 2023.
  44. Leveson, N. G. Engineering a Safer World: Systems Think- ing Applied to Safety. MIT Press, 2012.
  45. Li, D., Zhang, C., Dong, K., Deik, D. G. X., Tang, R., and Liu, Y. Aligning crowd feedback via distributional preference reward modeling. arXiv, 2024.
  46. Liu, S., Chen, C., Qu, X., Tang, K., and Ong, Y.-S. Large language models as evolutionary optimizers. In 2024 IEEE Congress on Evolutionary Computation (CEC), pp. 1-8. IEEE, 2024.
  47. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv, 2024.
  48. Morris, M. R., Sohl-Dickstein, J., Fiedel, N., Warkentin, T., Dafoe, A., Faust, A., Farabet, C., and Legg, S. Position: Levels of agi for operationalizing progress on the path to agi. In ICML, 2024.
  49. Moskovitz, T., Singh, A. K., Strouse, D., Sandholm, T., Salakhutdinov, R., Dragan, A., and McAleer, S. M. Con- fronting reward model overoptimization with constrained RLHF. In ICLR, 2024.
  50. Mouret, J.-B. and Clune, J. Illuminating search spaces by mapping elites. arXiv, 2015.
  51. Nisioti, E., Glanois, C., Najarro, E., Dai, A., Meyerson, E., Pedersen, J. W., Teodorescu, L., Hayes, C. F., Sudhakaran, S., and Risi, S. From text to life: On the reciprocal relationship between artificial life and large language models. In Artificial Life Conference Proceedings 36, volume 2024, pp. 39. MIT Press, 2024.
  52. Packard, N., Bedau, M. A., Channon, A., Ikegami, T., Ras- mussen, S., Stanley, K. O., and Taylor, T. An overview of open-ended evolution: Editorial introduction to the open-ended evolution ii special issue. Artificial life, 25 (2):93-103, 2019.
  53. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.- M., Rothchild, D., So, D., Texier, M., and Dean, J. Carbon emissions and large neural network training. arXiv, 2021.
  54. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In ICML, 2023.
  55. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In ICML, 2021.
  56. Richens, J. and Everitt, T. Robust agents learn causal world models. ICLR, 2024.
  57. Rivera, C. G., Lyons, O., Summitt, A., Fatima, A., Pak, J., Shao, W., Chalmers, R., Englander, A., Staley, E. W., Wang, I., et al. Tanksworld: a multi-agent environment for ai safety research. arXiv, 2020.
  58. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  59. Samvelyan, M., Raparthy, S. C., Lupu, A., Hambro, E., Markosyan, A. H., Bhatt, M., Mao, Y., Jiang, M., Parker- Holder, J., Foerster, J. N., Rocktäschel, T., and Raileanu, R. Rainbow teaming: Open-ended generation of diverse adversarial prompts. In NeurIPS, 2024.
  60. Secretan, J., Beato, N., D Ambrosio, D. B., Rodriguez, A., Campbell, A., and Stanley, K. O. Picbreeder: evolving pictures collaboratively online. In the SIGCHI conference on human factors in computing systems, 2008.
  61. Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna, V., Uesato, J., and Kenton, Z. Goal misgeneralization: Why correct specifications aren't enough for correct goals. arXiv, 2022.
  62. Sigaud, O., Baldassarre, G., Colas, C., Doncieux, S., Duro, R., Oudeyer, P.-Y., Perrin-Gilbert, N., and Santucci, V. G. A definition of open-ended learning problems for goal- conditioned agents. arXiv, 2023.
  63. Soros, L. and Stanley, K. Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria. In Artificial Life Conference Proceedings, pp. 793-800. MIT Press, 2014.
  64. Stanley, K. O. Why open-endedness matters. Artificial life, 25(3):232-235, 2019.
  65. Stanley, K. O. and Lehman, J. Why greatness cannot be planned: The myth of the objective. Springer, 2015.
  66. Stanley, K. O. and Soros, L. The role of subjectivity in the evaluation of open-endedness. In Presentation deliv- ered in OEE2: The Second Workshop on Open-Ended Evolution, at ALIFE 2016, 2016.
  67. Sui, Y., Gotovos, A., Burdick, J., and Krause, A. Safe exploration for optimization with gaussian processes. In ICML, 2015.
  68. Suyama, K. Probabilistic safety assessment and manage- ment of control laws. In the 35th Annual Conference of IEEE Industrial Electronics, 2005.
  69. Team, O. E. L., Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., et al. Open-ended learning leads to gener- ally capable agents. arXiv, 2021.
  70. Turchetta, M., Berkenkamp, F., and Krause, A. Safe explo- ration in finite markov decision processes with gaussian processes. NeurIPS, 2016.
  71. Wang, R., Lehman, J., Clune, J., and Stanley, K. O. Paired open-ended trailblazer (poet): Endlessly generating in- creasingly complex and diverse learning environments and their solutions. arXiv, 2019.
  72. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In ICLR, 2023.
  73. Wang, Y., Zhang, T., Guo, X., and Shen, Z. Gradient based feature attribution in explainable ai: A technical review, 2024b. URL https://arxiv.org/abs/ 2403.10415.
  74. Wen, J., Zhong, R., Khan, A., Perez, E., Steinhardt, J., Huang, M., Bowman, S. R., He, H., and Feng, S. Lan- guage models learn to mislead humans via rlhf. arXiv, 2024.
  75. Zammit, M., Liapis, A., and Yannakakis, G. N. Map-elites with transverse assessment for multimodal problems in creative domains. In International Conference on Com- putational Intelligence in Music, Sound, Art and Design (Part of EvoStar). Springer, 2024.
  76. Zhang, H., Lei, Y., Gui, L., Yang, M., He, Y., Wang, H., and Xu, R. Cppo: Continual learning for reinforcement learning with human feedback. In ICLR, 2024a.
  77. Zhang, J., Lehman, J., Stanley, K., and Clune, J. OMNI: Open-endedness via models of human notions of interest- ingness. In ICLR, 2024b.

Cited by

Sorted by citations

Related

Metadata

Authors
Sheth, I., Wehner, J., Abdelnabi, S., Binkyte, R., & Fritz, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.04512
arXiv
2502.04512
OpenAlex
W4407308858
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • European Commission (101070617)
  • Bundesministerium für Bildung und Forschung (16KIS2012)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
AI safety, self-evolving agents, emergent misalignment, autonomous AI agents, responsible AI development

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review