CoChat
Sign Up
Cited by 41Ethics and Social Impacts of AIPsychiatry, Mental Health, NeuroscienceArtificial Intelligence Applications

Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?

Open access
Bengio, Yoshua, Cohen, Michael, Fornasiere, Damiano, Ghosn, Joumana, Greiner, Pietro, MacDermott, Matt, Mindermann, S¨oren, Oberman, Adam +5 more·2025·SuperIntelligence - Robotics - Safety & Alignment
DOI 10.70777/si.v2i5.15569
Open this paper in CoChat — free accountOpen PDF

Abstract

The leading AI companies are increasingly focused on building generalist AI agents—systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. We discuss how these risks arise from current AI training methods. Indeed, various scenarios and experiments have demonstrated the possibility of AI agents engaging in deception or pursuing goals that were not specified by human operators and that conflict with human interests, such as self-preservation. Following the precautionary principle, we see a strong need for safer, yet still useful, alternatives to the current agency-driven trajectory. Accordingly, we propose as a core building block for further advances the development of a non-agentic AI system that is trustworthy and safe by design, which we call Scientist AI. This system is designed to explain the world from observations, as opposed to taking actions in it to imitate or please humans. It comprises a world model that generates theories to explain data and a question-answering inference machine. Both components operate with an explicit notion of uncertainty to mitigate the risks of overconfident predictions. In light of these considerations, a Scientist AI could be used to assist human researchers in accelerating scientific progress, including in AI safety. In particular, our system can be employed as a guardrail against AI agents that might be created despite the risks involved. Ultimately, focusing on non-agentic AI may enable the benefits of AI innovation while avoiding the risks associated with the current trajectory. We hope these arguments will motivate researchers, developers, and policymakers to favor this safer path.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.70777/si.v2i5.15569

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 168

  1. Akhound-Sadegh, Tara et al. (2024). "Iterated denoising energy matching for sampling from Boltzmann densities". In: Proc. International Conference on Machine Learning. url: https://openreview.net/ forum?id=gVjMwLDFoQ.
  2. Al Kuwaiti, Ahmed et al. (2023). "A review of the role of artificial intelligence in healthcare". In: Journal of personalized medicine 13.6, p. 951. url: https://pmc.ncbi.nlm.nih.gov/articles/PMC10301994/.
  3. Altmeyer, Patrick et al. (2024). "Position: stop making unscientific AGI performance claims". In: Proc. International Conference on Machine Learning. url: https://dl.acm.org/doi/10.5555/3692070.3692 121. · DOI
  4. Angelopoulos, Anastasios N. and Stephen Bates (2023). "Conformal prediction: A gentle introduction". In: Foundations and Trends in Machine Learning 16.4, pp. 494-591. url: https://dl.acm.org/doi/10.15 61/2200000101. · DOI
  5. Armstrong, Stuart and Xavier O'Rorke (2017). "Good and safe uses of AI Oracles". In: ArXiv preprint 1711.05541. url: https://arxiv.org/abs/1711.05541. · arXiv
  6. Armstrong, Stuart, Anders Sandberg, and Nick Bostrom (2012). "Thinking inside the box: Controlling and using an oracle AI". In: Minds and Machines 22, pp. 299-324. url: https : / / link . springer . com / article/10.1007/s11023-012-9282-2. · DOI · Paper page
  7. Atanackovic, Lazar and Emmanuel Bengio (2024). "Investigating Generalization Behaviours of Generative Flow Networks". In: Proc. Workshop on Structured Probabilistic Inference & Generative Modeling. url: https://openreview.net/forum?id=umFrtGMWaQ.
  8. Augustin, Thomas et al. (2014). Introduction to imprecise probabilities. Vol. 591. John Wiley & Sons. url: https://books.google.ca/books?id=9qXIEAAAQBAJ.
  9. Ayyamperumal, Suriya Ganesh and Limin Ge (2024). "Current state of LLM Risks and AI Guardrails". In: ArXiv preprint 2406.12934. url: https://arxiv.org/abs/2406.12934. · arXiv
  10. Badue, Claudine et al. (2021). "Self-driving cars: A survey". In: Expert systems with applications 165. url: https://www.sciencedirect.com/science/article/abs/pii/S095741742030628X.
  11. Bai, Yuntao, Andy Jones, et al. (2022). "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback". In: ArXiv preprint 2204.05862. url: https://arxiv.org/abs/2204.0 5862. · arXiv
  12. Bai, Yuntao, Saurav Kadavath, et al. (2022). "Constitutional AI: Harmlessness from AI Feedback". In: ArXiv preprint 2212.08073. url: https://arxiv.org/abs/2212.08073. · arXiv
  13. Belghazi, Mohamed Ishmael et al. (2018). "Mutual information neural estimation". In: Proc. International Conference on Machine Learning, pp. 531-540. url: https://proceedings.mlr.press/v80/belghazi1 8a.html.
  14. Bengio, Emmanuel et al. (2021). "Flow network based generative models for non-iterative diverse candidate generation". In: Proc. Neural Information Processing Systems. Vol. 34, pp. 27381-27394. url: https:// proceedings.neurips.cc/paper/2021/hash/e614f646836aaed9f89ce58e837e2310-Abstract.html.
  15. Bengio, Yoshua (2023). "AI and catastrophic risk". In: Journal of Democracy 34.4, pp. 111-121. url: https: //muse.jhu.edu/pub/1/article/907692/summary.
  16. Bengio, Yoshua, Michael K. Cohen, et al. (2024). "Can a Bayesian Oracle Prevent Harm from an Agent?" In: arXiv preprint 2408.05284. url: https://arxiv.org/abs/2408.05284. · arXiv
  17. Bengio, Yoshua, Aaron Courville, and Pascal Vincent (2013). "Representation learning: A review and new perspectives". In: IEEE Transactions on Pattern Analysis and Machine Intelligence 35.8, pp. 1798-1828. url: https://dl.acm.org/doi/10.1109/tpami.2013. · DOI · Paper page
  18. Bengio, Yoshua, Tristan Deleu, et al. (2020). "A meta-transfer objective for learning to disentangle causal mechanisms". In: Proc. International Conference on Learning Representations. url: https://openrevie w.net/forum?id=ryxWIgBFPS. · DOI · Paper page
  19. Bengio, Yoshua, Grégoire Mesnil, et al. (2013). "Better mixing via deep representations". In: Proc. Inter- national Conference on Machine Learning, pp. 552-560. url: https://proceedings.mlr.press/v28 /bengio13.html.
  20. Bereska, Leonard and Efstratios Gavves (2024). "Mechanistic Interpretability for AI Safety -A Review". In: arXiv preprint 2404.14082. url: https://arxiv.org/abs/2404.14082. · arXiv
  21. Bishop, Christopher M. (2013). "Model-based machine learning". In: Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 371.1984. url: https://royalsocietypubl ishing.org/doi/full/10.1098/rsta.2012.0222. · DOI
  22. Blanchard, Thomas, Tania Lombrozo, and Shaun Nichols (2018). "Bayesian Occam's razor is a razor of the people". In: Cognitive science 42.4, pp. 1345-1359. url: https://onlinelibrary.wiley.com/doi/full/ 10.1111/cogs.12573. · DOI
  23. Blundell, Charles et al. (2015). "Weight uncertainty in neural network". In: Proc. International Conference on Machine Learning, pp. 1613-1622. url: https://proceedings.mlr.press/v37/blundell15.
  24. Bojarski, Mariusz (2016). "End to end learning for self-driving cars". In: ArXiv preprint 1604.07316. url: https://arxiv.org/abs/1604.07316. · arXiv
  25. Bostrom, Nick (2012). "The superintelligent will: Motivation and instrumental rationality in advanced arti- ficial agents". In: Minds and Machines 22, pp. 71-85. url: https://link.springer.com/article/10.1 007/s11023-012-9281-3. · DOI · Paper page
  26. Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. url: https: //books.google.ca/books?id=7_H8AwAAQBAJ.
  27. Breum, Simon Martin et al. (2024). "The persuasive power of large language models". In: Proc. AAAI Conference on Web and Social Media. Vol. 18, pp. 152-163. url: https://ojs.aaai.org/index.php/ ICWSM/article/view/31304.
  28. Bronstein, J.L. (2015). Mutualism. Oxford University Press. url: https://books.google.ca/books?id= tlIdCgAAQBAJ.
  29. Brown, Tom et al. (2020). "Language models are few-shot learners". In: Proc. Neural Information Processing Systems 33, pp. 1877-1901. url: https://dl.acm.org/doi/abs/10.5555/3495724.3495883. · DOI
  30. Brown-Cohen, Jonah, Geoffrey Irving, and Georgios Piliouras (2023). "Scalable AI safety via doubly-efficient debate". In: ArXiv preprint 2311.14125. url: https://arxiv.org/abs/2311.14125. · arXiv
  31. Bubeck, Sébastien et al. (2023). "Sparks of artificial general intelligence: early experiments with GPT-4". In: ArXiv preprint 2303.12712. url: https://arxiv.org/abs/2303.12712. · arXiv
  32. Bullock, Justin B et al. (2024). The Oxford handbook of AI governance. Oxford University Press. url: https://academic.oup.com/edited-volume/41989.
  33. Carlsmith, Joseph (2022). "Is power-seeking AI an existential risk?" In: ArXiv preprint 2206.13353. url: https://arxiv.org/abs/2206.13353. · arXiv
  34. Ceballos, Gerardo et al. (June 2018). "Accelerated modern human-induced species losses: Entering the sixth mass extinction". In: Science advances 4.7. url: https://www.science.org/doi/10.1126/sciadv.140 0253. · DOI · Paper page
  35. Chen, Ming-Hui, Qi-Man Shao, and Joseph G Ibrahim (2012). Monte Carlo methods in Bayesian computa- tion. Springer Science & Business Media. url: https://link.springer.com/book/10.1007/978-1-461 2-1276-8. · DOI
  36. Christiano, Paul, Jan Leike, et al. (2017). "Deep reinforcement learning from human preferences". In: Proc. Neural Information Processing Systems, pp. 4302-4310. url: https://dl.acm.org/doi/10.5555/32949 96.3295184. · DOI
  37. Cohen, Michael, Marcus Hutter, and Michael Osborne (2022). "Advanced artificial agents intervene in the provision of reward". In: AI magazine 43.3, pp. 282-293. url: https://ojs.aaai.org/aimagazine/ index.php/aimagazine/article/view/15084.
  38. Cohen, Michael K et al. (2024). "Regulating advanced artificial agents". In: Science 384.6691, pp. 36-38. url: https://www.science.org/doi/10.1126/science.adl0625. · DOI
  39. Colombatto, Clara and Stephen M Fleming (2024). "Folk psychological attributions of consciousness to large language models". In: Neuroscience of Consciousness 2024.1. url: https://academic.oup.com/ nc/article/2024/1/niae013/7644104.
  40. Colombo, Pierre, Pablo Piantanida, and Chloé Clavel (Aug. 2021). "A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations". In: Proc. Association for Computational Linguistics. Association for Computational Linguistics, pp. 6539-6550. url: https://aclanthology.org/2021.acl- long.511/.
  41. Coupé, Christophe et al. (2019). "Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche". In: Science advances 5.9. url: https://www.science. org/doi/10.1126/sciadv.aaw2594. · DOI
  42. Critch, Andrew and David Krueger (2020). "AI research considerations for human existential safety (ARCHES)". In: arXiv preprint 2006.04948. url: https://arxiv.org/abs/2006.04948. · arXiv
  43. Cuzzolin, Fabio (2021). The geometry of uncertainty. Springer Nature. url: https://www.google.co.uk/ books/edition/The_Geometry_of_Uncertainty/jNQPEAAAQBAJ.
  44. Czyż Pawe land Grabowski, Frederic et al. (2023). "Beyond Normal: On the Evaluation of Mutual Information Estimators". In: Proc. Neural Information Processing Systems. Vol. 36, pp. 16957-16990. url: https: //proceedings.neurips.cc/paper_files/paper/2023/file/36b80eae70ff629d667f210e13497edf- Paper-Conference.pdf.
  45. Dafoe, Allan et al. (2020). "Open problems in cooperative AI". In: ArXiv preprint 2012.08630. url: https: //arxiv.org/abs/2012.08630. · arXiv
  46. Dalrymple, David "davidad" et al. (2024). "Towards guaranteed safe AI: A framework for ensuring robust and reliable AI systems". In: ArXiv preprint 2405.06624. url: https://arxiv.org/abs/2405.06624. · arXiv
  47. Deleu, Tristan, António Góis, et al. (2022). "Bayesian structure learning with generative flow networks". In: Proc. Uncertainty in Artificial Intelligence, pp. 518-528. url: https://proceedings.mlr.press/v180 /deleu22a.html.
  48. Deleu, Tristan, Mizu Nishikawa-Toomey, et al. (2023). "Joint bayesian inference of graphical structure and parameters with a single generative flow network". In: Proc. Neural Information Processing Systems. Vol. 36, pp. 31204-31231. url: https://proceedings.neurips.cc/paper_files/paper/2023/hash/63 9a9a172c044fbb64175b5fad42e9a5-Abstract-Conference.html.
  49. Denison, Carson et al. (2024). "Sycophancy to subterfuge: investigating reward-tampering in large language models". In: ArXiv preprint 2406.10162. url: https://arxiv.org/abs/2406.10162. · arXiv
  50. Devlin, Jacob et al. (June 2019). "BERT: pre-training of deep bidirectional transformers for language un- derstanding". In: Proc. Association for Computational Linguistics, pp. 4171-4186. doi: 10.18653/v1/N1 9-1423. url: https://aclanthology.org/N19-1423/. · DOI · Paper page
  51. Di Langosco, Lauro Langosco et al. (2022). "Goal misgeneralization in deep reinforcement learning". In: Proc. International Conference on Machine Learning, pp. 12004-12019. url: https : / / proceedings . mlr.press/v162/langosco22a.html.
  52. Dias, Raquel and Ali Torkamani (2019). "Artificial intelligence in clinical and genomic diagnostics". In: Genome medicine 11.1, p. 70. url: https://genomemedicine.biomedcentral.com/articles/10.1186 /s13073-019-0689-8. · DOI
  53. Fang, Richard, Rohan Bindu, Akul Gupta, and Daniel Kang (2024). "LLM agents can autonomously exploit one-day vulnerabilities". In: arXiv preprint 2404.08144. url: https://arxiv.org/abs/2404.08144. · arXiv
  54. Fang, Richard, Rohan Bindu, Akul Gupta, Qiusi Zhan, et al. (2024). "LLM Agents can Autonomously Hack Websites". In: ArXiv preprint 2402.06664. url: https://arxiv.org/abs/2402.06664. · arXiv
  55. Finzi, Marc, Max Welling, and Andrew Gordon Wilson (2021). "A practical method for constructing equiv- ariant multilayer perceptrons for arbitrary matrix groups". In: Proc. International Conference on Machine Learning, pp. 3318-3328. url: https://proceedings.mlr.press/v139/finzi21a.
  56. Galatzer-Levy, Isaac R. et al. (2024). "The cognitive capabilities of generative AI: A comparative analysis with human benchmarks". In: ArXiv preprint 2410.07391. url: https://arxiv.org/abs/2410.07391. · arXiv
  57. Garnett, Roman (2023). Bayesian Optimization. Cambridge University Press. url: https://bayesoptbook. com/.
  58. Ghahramani, Zoubin (2015). "Probabilistic machine learning and artificial intelligence". In: Nature 521.7553, pp. 452-459. url: https://www.nature.com/articles/nature14541.
  59. Goan, Ethan and Clinton Fookes (2020). "Bayesian neural networks: An introduction and survey". In: Case Studies in Applied Bayesian Data Science, pp. 45-87. url: https://link.springer.com/chapter/10.1 007/978-3-030-42553-1_3. · DOI
  60. Good, Irving John (1966). "Speculations concerning the first ultraintelligent machine". In: Advances in computers 6, pp. 31-88. url: https://www.sciencedirect.com/science/article/abs/pii/S0065245 808604180.
  61. Goodfellow, Ian J, Jonathon Shlens, and Christian Szegedy (2014). "Explaining and harnessing adversarial examples". In: arXiv preprint 1412.6572. url: https://arxiv.org/abs/1412.6572. · arXiv
  62. Goodhart, Charles (1984). Problems of monetary management: the UK experience. Springer. url: https: //link.springer.com/chapter/10.1007/978-1-349-17295-5_4. · DOI
  63. Goswami, Lipichanda, Manoj Kumar Deka, and Mohendra Roy (2023). "Artificial intelligence in material engineering: A review on applications of artificial intelligence in material engineering". In: Advanced Engi- neering Materials 25.13. url: https://onlinelibrary.wiley.com/doi/abs/10.1002/adem.202300104. · DOI
  64. Goyal, Anirudh and Yoshua Bengio (2022). "Inductive biases for deep learning of higher-level cognition". In: Proceedings of the Royal Society A 478.2266. url: https://royalsocietypublishing.org/doi/full/1 0.1098/rspa.2021.0068. · DOI · Paper page
  65. Grace, Katja et al. (2024). "Thousands of AI authors on the future of AI". In: ArXiv preprint 2401.02843. url: https://arxiv.org/abs/2401.02843. · arXiv
  66. Greenblatt, Ryan, Carson Denison, et al. (2024). "Alignment faking in large language models". In: ArXiv preprint 2412.14093. url: https://arxiv.org/abs/2412.14093. · arXiv
  67. Greenblatt, Ryan, Buck Shlegeris, et al. (July 2024). "AI Control: Improving Safety Despite Intentional Subversion". In: pp. 16295-16336. url: https://proceedings.mlr.press/v235/greenblatt24a.html.
  68. Hadfield-Menell, Dylan and Gillian K Hadfield (2019). "Incomplete contracting and AI alignment". In: Proc. AAAI/ACM Conference on AI, Ethics, and Society, pp. 417-422. url: https://dl.acm.org/doi/abs/1 0.1145/3306618.3314250. · DOI
  69. Hadfield-Menell, Dylan, Stuart J Russell, et al. (2016). "Cooperative inverse reinforcement learning". In: Proc. Neural Information Processing Systems. Vol. 29. url: https://proceedings.neurips.cc/paper_ files/paper/2016/hash/c3395dd46c34fa7fd8d729d8cf88b7a8-Abstract.html.
  70. Haghighat, Ehsan, Danial Amini, and Ruben Juanes (2022). "Physics-informed neural network simulation of multiphase poroelasticity using stress-split sequential training". In: Computer Methods in Applied Me- chanics and Engineering 397. url: https://www.sciencedirect.com/science/article/abs/pii/S004 5782522003152.
  71. Hameleers, Michael, Toni G.L.A. van der Meer, and Tom Dobber (2024). "Distorting the truth versus blatant lies: The effects of different degrees of deception in domestic and foreign political deepfakes". In: Computers in Human Behavior 152. url: https://www.sciencedirect.com/science/article/pii/S0747563223 004478.
  72. He, Haoran et al. (2024). "Rectifying reinforcement learning for reward matching". In: ArXiv preprint 2406.02213. url: https://arxiv.org/abs/2406.02213. · arXiv
  73. He, Kaiming et al. (2022). "Masked autoencoders are scalable vision learners". In: Proc. Conf. on Computer Vision and Pattern Recognition, pp. 16000-16009. url: https://openaccess.thecvf.com/content/ CVPR2022/html/He_Masked_Autoencoders_Are_Scalable_Vision_Learners_CVPR_2022_paper.
  74. Hejna, Joey et al. (2025). "Robot Data Curation with Mutual Information Estimators". In: arXiv preprint 2502.08623. url: https://arxiv.org/abs/2502.08623. · arXiv
  75. Hendrycks, Dan (2023). "Natural selection favors AIs over humans". In: ArXiv preprint 2303.16200. url: https://arxiv.org/abs/2303.16200. · arXiv
  76. Taylor & Francis. url: https://www.aisafetybook.com/textbook/rogue-ai.
  77. Hendrycks, Dan, Nicholas Carlini, et al. (2021). Unsolved problems in ML safety. url: https://arxiv.org/ abs/2109.13916. · arXiv
  78. Hendrycks, Dan, Mantas Mazeika, and Thomas Woodside (2023). "An overview of catastrophic AI risks". In: ArXiv preprint 2306.12001. url: https://arxiv.org/abs/2306.12001. · arXiv
  79. Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean (2015). "Distilling the knowledge in a neural network". In: ArXiv preprint 1503.02531. url: https://arxiv.org/abs/1503.02531. · arXiv
  80. Ho, Jonathan, Ajay Jain, and Pieter Abbeel (2020). "Denoising diffusion probabilistic models". In: Proc. Neural Information Processing Systems. Vol. 33, pp. 6840-6851. url: https://proceedings.neurips. cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html.
  81. Hoffmann, Jordan et al. (2022). "Training compute-optimal large language models". In: ArXiv preprint 2203.15556. url: https://arxiv.org/abs/2203.15556. · arXiv
  82. Hu, Edward J, Moksh Jain, et al. (2024). "Amortizing intractable inference in large language models". In: Proc. International Conference on Learning Representations. url: https://openreview.net/forum?id= Ouj6p4ca60.
  83. Hu, Edward J, Nikolay Malkin, et al. (2023). "GFlowNet-EM for learning compositional latent variable models". In: Proc. International Conference on Machine Learning, pp. 13528-13549. url: https : / / proceedings.mlr.press/v202/hu23c.html.
  84. Huang, Biwei et al. (2020). "Causal discovery from heterogeneous/nonstationary data". In: Journal of Ma- chine Learning Research 21.89. url: https://www.jmlr.org/papers/v21/19-232.html.
  85. Hubinger, Evan, Adam Jermyn, et al. (2023). "Conditioning predictive models: risks and strategies". In: ArXiv preprint 2302.00805. url: https://arxiv.org/abs/2302.00805. · arXiv
  86. Hubinger, Evan, Chris van Merwijk, et al. (2019). "Risks from learned optimization in advanced machine learning systems". In: arXiv preprint 1906.01820. url: https://arxiv.org/abs/1906.01820. · arXiv
  87. Hurst, Alexander (2025). "I met the 'godfathers of AI' in Paris -here's what they told me to really worry about". In: The Guardian. https://www.iaseai.org/conference/livestream, video Feb 7th 9:30 AM CET, at 57:51. url: https://www.theguardian.com/commentisfree/2025/feb/14/ai-godfathers- paris-industry-dangers-future.
  88. Hussein, Ahmed et al. (2017). "Imitation learning: A survey of learning methods". In: Computing Surveys 50.2, pp. 1-35. url: https://dl.acm.org/doi/10.1145/3054912. · DOI · Paper page
  89. Irving, Geoffrey, Paul Christiano, and Dario Amodei (2018). "AI safety via debate". In: ArXiv preprint 1805.00899. url: https://arxiv.org/abs/1805.00899. · arXiv
  90. Ivanova, Desi R., Marvin Schmitt, and Stefan T. Radev (2024). "Data-efficient variational mutual infor- mation estimation via Bayesian self-consistency". In: Proc. Workshop on Bayesian Decision-Making and Uncertainty. url: https://openreview.net/forum?id=QfiyElaO1f&noteId=aRvehpmMkK.
  91. Jain, Moksh, Emmanuel Bengio, et al. (2022). "Biological sequence design with GFlowNets". In: Proc. International Conference on Machine Learning, pp. 9786-9801. url: https://proceedings.mlr.press/ v162/jain22a.html.
  92. Jain, Moksh, Tristan Deleu, et al. (2023). "GFlowNets for AI-driven scientific discovery". In: Digital Discovery 2.3, pp. 557-577. url: https://pubs.rsc.org/en/content/articlehtml/2023/dd/d3dd00002h.
  93. Järviniemi, Olli and Evan Hubinger (2024). "Uncovering deceptive tendencies in language models: A simu- lated company AI assistant". In: ArXiv preprint 2405.01576. url: https://arxiv.org/abs/2405.01576. · arXiv
  94. Jimenez, Carlos E et al. (2024). "SWE-Bench: Can language models resolve real-world Github issues?" In: Proc. International Conference on Learning Representations. url: https://openreview.net/forum?id= VTF8yNQM66.
  95. Jin, Zehao et al. (2025). "Causal discovery in astrophysics: Unraveling supermassive black hole and galaxy coevolution". In: The Astrophysical Journal 979.2, p. 212. url: https://iopscience.iop.org/article/1 0.3847/1538-4357/ad9ded/meta. · DOI
  96. Jumper, John et al. (2021). "Highly accurate protein structure prediction with AlphaFold". In: Nature 596.7873, pp. 583-589. url: https://www.nature.com/articles/s41586-021-03819-2.
  97. Kakade, Sham Machandranath (2003). "On the sample complexity of reinforcement learning". PhD thesis. University of London, University College London. url: https://homes.cs.washington.edu/ ~sham/ papers/thesis/sham_thesis.pdf.
  98. Kaplan, Jared et al. (2020). "Scaling laws for neural language models". In: ArXiv preprint 2001.08361. url: https://arxiv.org/abs/2001.08361. · arXiv
  99. Kashinath, Karthik et al. (2021). "Physics-informed machine learning: case studies for weather and climate modelling". In: Philosophical Transactions of the Royal Society 379.2194. url: https://royalsocietyp ublishing.org/doi/full/10.1098/rsta.2020.0093. · DOI
  100. Kaufmann, Timo et al. (2023). "A survey of reinforcement learning from human feedback". In: ArXiv preprint 2312.14925. url: https://arxiv.org/abs/2312.14925. · arXiv
  101. Kim, Minsu, Sanghyeok Choi, Hyeonah Kim, et al. (2024). "Ant colony sampling with GFlowNets for com- binatorial optimization". In: ArXiv preprint 2403.07041. url: https://arxiv.org/abs/2403.07041. · arXiv
  102. Kim, Minsu, Sanghyeok Choi, Taeyoung Yun, et al. (2025). "Adaptive teachers for amortized samplers". In: Proc. International Conference on Learning Representations. url: https://openreview.net/forum?id= BdmVgLMvaf.
  103. Kochkov, Dmitrii et al. (2024). "Neural general circulation models for weather and climate". In: Nature 632.8027, pp. 1060-1066. url: https://www.nature.com/articles/s41586-024-07744-y.
  104. Krueger, David (2024). "Safe and trustworthy agents: an oxymoron?" In: Proc. Workshop Towards Safe & Trustworthy Agents. url: https://neurips.cc/virtual/2024/workshop/84748.
  105. Kruger, Justin and David Dunning (1999). "Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments". In: Journal of personality and social psychology 77.6, p. 1121. url: https://pubmed.ncbi.nlm.nih.gov/10626367/.
  106. Kuhlau, Frida et al. (2011). "A precautionary principle for dual use research in the life sciences". In: Bioethics 25.1. url: https://pubmed.ncbi.nlm.nih.gov/19594724/.
  107. Kunda, Ziva (1990). "The case for motivated reasoning." In: Psychological bulletin 108.3, p. 480. url: https://psycnet.apa.org/record/1991-06436-001.
  108. Laubscher, Ryno (2021). "Simulation of multi-species flow and heat transfer using physics-informed neural networks". In: Physics of Fluids 33.8. url: https://pubs.aip.org/aip/pof/article-abstract/33/8 /087101/1080391/Simulation-of-multi-species-flow-and-heat-transfer.
  109. LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton (2015). "Deep Learning". In: Nature 521.7553, pp. 436- 444. url: https://www.nature.com/articles/nature14539.
  110. Lee, Chang-Shing et al. (2016). "Human vs. computer Go: Review and prospect". In: IEEE Computational Intelligence Magazine 11.3, pp. 67-72. url: https://ieeexplore.ieee.org/abstract/document/7515 285.
  111. Lee, Seanie et al. (2025). "Learning diverse attacks on large language models for robust red-teaming and safety tuning". In: Proc. International Conference on Learning Representations. url: https://openreview.net/ forum?id=1mXufFuv95.
  112. Leung, YinYee (2015). "Regret minimization and related decision rules". PhD thesis. Cornell University. url: https://ecommons.cornell.edu/server/api/core/bitstreams/ef0ef95b-6156-487e-900e-6 c33714ed0c3/content.
  113. Liu, Gary et al. (2023). "Deep learning-guided discovery of an antibiotic targeting Acinetobacter baumannii". In: Nature Chemical Biology 19.11, pp. 1342-1350. url: https://www.nature.com/articles/s41589-0 23-01349-8.
  114. Luo, Fan-Ming et al. (2024). "A survey on model-based reinforcement learning". In: Science China Informa- tion Sciences 67.2. url: https://link.springer.com/article/10.1007/s11432-022-3696-5. · DOI
  115. MacPhee, Ross D.E. (1999). Extinctions in near time. Vol. 2. Springer Science & Business Media. url: https://link.springer.com/book/10.1007/978-1-4757-5202-1. · DOI
  116. Malkin, Nikolay et al. (2023). "GFlowNets and variational inference". In: Proc. International Conference on Learning Representations. url: https://openreview.net/forum?id=uKiE0VIluA-.
  117. Marzo, Giordano De, Claudio Castellano, and David Garcia (2024). "Large language model agents can coordinate beyond human scale". In: ArXiv preprint 2409.02822. url: https://arxiv.org/abs/2409.0 2822. · arXiv
  118. McKinney, Scott Mayer et al. (2020). "International evaluation of an AI system for breast cancer screening". In: Nature 577.7788, pp. 89-94. url: https://www.nature.com/articles/s41586-019-1799-6.
  119. Meinke, Alexander et al. (2024). "Frontier models are capable of in-context scheming". In: ArXiv preprint 2412.04984. url: https://arxiv.org/abs/2412.04984. · arXiv
  120. Microsoft Corporation (Oct. 2023). Fiscal Year 2024 First Quarter Earnings Conference Call. Webpage: https://www.microsoft.com/en-us/investor/events/fy-2024/earnings-fy-2024-q1. Accessed: 2025-02-06.
  121. Milli, Smitha et al. (2017). "Should robots be obedient?" In: arXiv preprint 1705.09990. url: https:// arxiv.org/abs/1705.09990. · arXiv
  122. Mooney, H. A. and E. E. Cleland (2001). "The evolutionary impact of invasive species". In: Proc. of the National Academy of Sciences 98.10, pp. 5446-5451. url: https://www.pnas.org/doi/abs/10.1073 /pnas.091093398. · DOI
  123. Neumann, John von, Oskar Morgenstern, and Ariel Rubinstein (1944). Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition). Princeton University Press. url: https://books. google.ca/books?id=jCN5aNJ-n-0C.
  124. Ngo, Richard, Lawrence Chan, and Sören Mindermann (2024). "The alignment problem from a deep learning perspective". In: Proc. International Conference on Learning Representations. url: https://openreview. net/forum?id=fh8EYKFKns.
  125. Omohundro, Stephen M (2018). "The basic AI drives". In: Artificial intelligence safety and security. Chapman and Hall/CRC, pp. 47-55. url: https://dl.acm.org/doi/10.5555/1566174.1566226. · DOI
  126. Ouyang, Long et al. (2022). "Training language models to follow instructions with human feedback". In: Proc. Neural Information Processing Systems. Vol. 35, pp. 27730-27744. url: https://proceedings.neurips. cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
  127. Park, Peter S et al. (2024). "AI deception: A survey of examples, risks, and potential solutions". In: Patterns 5.5. url: https://www.cell.com/patterns/fulltext/S2666-3899%2824%2900103-X?s=08.
  128. Perdomo, Juan et al. (2020). "Performative prediction". In: Proc. International Conference on Machine Learning, pp. 7599-7609. url: https://proceedings.mlr.press/v119/perdomo20a.html.
  129. Peters, Jonas, Dominik Janzing, and Bernhard Schölkopf (2017). Elements of causal inference: foundations and learning algorithms. The MIT Press. url: https://library.oapen.org/handle/20.500.12657/26 040.
  130. Peyrard, Maxime and Kyunghyun Cho (2025). Meta-Statistical Learning: Supervised Learning of Statistical Inference. url: https://arxiv.org/abs/2502.12088. · arXiv
  131. Poole, Ben et al. (June 2019). "On Variational Bounds of Mutual Information". In: Proc. International Conference on Machine Learning. Vol. 97, pp. 5171-5180. url: https://proceedings.mlr.press/v97 /poole19a.html.
  132. Popova, Mariya, Olexandr Isayev, and Alexander Tropsha (2018). "Deep reinforcement learning for de novo drug design". In: Science advances 4.7. url: https://www.science.org/doi/10.1126/sciadv.aap7885. · DOI
  133. Rai, Daking et al. (2024). "A practical review of mechanistic interpretability for transformer-based language models". In: ArXiv preprint 2407.02646. url: https://arxiv.org/abs/2407.02646. · arXiv
  134. Rainforth, Tom et al. (2024). "Modern Bayesian experimental design". In: Statistical Science 39.1, pp. 100- 114. url: https : / / projecteuclid . org / journals / statistical -science / volume -39 / issue -1 /Modern-Bayesian-Experimental-Design/10.1214/23-STS915.short. · DOI
  135. Ramsey, Frank P. (1926). "Truth and Probability". In: The Foundations of Mathematics and other Logical Essays. Ed. by R.B. Braithwaite. 1999 electronic edition. London: Kegan, Paul, Trench, Trubner & Co. Chap. VII, pp. 156-198. url: https://books.google.ca/books?id=1st-3kYOEPQC.
  136. Reed, Scott et al. (2022). "A generalist agent". In: Transactions on Machine Learning Research. url: https: //openreview.net/forum?id=1ikK0kHjvj.
  137. Rempe, Davis et al. (2021). "Generating useful accident-prone driving scenarios via a learned traffic prior". In: Proc. Conf. on Computer Vision and Pattern Recognition, pp. 17284-17294. url: https : / / www . computer.org/csdl/proceedings-article/cvpr/2022/694600r7284/1H1k7GlOq9G.
  138. Richardson, Oliver E. (2022). "Loss as the inconsistency of a probabilistic dependency graph: Choose your model, not your loss function". In: Proc. International Conference on Artificial Intelligence and Statistics, pp. 2706-2735. url: https://proceedings.mlr.press/v151/richardson22b.html.
  139. Rowe, Luke et al. (2024). "CtRL-Sim: reactive and controllable driving agents with offline reinforcement learning". In: Conference on Robot Learning. url: https://openreview.net/forum?id=MfIUKzihC8.
  140. Ruan, Yangjun et al. (2023). "Identifying the risks of LM agents with an LM-emulated sandbox". In: Proc. International Conference on Learning Representations. url: https : / / openreview . net / forum ? id = GEcwtMk1uA.
  141. Russell, Stuart (2019). Human compatible: AI and the problem of control. Penguin UK. url: https : / / books.google.ca/books/about/Human_Compatible.html?id=VMq_wwEACAAJ.
  142. Russell, Stuart (2022). "If we succeed". In: Daedalus 151.2, pp. 43-57. url: https://www.amacad.org/ publication/daedalus/if-we-succeed.
  143. Savage, L. J. (1954). Foundations of Statistics. New York: Wiley. url: https://books.google.ca/books? id=zSv6dBWneMEC.
  144. Schölkopf, Bernhard et al. (2021). "Toward causal representation learning". In: Proc. IEEE. Vol. 109. IEEE, pp. 612-634. url: https://ieeexplore.ieee.org/abstract/document/9363924.
  145. Schrittwieser, Julian et al. (2020). "Mastering Atari, Go, chess and shogi by planning with a learned model". In: Nature 588.7839, pp. 604-609. url: https://www.nature.com/articles/s41586-020-03051-4.
  146. Sendera, Marcin et al. (2024). "Improved off-policy training of diffusion samplers". In: Proc. Neural Infor- mation Processing Systems. url: https://openreview.net/forum?id=vieIamY2Gi.
  147. Shah, Rohin et al. (2022). "Goal misgeneralization: why correct specifications aren't enough for correct goals". In: ArXiv preprint 2210.01790. url: https://arxiv.org/abs/2210.01790. · arXiv
  148. Silver, David, Aja Huang, et al. (2016). "Mastering the game of Go with deep neural networks and tree search". In: Nature 529.7587, pp. 484-489. url: https://www.nature.com/articles/nature16961.
  149. Silver, David, Thomas Hubert, et al. (2018). "A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play". In: Science 362.6419, pp. 1140-1144. url: https://www.science.org/ doi/10.1126/science.aar6404. · DOI · Paper page
  150. Silver, David, Julian Schrittwieser, et al. (2017). "Mastering the game of Go without human knowledge". In: nature 550.7676, pp. 354-359. url: https://www.nature.com/articles/nature24270.
  151. Skalse, Joar et al. (2022). "Defining and characterizing reward gaming". In: Proc. Neural Information Pro- cessing Systems. Vol. 35, pp. 9460-9471. url: https://proceedings.neurips.cc/paper_files/paper/2 022/hash/3d719fee332caa23d5038b8a90e81796-Abstract-Conference.html.
  152. Sohl-Dickstein, Jascha et al. (2015). "Deep unsupervised learning using nonequilibrium thermodynamics". In: Proc. International Conference on Machine Learning, pp. 2256-2265. url: http://proceedings.mlr. press/v37/sohl-dickstein15.html.
  153. Solomonoff, Ray J (1964). "A formal theory of inductive inference. Part I". In: Information and control 7.1, pp. 1-22. url: https://www.sciencedirect.com/science/article/pii/S0019995864902232.
  154. Thornley, Elliott (2024). "The shutdown problem: an AI engineering puzzle for decision theorists". In: Philo- sophical Studies, pp. 1-28. url: https://link.springer.com/article/10.1007/s11098-024-02153-3. · DOI
  155. Turtayev, Rustem et al. (2024). "Hacking CTFs with plain agents". In: ArXiv preprint 2412.02776. url: https://arxiv.org/abs/2412.02776. · arXiv
  156. Vaccari, Cristian and Andrew Chadwick (2020). "Deepfakes and disinformation: Exploring the impact of synthetic political video on deception, uncertainty, and trust in news". In: Social media + society 6.1. url: https://journals.sagepub.com/doi/10.1177/2056305120903408. · DOI · Paper page
  157. Vargas, Francisco, Will Grathwohl, and Arnaud Doucet (2023). "Denoising diffusion samplers". In: Proc. International Conference on Learning Representations. url: https : / / openreview . net / forum ? id =
  158. pvnfTAbu1f. Venkatraman, Siddarth et al. (2024). "Amortizing Intractable Inference in Diffusion Models for Bayesian Inverse Problems". In: Proc. Workshop on Machine Learning and the Physical Sciences. Accessed: 2025- 02-06. url: https://ml4physicalsciences.github.io/2024/files/NeurIPS_ML4PS_2024_188.pdf.
  159. Villalobos, Pablo et al. (2024). "Position: will we run out of data? Limits of LLM scaling based on human- generated data". In: Proc. International Conference on Machine Learning. url: https://openreview. net/forum?id=ViZcgDQjyG.
  160. Vincent, Pascal (2011). "A connection between score matching and denoising autoencoders". In: Neural computation 23, pp. 1661-1674. url: https://ieeexplore.ieee.org/abstract/document/6795935.
  161. Wang, Hanchen et al. (2023). "Scientific discovery in the age of artificial intelligence". In: Nature 620.7972, pp. 47-60. url: https://www.nature.com/articles/s41586-023-06221-2.
  162. Wang, Jingkang et al. (2021). "AdvSim: Generating safety-critical scenarios for self-driving vehicles". In: Proc. Conf. on Computer Vision and Pattern Recognition, pp. 9909-9918. url: https://ieeexplore. ieee.org/document/9578745.
  163. Wei, Jason et al. (2022). "Emergent abilities of large language models". In: Transactions on Machine Learning Research. url: https://openreview.net/forum?id=yzkSU5zdwD.
  164. Wijk, Hjalmar et al. (2024). "RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts". In: arXiv preprint 2411.15114. url: https://arxiv.org/abs/2411.15114. · arXiv
  165. Zhang, Dinghuai, Ricky TQ Chen, et al. (2022). "Unifying generative models with GFlowNets and beyond". In: arXiv preprint 2209.02606. url: https://arxiv.org/abs/2209.02606. · arXiv
  166. Zhang, Dinghuai, Nikolay Malkin, et al. (2022). "Generative flow networks for discrete probabilistic model- ing". In: Proc. International Conference on Machine Learning, pp. 26412-26428. url: https://proceed ings.mlr.press/v162/zhang22v.html.
  167. Zhang, Qinsheng and Yongxin Chen (2022). "Path integral sampler: a stochastic control approach for sam- pling". In: Proc. International Conference on Learning Representations. url: https://openreview.net/ forum?id=_uCb2ynRu7Y.
  168. Zou, Andy et al. (2023). "Universal and transferable adversarial attacks on aligned language models". In: ArXiv preprint 2307.15043. url: https://arxiv.org/abs/2307.15043. · arXiv

Cited by

Sorted by citations

Authors
Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., Mindermann, S., Oberman, A., Richardson, J., Richardson, O., Rondeau, M.-A., St-Charles, P.-L., & Williams-King, D.
Affiliations
  • Mila - Quebec Artificial Intelligence Institute (CA)
  • University of California, Berkeley (US)
ORCID
Venue
SuperIntelligence - Robotics - Safety & Alignment
Publisher
Carlson Research LLC
Published
2025-08-15
Volume
2
Issue
5
ISSN
3067-2627 (ISSN-L)
DOI
10.70777/si.v2i5.15569
OpenAlex
W4413417613
Cited by
41
References
0
Citations by year
2026: 26 · 2025: 15
FWCI
29.73
Type
journal article
Language
en
Open access
Licence
Locations
  • SuperIntelligence - Robotics - Safety & Alignment (publishedVersion, cc-by-nd)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • CERN Document Server (European Organization for Nuclear Research) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
AI safety, world models, AI deception, goal misalignment, precautionary principle, uncertainty estimation, AI guardrails, AI alignment, scientific question answering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review