Managing extreme AI risks amid rapid progress
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Preparation requires technical research and development, as well as adaptive, proactive governance
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Any advancement that increases the autonomous capabilities of AI systems introduces its own set of safety considerations (Bengio et al., 2024), especially for systems that improve in an open-ended way (Ecoffet et al., 2020; Clune, 2019).”
From Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents (2025) · cites this paper as Bengio et al., 2024
In the citing paper
“Advanced AI also introduces phenomena that differ fundamentally from previous generations of AI or other technologies, requiring new approaches to mitigating these risks (Bengio et al., 2024).”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Bengio et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationA Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925 · cited 2×
“However, AI is making progress in critical domains such as hacking, social manipulation, and strategic planning 24,31 , and may soon pose unprecedented con-trol challenges.”
BackgroundEmergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · cited 1×
“Especially in the hands of a few powerful actors, AI could cement or exacerbate global inequities, or facilitate automated warfare, customized mass manipulation, and pervasive surveillance 18–23 .”
BackgroundA graph placement methodology for fast chip design
Mirhoseini, A., Goldie, A., Yazgan, M., Jiang, J. W., Songhori, E., Wang, S., Lee, Y.-J., Johnson, E., Pathak, O., Nova, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03544-w · Cited by 618 · cited 1×
“Progress in AI also enables faster AI progress 8 : AI assistants are increasingly used to automate programming 9 , data collection 10,11 , and chip design 12 .”
MethodSuperhuman AI for multiplayer poker
Brown, N., & Sandholm, T. (2019) · Science · doi:10.1126/science.aay2400 · Cited by 592
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1657 · Cited by 475
Research works printed but not linked: 8
- M. Kinniment et al. arXiv:2312.11671 (18 December 2023). · arXiv
- I. R.McKenzie. arXiv:2306.09479 (15 June 2023). · arXiv
- R.Ngo L.Chan S.Mindermann arXiv:2209.00626 (20 August 2022). · arXiv
- E.Hubinger. arXiv:2401.05566 (10 January 2024). · arXiv
- L.Koessler J.Schuett arXiv:2307.08823 (17 July 2023). · arXiv
- N. Kolt et al. arXiv:2404.02675 (3 April 2024). · arXiv
- M. Phuong et al. arXiv:2403.13793 (20 March 2024). · arXiv
- T. A. Kelly, SAE Trans. J. Mater. Manuf. 113, 257 (2004).
Cited by
Items 101–200 of 618
The Loss of Control Playbook: Degrees, Dynamics, and PreparednessPreprint
Stix, C., Hallensleben, A., Ortega, A., & Pistillo, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.15846 · Cited by 19
International AI Safety Report
Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox, P., Garfinkel, B., Goldfarb, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.17805 · Cited by 16
Self-citationKollig, F., Pater, J. A., Nova, F. F., & Fiesler, C. L. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713322 · Cited by 18
“Careful decisions must be made about where to direct energy, and practically, more proactive and fexible eforts to address questionable CAI health technologies might come from multiple agencies [17].”
InfluentialArnold, Z., Schiff, D. S., Schiff, K. J., Love, B., Melot, J., Singh, N., Jenkins, L., Lin, A., Pilz, K., Enweareazu, O., et al. (2024) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v7i1.31615 · Cited by 16
Materiality and risk in the age of pervasive AI sensors
Sloane, M., Moss, E., Kennedy, S., Stewart, M. P., Warden, P., Plancher, B., & Reddi, V. J. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01017-7 · Cited by 19
“In doing so, we identify how successive stages of sensor development have added to the risks produced by sensors as components of AI systems, which we discuss in the following section (Section 3).”
Lessons from the trenches on evaluating machine learning systems in materials science
Alampara, N., Schilling-Wilhelmi, M., & Jablonka, K. M. (2025) · Computational Materials Science · doi:10.1016/j.commatsci.2025.114041 · Cited by 22
Capturing a moving target: Developing research on and with AI for Human Relations
Stollberger, J., Anand, S., & Dick, P. (2025) · Human Relations · doi:10.1177/00187267251332075 · Cited by 19
Chimera: Communication Fusion for Hybrid Parallelism in Large Language Models
Qin, L., Cui, J., Cai, W., & Huang, J. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731025 · Cited by 20
Path to Artificial General Intelligence: Past, present, and future
Sarikaya, R. (2025) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2025.101021 · Cited by 21
AI in the L2 Classroom: Serving Language Educators through Professional Development
Minnillo, S., Jones, L., & García, S. (2024) · L2 Journal · doi:10.5070/l2.21198 · Cited by 16
AI Literacy: Foundations, Importance, and Broader Implications
Asrifan, A., Said, U. M. R., Jakob, J. C., & Wanci, R. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-8252-3.ch002 · Cited by 15
Security practices in AI development
Špelda, P., & Střítecký, V. (2025) · AI & Society · doi:10.1007/s00146-025-02247-4 · Cited by 19
Existential risk narratives about AI do not distract from its immediate harms
Hoes, E., & Gilardi, F. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2419055122 · Cited by 18
ijlessjournal (2026) · مجلة القانون والأعمال الدولية · doi:10.63585/ozsp2793 · Cited by 15
Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents
Piatti, G., Jin, Z., Kleiman-Weiner, M., Schölkopf, B., Sachan, M., & Mihalcea, R. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3548 · Cited by 14
What it takes to control AI by design: human learning
Te’eni, D., Yahav, I., & Schwartz, D. G. (2025) · AI & Society · doi:10.1007/s00146-025-02401-y · Cited by 19
“On the one hand, the transparency of AI models has been recognized as a design goal to enable human oversight, promote trust, and facilitate accountability (Bengio et al. 2024).”
Strategic insights from simulation gaming of AI race dynamics
Gruetzemacher, R., Avin, S., Fox, J., & Saeri, A. K. (2025) · Futures · doi:10.1016/j.futures.2025.103563 · Cited by 19
Can attention detect AI-generated text? A novel Benford's law-based approach
Wang, Z., Xu, G., & Ren, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104139 · Cited by 19
Position Paper: Technical Research and Talent is Needed for Effective AI GovernancePreprint
Reuel, A., Soder, L., Bucknall, B., & Undheim, T. A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.06987 · Cited by 30
STACK: Adversarial Attacks on LLM Safeguard PipelinesPreprint
McKenzie, I. R., Hollinsworth, O. J., Tseng, T., Davies, X., Casper, S., Tucker, A. D., Kirk, R., & Gleave, A. (2025) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.48550/arxiv.2506.24068 · Cited by 15
“Our focus is on catastrophic misuse of proprietary AI models, i.e. malicious use of AI models leading to catastrophic events [Hendrycks et al., 2023; Bengio et al., 2024] such as mass casualties or large-scale economic damage.”
Research Frontiers in Machine Learning & Knowledge Extraction
Holzinger, A., Longo, L., Cangelosi, A., & Ser, J. D. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make8010006 · Cited by 14
“The risks of unsafe AI are multifaceted and deeply concerning, particularly as systems transition from controlled laboratory settings to complex, dynamic, and ecological real-world environment conditions [106].”
A comprehensive introspection on AI risks: taxonomy, challenges, and future directions
Ray, P. P. (2026) · Iran Journal of Computer Science · doi:10.1007/s42044-025-00370-5 · Cited by 16
Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
Hu, R., Cheng, Y., Meng, L., Xia, J., Zong, Y., Shi, X., & Lin, W. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715265 · Cited by 26
“Alongside rapid progress and widespread adoption comes an increasing concern on the large-scale potential risks [3].”
Marin, L. G. U.-B., Rijsbosch, B., Spanakis, G., & Kollnig, K. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-19096-3_6 · Cited by 18
Ispas, L., Mironeasa, C., Severin, T.-L., CERLINCĂ, D.-A., & Mironeasa, S. (2025) · Systems · doi:10.3390/systems13110967 · Cited by 17
“Loss of human control: The rapid development of autonomous AI systems may lead to the loss of human control, with potential negative social and economic consequences [123,124].”
Value Drifts: Tracing Value Alignment During LLM Post-Training
Bhatia, M., Nayak, S., Kamath, G., Mosbach, M., Stańczak, K., Shwartz, V., & Reddy, S. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.790 · Cited by 19
Media and responsible AI governance: a game-theoretic and LLM analysisPreprint
Balabanova, N., Bashir, A., Bova, P., Buscemi, A., Cimpeanu, T., da Fonseca, H. C., Stefano, A. D., Duong, M. H., Domingos, E. F., Fernandes, A. B., et al. (2025) · University of Birmingham Research Portal (University of Birmingham) · doi:10.48550/arxiv.2503.09858 · Cited by 21
AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM SecurityPreprint
Cai, Z., Shabihi, S., An, B., Che, Z., Bartoldson, B. R., Kailkhura, B., Goldstein, T., & Huang, F. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.20965 · Cited by 18
Artificial intelligence and the problem of siguilarity in economics
Kapeliushnikov, R. (2025) · Voprosy Ekonomiki · doi:10.32609/0042-8736-2025-5-5-45 · Cited by 19
It's time to consider global catastrophic food failures
Wescombe, N. J., Martínez, J. B. G., Jehn, F. U., Wunderling, N., Tzachor, A., Sandström, V., Cassidy, M., Ainsworth, R., & Denkenberger, D. (2025) · Global Food Security · doi:10.1016/j.gfs.2025.100880 · Cited by 13
Can Theoretical Physics Research Benefit from Language Agents?Preprint
Lu, S., Jin, Z., Zhang, T. J., Kos, P., Cirac, J. I., & Schölkopf, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.06214 · Cited by 22
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
Su, H., Luo, J., Liu, C., Yang, X., Zhang, Y., Dong, Y., & Zhu, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688650 · Cited by 11
“Infact,Bengioetal.warnthatourpaceof progresshasoutstrippedsafetyefforts,andcallforproactive risk management across multiple dimensions [18].”
Self-citationPotential for near-term AI risks to evolve into existential threats in healthcare
Subasri, V., Baghbanzadeh, N., Celi, L. A., & Seyyed-Kalantari, L. (2025) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2024-101130 · Cited by 13
“30 To prevent this, evaluations and risk assessments are required to understand how individuals, society, processes, institutions and power structures are changed and shifted by AI systems, in addition to the implementation of evidence-based risk mitigation strategies.”
Confabulation dynamics in a reservoir computer: Filling in the gaps with untrained attractors
O’Hagan, J., Keane, A., & Flynn, A. (2025) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/5.0283285 · Cited by 13
Winslow, B. D., Garibay, Ö. Ö., Goyal, T., Koon, S., Margetis, G., Salvendy, G., Shneiderman, B., Tayebi, A., & Vardoulakis, L. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2641703 · Cited by 16
Toward Transdisciplinary Approaches to Audio Deepfake DiscernmentPreprint
Janeja, V., & Mallinson, C. (2024) · arXiv (Cornell University) · doi:10.13016/m2ap8k-krhc · Cited by 11
Giving AI Agents Access to Cryptocurrency and Smart Contracts Creates New Vectors of AI HarmPreprint
Marino, B., & Juels, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.08249 · Cited by 13
“This is not the case, of course, in TradFi, where trans-actions can often be reversed (De Filippi et al., 2022; Stripe, 2025).”
Positive Alignment: Artificial Intelligence for Human FlourishingPreprint
Laukkonen, R., Krier, S., Bakalar, C., Chandaria, S., Kringelbach, M., Elwood, A., Ford, D., Rosas, F. E., Boháček, M., Franklin, M., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.10310 · Cited by 15
“…first, systematically reconfiguring how legacy institutions operate to embed these new sociotechnical tools; and second, fostering the creation of competing, overlapping AI-native institutions [Bengio et al., 2024, Ilcic et al., 2025, Aarab et al., 2025, Arslan and Alqatan, 2020, Ovadya, 2023].”
Multi-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 305
Self-citationAn Information Framework of Intelligence
Hochberg, M. (2025) · Biosystems · doi:10.1016/j.biosystems.2025.105548 · Cited by 12
Governance efficiency and upgrade pathways of international generative AI policies and regulations
Wang, X., Xie, F., & Liu, B. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.103082 · Cited by 10
JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift
Piet, J., Huang, X., Jacob, D., Chow, A., Al-Rashed, M., Zhao, G., Hu, Z., Sitawarin, C., Alomair, B., & Wagner, D. (2025) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3733799.3762981 · Cited by 16
Toward improving reproducibility in neuroimaging deep learning studies
Pup, F. D., & Atzori, M. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1509358 · Cited by 10
Mapping Technical Safety Research at AI Companies: A literature review and incentives analysisPreprint
Delaney, O., Guest, O., & Williams, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.07878 · Cited by 14
“There are significant concerns from governments, civil society, and AI companies 4 that advanced AI systems may pose large-scale risks from misuse or accidents (The White House, 2023; Bengio et al., 2024; Center for AI Safety, 2023).”
BackgroundIs Decentralized Artificial Intelligence Governable? Towards Machine Sovereignty and Human SymbiosisPreprint
Hu, B. A., Rong, H., & Tay, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5110089 · Cited by 17
Martínez-Velasco, J. A., Serrano‐Fontova, A., Bosch-Tous, R., & Torrens, P. C. (2025) · Preprints.org · doi:10.20944/preprints202504.1794.v1 · Cited by 15
A Task-Driven Human-AI Collaboration: When to Automate, When to Collaborate, When to ChallengePreprint
Afroogh, S., Varshney, K. R., & D’Cruz, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.18422 · Cited by 11
“For autonomous AI, most agency transfers to the system [19],[20]; for assistive AI, agency is balanced; for adversarial AI, humans retain primary agency while AI challenges assumptions, and importantly, optimal AI roles evolve dynamically [22],[23].”
Keeping an Eye on AI: A Framework for Effective Human Oversight of AI SystemsPreprint
Gaube, S., Langer, M., Miller, T., Baum, K., Dachselt, R., Feit, A. M., Gadiraju, U., Kaur, H., Keane, M. T., Landers, R., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.16278 · Cited by 11
AI for collective intelligence
Riedl, C., & De Cremer, D. (2025) · Collective Intelligence · doi:10.1177/26339137251328909 · Cited by 9
A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms
Abercrombie, G., Benbouzid, D., Giudici, P., Golpayegani, D., Hernandez, J., Noro, P., Pandit, H., Paraschou, E., Pownall, C., Prajapati, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.01294 · Cited by 9
Trust, experience, and innovation: Key factors shaping American attitudes about AI
Palm, R. I., Kingsland, J. T., & Bolsen, T. (2025) · Public Understanding of Science · doi:10.1177/09636625251372510 · Cited by 9
Bridging artificial intelligence with photovoltaics
Lee, S., Lee, S., Lee, S., Hwang, J.-S., Oh, W., Winther, K. T., Lee, D. U., Kim, D.-H., Nielander, A. C., Jaramillo, T. F., et al. (2025) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2025.102893 · Cited by 14
Designing Incident Reporting Systems for Harms from General-Purpose AI
Wei, K., & Heim, L. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i44.41139 · Cited by 13
Governing artificial intelligence for planetary health.
Creutzig, F., Denton, F., Hine, E., Joshi, S., Ke, G., Kyrychenko, Y., Lan, X., Messner, D., & O'Neill, D. W. (2026) · Lancet Planetary Health · doi:10.1016/j.lanplh.2025.101408 · Cited by 9
Intelligent Systems, Vulnerable Minds: A Framework for Radicalization to Violence in the Age of AI
Kunst, J. R., Obaidi, M., Gollwitzer, A., Brandtzæg, P. B., Hinrichs, Y., Saini, N., & Schroeder, D. T. (2026) · Personality and Social Psychology Review · doi:10.1177/10888683261430089 · Cited by 14
“The core psychological processes driving this stage (specifically the negativity bias, moral outrage, and the mere exposure effect) are well-supported (Bengio et al., 2024; Wack et al., 2025; Williams et al., 2025).”
Hopfield and Hinton’s neural network revolution and the future of AI
Wang, J. Z., & Wyble, B. (2024) · Patterns · doi:10.1016/j.patter.2024.101094 · Cited by 9
Growiec, J., & Prettner, K. (2026) · Economic Modelling · doi:10.1016/j.econmod.2026.107718 · Cited by 10
Safeguarding AI Agents: Developing and Analyzing Safety ArchitecturesPreprint
Domkundwar, I., Mukunda, N. S., Bhola, I., & Kochhar, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.03793 · Cited by 15
“With this increase in AI usage, their presence in our daily lives and the work of companies carries significant safety considerations and possible danger [5],[6].”
BackgroundKlüver, C., Greisbach, A., Kindermann, M., & Püttmann, B. (2024) · Security and Safety · doi:10.1051/sands/2024020 · Cited by 8
“In addition, as soon as safety is affected, specific regulations must be followed, especially in functional safety-related systems (ISO/IEC TR 5469:2024), or in SCS respectively, which are meanwhile enshrined in law, e.g. , through the European AI ACT [1–3].”
Automated Design of Agentic Systems
Hu, S., Lu, C., & Clune, J. (2024) · arXiv · doi:10.48550/arxiv.2408.08435 · Cited by 10
“This raises broader questions about advancing AI capabilities, a topic extensively debated in prior works (Clune, 2019; Ecoffet et al., 2020; Bostrom, 2002; Yudkowsky et al., 2008; Bengio et al., 2024), though beyond this paper’s scope.”
Self-citationChen, J., Tang, X., Wang, M., Li, C., Li, Z., & Qin, Y. (2025) · Energy · doi:10.1016/j.energy.2025.135115 · Cited by 8
Jaber, J., & Issa, H. (2025) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2025.124209 · Cited by 9
Lee, S. B. (2025) · Scientific Reports · doi:10.1038/s41598-025-02294-3 · Cited by 8
Li, W., Chang, Y., He, Y., & Li, H. (2025) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2025.104404 · Cited by 9
Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Experts
Ziakas, C., Loo, N., Jain, N., & Russo, A. M. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.2156 · Cited by 9
Automated robustness testing for LLM-based natural language processing software
Xiao, M., Xiao, Y., Ji, S., Cai, H., Xue, L., & Zhang, P. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130642 · Cited by 7
Self-citationXu, X., He, Z., Zheng, K., Che, L., & Feng, W. (2025) · Metals · doi:10.3390/met15121349 · Cited by 11
The science and practice of proportionality in AI risk evaluations
Mougán, C., Morlock, L., Aguirre, J., Black, J. R., Brauner, J., Campos, S., Dev, S., Llorca, D. F., Franzin, A., Fritz, M., et al. (2026) · Science · doi:10.1126/science.aea3835 · Cited by 8
Self-citationAI takeover and human disempowerment
Bales, A. (2024) · The Philosophical Quarterly · doi:10.1093/pq/pqae034 · Cited by 12
Gutiérrez, E. J. D., & Jarquín-Ramírez, M.-R. (2025) · Edutec Revista Electrónica de Tecnología Educativa · doi:10.21556/edutec.2025.91.3665 · Cited by 9
Aguirre, T. (2024) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2406.01722 · Cited by 8
Coordinated Disclosure of Dual-Use Capabilities: An Early Warning System for Advanced AIPreprint
O'Brien, J., Ee, S., Kraprayoon, J., Anderson-Samways, B., Delaney, O., & Williams, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.01420 · Cited by 10
Contemplative Artificial IntelligencePreprint
Laukkonen, R., Inglis, F., Chandaria, S., Sandved-Smith, L., Lopez-Sola, E., Hohwy, J., Gold, J., & Elwood, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.15125 · Cited by 11
“This phase is dangerous precisely because the AI's raw capabilities outstrip its moral grounding and its wisdom, magnifying the potential for catastrophic outcomes if it has not been aligned with context-sensitive values and epistemic humility (Bengio, 2024).”
Foundation Model Self-Play: Open-Ended Strategy Innovation via Foundation ModelsPreprint
Dharna, A., Lu, C., & Clune, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.06466 · Cited by 10
Self-citationBao, H., Sun, M., & Teplitskiy, M. (2025) · Quantitative Science Studies · doi:10.1162/qss_a_00368 · Cited by 7
Promising Topics for US–China Dialogues on AI Risks and Governance
Siddiqui, S., Ibrahim, L., Loke, K., Clare, S., Lu, M., Richardson, A., McGlynn, C., & Ding, J. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732080 · Cited by 7
Kostenko, O. V. (2025) · doi:10.69635/978-1-0690482-5-7 · Cited by 10
Exploring AI competency in Chinese undergraduates through the UNESCO framework
Wang, J., Hong, W. C. H., Xu, X., Zhang, Y., & Yin, Y. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.106299 · Cited by 8
Self-citationThe conductor model of consciousness, our neuromorphic twins, and the human-AI deal
Benitez, F., Pennartz, C. M. A., & Senn, W. (2024) · AI and Ethics · doi:10.1007/s43681-024-00580-w · Cited by 7
“Others may not even accept the deal, and the discussion must be intensified on how to prevent the risk of AI getting out of control, either overtly or in more subtle, hidden ways [15].”
Behavioral Emotion Analysis Model for Large Language Models
Chang, E. Y. (2024) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr62202.2024.00094 · Cited by 9
“C ONCLUDING R EMARKS Recent declarations by [4] and [17] underscore the importance of addressing AI risks and safety concerns given the technology’s rapid advancement.”
BackgroundTomorrow’s demons: a scoping review of the risks associated with emerging technologies
Salmon, P. M., King, B. J., Elstak, I., McLean, S. G., & Read, G. J. M. (2024) · Ergonomics · doi:10.1080/00140139.2024.2416554 · Cited by 9
“Governance frameworks tend to lag technological progress and are unable to cope with step-change breakthroughs in capability (Bengio et al. 2024).”
Molina, D., Poyatos, J., Ser, J. D., García, S., Ishibuchi, H., Triguero, I., Xue, B., Yao, X., & Herrera, F. (2025) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2025.3530096 · Cited by 11
“The possibility that GPAIS develop emerging generalization capabilities, though still unproven, has drawn much attention for their practical implications in terms of AI safety [11], [12].”
Kondamudi, M. R., & Sahoo, S. R. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.343 · Cited by 7
Risk Management of Future of Defi Using Artificial Intelligence as a Tool
Sah, J., Padma, S., Ramakrishna, Y., & Irfan, M. (2024) · Advances in finance, accounting, and economics book series · doi:10.4018/979-8-3693-6321-8.ch011 · Cited by 8
Designing Culturally Aligned AI Systems For Social Good in Non-Western Contexts
Dennison, D. V., Jain, M., Ganu, T., & Vashistha, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791513 · Cited by 9
Large Language Models Amplify Human Biases in Moral Decision-MakingPreprint
Cheung, V., Maier, M., & Lieder, F. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/aj46b · Cited by 7
Towards Data Governance of Frontier AI ModelsPreprint
Hausenloy, J., McClements, D., & Thakur, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.03824 · Cited by 10
“There is a growing scientific consensus that, as well as tremendous benefit, such models may pose risks to public safety [21, 11].”
Exposing the Impact of GenAI for Cybercrime: An Investigation into the Dark SidePreprint
Luu, T. (., & Samuel, B. M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5348464 · Cited by 7
Dimensional Characterization and Pathway Modeling for Catastrophic AI RisksPreprint
Chin, Z. S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.06411 · Cited by 7
“The rapidly developing field of artificial intelligence (AI) has raised the discourse on risks ranging from immediate and tangible harms to catastrophic threats [1].”
Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignment
Corrêa, N. K. (2024) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · doi:10.48550/arxiv.2406.11039 · Cited by 8
Reward Hacking in Language Model Agents: Revisiting AI Safety GridworldsPreprint
Çağatan, Ö. V., & Zhao, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.15385 · Cited by 8
“Their behaviors are more complex, and the consequences more severe, as reasoning models can autonomously act within coding or interactive environments (Bengio et al., 2024).”
Dissociating Artificial Intelligence from Artificial Consciousness
Findlay, G., Marshall, W., Albantakis, L., David, I., Mayner, W. G., Koch, C., & Tononi, G. (2024) · arXiv · doi:10.48550/arxiv.2412.04571 · Cited by 41
“Artificial intelligence (AI) is progressing rapidly, fueled by theoretical and hardware developments in machine learning [1–3].”
From Black Box to Trustworthy AI: A Secure Framework for Explainable Cybersecurity Decision-Making
Safavi, S., Abdulnabi, M. S. H., Rana, M. E., & Alizadeh, S. (2025) · International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC) · doi:10.1109/assic64892.2025.11158699 · Cited by 6
“Moreover, ethical quandaries, which include biases that may be deeply entrenched within these opaque models, further exacerbate the challenges associated with their interpretation and application [5].”
Ren, B., Cheon, E., & Li, J. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757641 · Cited by 7
“As GenAI rapidly advances [1, 12] and becomes integrated into various scenarios of daily life, red teaming efforts have lagged behind in addressing the growing risks posed to users and must 2.1 Red Teaming in the Context of Generative AI Red teaming, originated in military contexts, traditionally…”
InfluentialFrom Turing to Tomorrow: The UK’s Approach to AI Regulation
Ritchie, O., Anderljung, M., & Rachman, T. (2026) · Springer eBooks · doi:10.1007/978-3-032-11814-1_7 · Cited by 9
Tuyet, T. T. L., & Nguyen, K. M. (2026) · Cognition Technology & Work · doi:10.1007/s10111-025-00851-4 · Cited by 7
Beulen, E., & Dans, M. A. (2026) · Information · doi:10.3390/info17040336 · Cited by 8
Digital soil mapping in the era of big data and artificial Intelligence
Lin, Y., & Li, X. (2026) · Annals of GIS · doi:10.1080/19475683.2026.2665149 · Cited by 6
“However, this wave of innovation comes with risks, such as data bias, lack of model interpretability and ethical implications (Bengio et al. 2024; Ntoutsi et al. 2020; Ozmen Garibay et al. 2023).”
AI Over-Hype: A Dangerous Threat (and How to Fix It)
Johnson, B., & Menzies, T. (2024) · IEEE Software · doi:10.1109/ms.2024.3439138 · Cited by 9
A Novel Benford’s Law-Driven Approach for Detecting Machine-Generated Text
Wang, Z., Zhang, C., & Ren, M. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748305 · Cited by 10
Metadata
- Authors
- Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P., Darrell, T., Harari, Y. N., Zhang, Y.-Q., Xue, L., Shalev-Shwartz, S., Hadfield, G., Clune, J., Maharaj, T., Hutter, F., Baydin, A. G., McIlraith, S., Gao, Q., Acharya, A., Krueger, D., Dragan, A., Torr, P., Russell, S., Kahneman, D., Brauner, J., & Mindermann, S.
- Affiliations
- Mila - Quebec Artificial Intelligence Institute (CA)
- University of Toronto (CA)
- Tsinghua University (CN)
- University of California, Berkeley (US)
- Hebrew University of Jerusalem (IL)
- Institute for AI International Governance, Tsinghua University, Beijing, China.
- School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem, Israel.
- University of British Columbia (CA)
- University of Freiburg (DE)
- University of Oxford (GB)
- East China University of Political Science and Law (CN)
- RAND Corporation (US)
- University of Cambridge (GB)
- Princeton University (US)
- ORCID
- Venue
- Science
- Published
- 2024-05-24
- Volume
- 384
- Issue
- 6698
- Pages
- 842-845
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4398133596
- Cited by
- 349
- References
- 3
- Citations by year
- 2026: 125 · 2025: 161 · 2024: 59
- FWCI
- 142.99
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI risks, AI governance, risk management, proactive governance, AI safety
- MeSH terms
- COVID-19, Artificial Intelligence, Humans, Risk Management
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.