Managing extreme AI risks amid rapid progress
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Preparation requires technical research and development, as well as adaptive, proactive governance
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Any advancement that increases the autonomous capabilities of AI systems introduces its own set of safety considerations (Bengio et al., 2024), especially for systems that improve in an open-ended way (Ecoffet et al., 2020; Clune, 2019).”
From Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents (2025) · cites this paper as Bengio et al., 2024
In the citing paper
“Advanced AI also introduces phenomena that differ fundamentally from previous generations of AI or other technologies, requiring new approaches to mitigating these risks (Bengio et al., 2024).”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Bengio et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationA Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925 · cited 2×
“However, AI is making progress in critical domains such as hacking, social manipulation, and strategic planning 24,31 , and may soon pose unprecedented con-trol challenges.”
BackgroundEmergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · cited 1×
“Especially in the hands of a few powerful actors, AI could cement or exacerbate global inequities, or facilitate automated warfare, customized mass manipulation, and pervasive surveillance 18–23 .”
BackgroundA graph placement methodology for fast chip design
Mirhoseini, A., Goldie, A., Yazgan, M., Jiang, J. W., Songhori, E., Wang, S., Lee, Y.-J., Johnson, E., Pathak, O., Nova, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03544-w · Cited by 618 · cited 1×
“Progress in AI also enables faster AI progress 8 : AI assistants are increasingly used to automate programming 9 , data collection 10,11 , and chip design 12 .”
MethodSuperhuman AI for multiplayer poker
Brown, N., & Sandholm, T. (2019) · Science · doi:10.1126/science.aay2400 · Cited by 592
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1657 · Cited by 475
Research works printed but not linked: 8
- M. Kinniment et al. arXiv:2312.11671 (18 December 2023). · arXiv
- I. R.McKenzie. arXiv:2306.09479 (15 June 2023). · arXiv
- R.Ngo L.Chan S.Mindermann arXiv:2209.00626 (20 August 2022). · arXiv
- E.Hubinger. arXiv:2401.05566 (10 January 2024). · arXiv
- L.Koessler J.Schuett arXiv:2307.08823 (17 July 2023). · arXiv
- N. Kolt et al. arXiv:2404.02675 (3 April 2024). · arXiv
- M. Phuong et al. arXiv:2403.13793 (20 March 2024). · arXiv
- T. A. Kelly, SAE Trans. J. Mater. Manuf. 113, 257 (2004).
Cited by
Foundational Challenges in Assuring Alignment and Safety of Large Language Models
Anwar, U., Saparov, A., Ramírez, J. R., Paleka, D., Turpin, M., Hase, P., Lubana, E. S., Jenner, E., Casper, S., Sourbut, O., et al. (2024) · arXiv (Cornell University) · doi:10.3929/ethz-b-000706962 · Cited by 17
Self-citationLevels of AGI for Operationalizing Progress on the Path to AGI
Morris, M. R., Sohl-Dickstein, J., Fiedel, N., Warkentin, T., Dafoe, A., Faust, A., Farabet, C., & Legg, S. (2023) · arXiv · doi:10.48550/arxiv.2311.02462 · Cited by 274
Artificial intelligence for modelling infectious disease epidemics
Kraemer, M. U. G., Tsui, J. L.-H., Chang, S. Y., Lytras, S., Khurana, M. P., Vanderslott, S., Bajaj, S., Scheidwasser, N., Curran-Sebastian, J. L., Semenova, E., et al. (2025) · Nature · doi:10.1038/s41586-024-08564-w · Cited by 169
“AI approaches that focus on developing generalized systems can potentially provide decision support 107 .”
Thousands of AI Authors on the Future of AI
Grace, K., Sandkühler, J. F., Stewart, H., Weinstein-Raun, B., Thomas, S., Stein-Perlman, Z., Salvatier, J., Brauner, J., & Korzekwa, R. C. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.19087 · Cited by 201
Self-citationTheory Is All You Need: AI, Human Cognition, and Causal Reasoning
(2024) · doi:10.1287/stsc.2024.0189 · Cited by 151
Mapping the ethics of generative AI: a comprehensive scoping review
Hagendorff, T. (2024) · Minds and Machines · doi:10.1007/s11023-024-09694-w · Cited by 141
Embodied large language models enable robots to complete complex tasks in unpredictable environments
Mon-Williams, R., Li, G., Long, R., Du, W., & Lucas, C. G. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01005-x · Cited by 132
The levers of political persuasion with conversational artificial intelligence
Hackenburg, K., Tappin, B. M., Hewitt, L., Saunders, E., Black, S., Lin, H., Fist, C., Margetts, H. Z., Rand, D. G., & Summerfield, C. (2025) · Science · doi:10.1126/science.aea3884 · Cited by 186
Novos desafios para a educação na Era da Inteligência Artificial
Azambuja, C. C., & da Silva, G. F. (2024) · Filosofia Unisinos · doi:10.4013/fsu.2024.251.07 · Cited by 112
Remarkable Robustness of LLMs: Stages of Inference?
Lad, V., Lee, J. H., Gurnee, W., & Tegmark, M. (2025) · neural information processing systems · doi:10.52202/085713-4332 · Cited by 146
Gradual Disempowerment: Systemic Existential Risks from Incremental AI DevelopmentPreprint
Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D., & Duvenaud, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.16946 · Cited by 129
Self-citationDoes Refusal Training in LLMs Generalize to the Past Tense?Preprint
Andriushchenko, M., & Flammarion, N. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.11969 · Cited by 107
Large Language Model Safety: A Holistic SurveyPreprint
Shi, D., Shen, T., Huang, Y., Li, Z., Leng, Y., Jin, R., Liu, C., Wu, X., Guo, Z., Yu, L., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.17686 · Cited by 119
Triguero, I., Molina, D., Poyatos, J., Del Ser, J., & Herrera, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102135 · Cited by 90
Two types of AI existential risk: decisive and accumulative
Kasirzadeh, A. (2025) · Philosophical Studies · doi:10.1007/s11098-025-02301-3 · Cited by 115
“In contrast, the accumulative risks, owing to their incremental nature and less obvious long-term impacts, often do not generate the same level of public interest or urgency.”
Black-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659037 · Cited by 86
Self-citationChow, J. C. L., & Li, K. (2025) · Information · doi:10.3390/info16070549 · Cited by 84
Large language models show amplified cognitive biases in moral decision-making
Cheung, V., Maier, M., & Lieder, F. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2412015122 · Cited by 64
Quantifying the use and potential benefits of artificial intelligence in scientific research
Gao, J., & Wang, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02020-5 · Cited by 81
“The rapid advances in artificial intelligence (AI) may lead to massive value creation and capture across many facets of human society 1–5 , creating a wealth of social and economic opportunities 6–8 , and just as many challenges 9–17 .”
Grote, G., Parker, S. K., & Crowston, K. (2024) · Academy of Management Review · doi:10.5465/amr.2023.0117 · Cited by 116
Universal Neurons in GPT2 Language Models
Gurnee, W., Horsley, T., Guo, Z. C., Kheirkhah, T. R., Sun, Q., Hathaway, W., Nanda, N., & Bertsimas, D. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2401.12181 · Cited by 145
Obschonka, M., Grégoire, D. A., Nikolaev, B., Ooms, F., Lévesque, M., Pollack, J. M., & Behrend, T. S. (2024) · Entrepreneurship Theory and Practice · doi:10.1177/10422587241304676 · Cited by 84
“The AI revolution has triggered urgent discussions on policy and regulation across various sectors, and entrepreneurship is no exception (Bengio et al., 2024).”
Influential$\texttt{ConflictBank}$: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMs
Su, Z., Zhang, J., Qu, X., Zhu, T., Li, Y., Sun, J., Li, J., Zhang, M., & Cheng, Y. (2024) · neural information processing systems · doi:10.52202/079017-3280 · Cited by 150
International AI Safety Report 2026Preprint
Bengio, Y., Clare, S., Prunkl, C., Andriushchenko, M., Bucknall, B., Murray, M., Bommasani, R., Casper, S., Davidson, T., Douglas, R., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.21012 · Cited by 167
Self-citationThe alignment problem from a deep learning perspective
Ngo, R., Chan, L., & Mindermann, S. (2022) · arXiv · doi:10.48550/arxiv.2209.00626 · Cited by 68
Self-citationA Systematic Review of Responsible Artificial Intelligence Principles and Practice
Gunasekara, L., El-Haber, N., Nagpal, S., Moraliyage, H., Issadeen, Z., Manic, M., & Silva, D. D. (2025) · Applied System Innovation · doi:10.3390/asi8040097 · Cited by 67
AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration
Zhou, A., Wu, K., Pinto, F., Chen, Z., Zeng, Y., Yu, Y., Yang, S., Koyejo, S., Zou, J., & Li, B. (2025) · neural information processing systems · doi:10.52202/085713-5654 · Cited by 72
Safeguarding human values: rethinking US law for generative AI’s societal impacts
Cheong, I., Caliskan, A., & Kohno, T. (2024) · AI and Ethics · doi:10.1007/s43681-024-00451-4 · Cited by 86
AI for chemistry teaching: responsible AI and ethical considerations
Blonder, R., & Feldman-Maggor, Y. (2024) · Chemistry Teacher International · doi:10.1515/cti-2024-0014 · Cited by 70
Managing workplace AI risks and the future of work
Howard, J., & Schulte, P. A. (2024) · American Journal of Industrial Medicine · doi:10.1002/ajim.23653 · Cited by 63
HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI InteractionsPreprint
Zhou, X.-H., Kim, H., Brahman, F., Jiang, L., Zhu, H., Lu, X., Xu, F. F., Lin, B. Y., Choi, Y.-J., Mireshghallah, N., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2409.16427 · Cited by 62
Large Language Models Show Amplified Cognitive Biases in Moral Decision-Making
Cheung, V., Maier, M., & Lieder, F. (2025) · doi:10.31234/osf.io/aj46b_v3 · Cited by 53
A collective AI via lifelong learning and sharing at the edge
Soltoggio, A., Ben-Iwhiwhu, E., Braverman, V., Eaton, E., Epstein, B., Ge, Y., Halperin, L., How, J., Itti, L., Jacobs, M. A., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00800-2 · Cited by 51
How Susceptible are LLMs to Influence in Prompts?
Anagnostidis, S., & Bulian, J. (2024) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000708815 · Cited by 70
Training Compute Thresholds: Features and Functions in AI RegulationPreprint
Heim, L., & Koessler, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.10799 · Cited by 54
“The development and deployment of advanced general-purpose artificial intelligence (GPAI) models 1 , also referred to as “frontier AI models” or “dual-use foundation models”, pose increasing risks of large-scale societal harm (Anderljung et al., 2023; Bengio et al., 2024; Hendrycks et al., 2023).”
BackgroundFrontier AI developers need an internal audit function
Schuett, J. (2024) · Risk Analysis · doi:10.1111/risa.17665 · Cited by 54
How to design an AI ethics board
Schuett, J., Reuel, A.-K., & Carlier, A. (2025) · AI and Ethics · doi:10.1007/s43681-023-00409-y · Cited by 46
AI, Global Governance, and Digital SovereigntyPreprint
Srivastava, S., & Bullock, J. B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4996387 · Cited by 54
Risk thresholds for frontier AIPreprint
Koessler, L., Schuett, J., & Anderljung, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.14713 · Cited by 51
“Frontier artificial intelligence (AI) systems 1 pose increasing risks 2 to public safety and security 3 (Bengio et al., 2024; Hendrycks et al., 2023; Anderljung et al., 2023).”
BackgroundTowards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI Accountability
Xia, B., Lu, Q., Zhu, L., Lee, S. U., Liu, Y., & Xing, Z. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644959 · Cited by 74
Reinforcement Learning from Human Feedback in LLMs: Whose Culture, Whose Values, Whose Perspectives?
Barman, K. G., Lohse, S., & de Regt, H. W. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00861-0 · Cited by 44
“Organising this process according to an ideal of inclusiveness makes things better (cf. Bengio et al., 2024).”
Sönmez, S. C., Sevgi, M., Antaki, F., Huemer, J. C., & Keane, P. A. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2024-325458 · Cited by 44
Safety case template for frontier AI: A cyber inability argumentPreprint
Goemans, A., Buhl, M. D., Schuett, J., Korbak, T., Wang, J. J., Hilton, B., & Irving, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.08088 · Cited by 52
“Several scholars have recently discussed the possibility of applying safety cases to frontier AI [11, 6, 64, 5].”
Chow, J. C. L., & Li, K. (2025) · JMIR Cancer · doi:10.2196/66633 · Cited by 46
“However, their integration also presents challenges, such as ensuring the accuracy of information, avoiding biased responses, and safeguarding patient privacy [14].”
Jetha, A., Bakhtari, H., Irvin, E., Biswas, A., Smith, M. J., Mustard, C., Arrandale, V. H., Dennerlein, J. T., & Smith, P. M. (2025) · Systematic Reviews · doi:10.1186/s13643-025-02869-1 · Cited by 43
“Growing numbers of technologists and multidisciplinary scholars have called for the design of responsible AI where AI systems are developed, assessed and imple - mented in safe, trustworthy and ethical ways that are free from bias and do not harm different groups of workers [46].”
Language Model Alignment in Multilingual Trolley ProblemsPreprint
Jin, Z., Kleiman-Weiner, M., Piatti, G., Levine, S., Liu, J., Gonzalez, F., Ortu, F., Strausz, A., Sachan, M., Mihalcea, R. F., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.02273 · Cited by 48
Aligning generalization between humans and machines
Ilievski, F., Hammer, B., van Harmelen, F., Paaßen, B., Saralajew, S., Schmid, U., Biehl, M. L., Bolognesi, M., Dong, X., Gashteovski, K., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01109-4 · Cited by 58
Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data
Treutlein, J., Choi, D., Betley, J., Marks, S., Anil, C., Grosse, R., & Evans, O. (2024) · neural information processing systems · doi:10.52202/079017-4465 · Cited by 78
Neural Retrievers are Biased Towards LLM-Generated Content
Dai, S., Zhou, Y., Pang, L., Liu, W., Hu, X., Liu, Y., Zhang, X., Wang, G., & Xu, J. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671882 · Cited by 35
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.09390 · Cited by 34
Lessons from complex systems science for AI governance
Kolt, N., Shur‐Ofry, M., & Cohen, R. (2025) · Patterns · doi:10.1016/j.patter.2025.101341 · Cited by 43
“7,8 We propose a different lens, grounded in decades of interdisciplinary research in physics, biology, and the social sciences: analyzing AI systems, their development process, and the environments in which they operate as complex systems.”
Evaluating AI Evaluation: Perils and ProspectsPreprint
Burden, J. (2024) · arXiv.org · doi:10.48550/arxiv.2407.09221 · Cited by 20
“Much of this new interest around evaluation has focused on so-called “dangerous capabilities” (e.g., Shevlane et al., 2023; Phuong et al., 2024; Anthropic, 2023; Bengio et al., 2024; Anderljung et al., 2023; Schuett et al., 2023).”
BackgroundAbdelouahed, M., Yateem, D., Amzil, C., Aribi, I., Abdelwahed, E. H., & Fredericks, S. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1518909 · Cited by 41
Responsible AI starts with the artifact: Challenging the concept of responsible AI in IS research
Mikalef, P., Benlian, A., Conboy, K., & Tarafdar, M. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2506875 · Cited by 44
“In addition, it also necessitates a better understanding of how such agency alters the structures and arrangements of the environments in which it is situated (Bengio et al., 2024).”
Danı́elsson, J., & Uthemann, A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4604628 · Cited by 32
Lessons from a Chimp: AI "Scheming" and the Quest for Ape LanguagePreprint
Summerfield, C., Luettgau, L., Dubois, M., Kirk, H. R., Hackenburg, K., Fist, C., Slama, K., Ding, N., Anselmetti, R., Strait, A., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.03409 · Cited by 33
Surprising gender biases in GPT
Fulgu, R. A., & Capraro, V. (2024) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2024.100533 · Cited by 43
Wong, K. K. L., Han, Y., Cai, Y., Ouyang, W., Du, H., & Liu, C. (2025) · Bioengineering · doi:10.3390/bioengineering12101070 · Cited by 39
“This preserves technical reproducibility and directly maps human-factor measures to decision outcomes, reducing the risk—documented in high-stakes settings—of “strong technical scores but user-side mismatch” [17,25,92].”
AI imaginaries shape technological identity and digital futures
Zhong, B., Song, Y., Feng, G. C., Shi, J. J., Zhu, Y., Xie, L., Zhou, W. A., Yu, S.-N., Lu, Y., Qin, Y., et al. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108682 · Cited by 35
Assurance of AI Systems From a Dependability PerspectivePreprint
Bloomfield, R., & Rushby, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.13948 · Cited by 39
An, B., Zhang, S., & Dredze, M. H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.18041 · Cited by 31
The BIG Argument for AI Safety CasesPreprint
Habli, I., Hawkins, R., Paterson, C., Ryan, P., Jia, Y., Sujan, M. A., & McDermid, J. A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.11705 · Cited by 40
Transforming mental health research and care through artificial intelligence
Opel, N., & Breakspear, M. J. (2026) · Science · doi:10.1126/science.adz9193 · Cited by 36
Bond, A., Retief, F., Alberts, R. C., Roos, C., Cilliers, D., & Moolman, J. (2024) · Impact Assessment and Project Appraisal · doi:10.1080/14615517.2024.2426117 · Cited by 28
“…led several distinguished scientists to argue for a greater societal response to counter the potential for extreme risks from AI in the future (Bengio et al. 2024) (one signatory of this open letter was the Nobel prize-winning economist Daniel Kahneman, whose contribution to the understanding…”
Trust AI regulation? Discerning users are vital to build trust and effective AI regulation
Alalawi, Z., Bova, P., Cimpeanu, T., Stefano, A. D., Duong, M. H., Domingos, E. F., Han, T. A., Krellner, M., Ogbo, N. B., Powers, S. T., et al. (2025) · Applied Mathematics and Computation · doi:10.1016/j.amc.2025.129627 · Cited by 37
How malicious AI swarms can threaten democracy
Schroeder, D. T., Cha, M., Baronchelli, A., Bostrom, N., Christakis, N. A., Garcia, D., Goldenberg, A., Kyrychenko, Y., Leyton-Brown, K., Lutz, N., et al. (2026) · Science · doi:10.1126/science.adz1697 · Cited by 53
Self-citationCrossing the principle–practice gap in AI ethics with ethical problem-solving
Corrêa, N. K., Santos, J. W., Galvão, C., Pasetti, M., Schiavon, D. E. B., Naqvi, F., Hossain, R., & Oliveira, N. F. D. (2024) · AI and Ethics · doi:10.1007/s43681-024-00469-8 · Cited by 35
Olugbade, O. E. (2025) · Global Public Policy and Governance · doi:10.1007/s43508-025-00113-z · Cited by 29
“International coordination among these three actors with the most advanced AI capabilities is essential to avoid an arms race and address military and dangerous AI applications such as LAWS, bioweapons, and other capability issues such as value alignment and control problems (Bengio et al., 2024).”
Sankaran, S. (2025) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa65407.2025.11166403 · Cited by 32
“Regular updates maintain relevance, with committees like JTC 1 adapting to AI’s evolving risks [33].”
Pitfalls of Evidence-Based AI Policy
Casper, S., Krueger, D., & Hadfield-Menell, D. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i2.14611 · Cited by 29
“Governing emerging technologies like AI is hard (Bengio et al., 2023).”
Self-citationEmerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models
Ye, R., Chai, J., Liu, X., Yang, Y., Wang, Y., & Chen, S. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2406.10630 · Cited by 33
A survey on biomedical automatic text summarization with large language models
Huang, Z., Chen, X., Wang, Y., Huang, J., & Zhao, X. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104216 · Cited by 27
Could humans and AI become a new evolutionary individual?
Rainey, P. B., & Hochberg, M. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2509122122 · Cited by 26
“2.”
Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration
Dai, S., Liu, W., Zhou, Y., Pang, L., Ruan, R., Wang, G., Dong, Z., Xu, J., & Wen, J.-R. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.421 · Cited by 40
Sophon: Non-Fine-Tunable Learning to Restrain Task Transferability For Pre-trained Models
Deng, J., Pang, S., Chen, Y., Xia, L., Bai, Y., Weng, H., & Xu, W. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00234 · Cited by 23
What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
Jain, S., Lubana, E. S., Öksüz, K., Joy, T., Torr, P. H. S., Sanyal, A., & Dokania, P. K. (2024) · neural information processing systems · doi:10.52202/079017-2963 · Cited by 72
Self-citationMorris, M. R., Bernstein, M. S., Bigham, J. P., Bruckman, A. S., & Monroy‐Hernández, A. (2024) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3678884.3689134 · Cited by 24
Adapting Probabilistic Risk Assessment for AIPreprint
Wisakanto, A. K., Rogero, J., Casheekar, A. M., & Mallah, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.18536 · Cited by 25
Artificial Intelligence in Science and Society: The Vision of USERN
Dorigo, T., Brown, G. D., Casonato, C., Cerdà, A., Ciarrochi, J., Lio, M. D., D’Souza, N., Gauger, N. R., Hayes, S. C., Hofmann, S. G., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3529357 · Cited by 27
Lehr, S. A., Saichandran, K. S., Harmon‐Jones, E., Vitali, N., & Banaji, M. R. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2501823122 · Cited by 39
“To those who have spoken about the dangers of generative AI ( 38 , 39 ), this result, by raising the possibility that LLMs will develop more advanced forms of self - interested and self - protective behavior, should be seen as alarming if left untested and unchecked.”
Lehalleur, S. P., Hoogland, J., Farrugia-Roberts, M., Wei, S. H., Oldenziel, A. G., Wang, G., Carroll, L., & Murfet, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.05475 · Cited by 25
Mitigating Societal Cognitive Overload in the Age of AI: Challenges and DirectionsPreprint
Lahlou, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.19990 · Cited by 29
An, B., Zhang, S., & Dredze, M. H. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.281 · Cited by 57
Martínez, J. B. G., Behr, J., Pearce, J. M., & Denkenberger, D. (2025) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2024.2431207 · Cited by 25
“…rapid progress in artificial intelligence (AI) could become an important GCIL risk factor: leading AI scientists are now calling for stronger action on AI risks from world leaders, citing concerns including large-scale cybercrime and development of novel biological weapons (Bengio et al. 2024).”
Why do experts disagree on existential risk? A survey of AI experts
Field, S. (2025) · AI and Ethics · doi:10.1007/s43681-025-00762-0 · Cited by 21
Duan, S., Yi, X., Zhang, P., Liu, Y., Liu, Z., Lu, T., Xie, X., & Gu, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.56 · Cited by 26
Eliciting the Priors of Large Language Models using Iterated In-Context LearningPreprint
Zhu, J.-Q., & Griffiths, T. L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.01860 · Cited by 21
Cho, C., & Ofosu-Anim, D. O. (2025) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2025.101418 · Cited by 23
Baker, M., Kulp, G., Marks, O., Brundage, M., & Heim, L. (2025) · RAND Corporation eBooks · doi:10.7249/wra4077-1 · Cited by 21
An AI System Evaluation Framework for Advancing AI Safety: Terminology, Taxonomy, Lifecycle Mapping
Xia, B., Lu, Q., Zhu, L., & Xing, Z. (2024) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3664646.3664766 · Cited by 35
Herzog, C., Blank, S., & Stahl, B. C. (2024) · AI & Society · doi:10.1007/s00146-024-02082-z · Cited by 26
“Another limitation of our contribution is that we do not engage with potential existential risks posed by (increasingly) autonomous AI as visibly discussed by notable figures like Yoshua Bengio, Geoffrey Hinton, Stuart Russell, Daniela Kahneman and others (Bengio et al. 2024).”
Schnake, T., Jafari, F. R., Lederer, J., Xiong, P., Nakajima, S., Gugler, S., Montavon, G., & Müller, K. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102923 · Cited by 27
“However, alongside the growing adoption of ML algorithms, there are increasing concerns about the potential risks associated with these models [12, 24].”
BackgroundBengio, Y., Tegmark, M., Russell, S., Song, D., Mindermann, S., Xue, L., Casper, S., Ong, C.-H. L., Wilfred, V., Maharaj, T., et al. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i5.15503 · Cited by 24
Self-citationYou Know What I'm Saying: Jailbreak Attack via Implicit ReferencePreprint
Wu, T., Mei, L., Yuan, R., Li, L., Xue, W., & Guo, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.03857 · Cited by 20
“…their widespread adoption has introduced critical security vulnerabilities that pose significant societal risks, including the generation of harmful or biased content, the dissemination of misinformation, and the facilitation of malicious activities such as cyberattacksp (Bengio et al., 2024).”
BackgroundEvaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI SystemsPreprint
Fan, Y., Zhang, W., Pan, X., & Yang, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.17815 · Cited by 18
“These extracted features were subsequently used to successfully steer the model’s behavior towards safer outputs, highlighting the potential of SAEs for alignment tasks [5].”
Prospects of business process management based on chatbots
Melnyk, L. H., Kalinichenko, L., Rozghon, Y., Derykolenko, O. M., Kovtun, O. A., & Tulyakov, O. (2024) · Problems and Perspectives in Management · doi:10.21511/ppm.22(2).2024.16 · Cited by 17
Mun, J., Jiang, L., Liang, J., Cheong, I., DeCario, N., Choi, Y., Kohno, T., & Sap, M. (2024) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v7i1.31698 · Cited by 38
The Earth alignment principle for artificial intelligence
Gaffney, O., Luers, A. L., Carrero-Martínez, F., Oztekin-Gunaydin, B., Creutzig, F., Dignum, V., Galaz, V., Ishii, N., Larosa, F., Leptin, M., et al. (2025) · Nature Sustainability · doi:10.1038/s41893-025-01536-6 · Cited by 23
Deberdt, R., Billon, P. L., Makinde, O., Dauvergne, P., Solwati, T., Razmi, S., Kumar, G., & Belhabib, D. (2025) · The Extractive Industries and Society · doi:10.1016/j.exis.2025.101690 · Cited by 23
Predicting Emergent Capabilities by FinetuningPreprint
Snell, C., Wallace, E., Klein, D., & Levine, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.16035 · Cited by 22
Metadata
- Authors
- Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P., Darrell, T., Harari, Y. N., Zhang, Y.-Q., Xue, L., Shalev-Shwartz, S., Hadfield, G., Clune, J., Maharaj, T., Hutter, F., Baydin, A. G., McIlraith, S., Gao, Q., Acharya, A., Krueger, D., Dragan, A., Torr, P., Russell, S., Kahneman, D., Brauner, J., & Mindermann, S.
- Affiliations
- Mila - Quebec Artificial Intelligence Institute (CA)
- University of Toronto (CA)
- Tsinghua University (CN)
- University of California, Berkeley (US)
- Hebrew University of Jerusalem (IL)
- Institute for AI International Governance, Tsinghua University, Beijing, China.
- School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem, Israel.
- University of British Columbia (CA)
- University of Freiburg (DE)
- University of Oxford (GB)
- East China University of Political Science and Law (CN)
- RAND Corporation (US)
- University of Cambridge (GB)
- Princeton University (US)
- ORCID
- Venue
- Science
- Published
- 2024-05-24
- Volume
- 384
- Issue
- 6698
- Pages
- 842-845
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4398133596
- Cited by
- 349
- References
- 3
- Citations by year
- 2026: 125 · 2025: 161 · 2024: 59
- FWCI
- 142.99
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI risks, AI governance, risk management, proactive governance, AI safety
- MeSH terms
- COVID-19, Artificial Intelligence, Humans, Risk Management
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.