Model evaluation for extreme risks

“For example, after GPT-3 had already existed for many months, researchers demonstrated that chain-of-thought prompting could significantly increase performance (Wei et al., 2022c).”

“(See related: Mitchell et al., 2018) 3.”

“Developers could have multiple organisational layers of safety evaluation, such as by establishing an internal safety evaluation function that is independent of the teams primarily responsible for building the models, reporting directly to organisational leaders (see Raji et al., 2020).”

“Frontier AI developers are making rapid progress in developing increasingly capable general-purpose models (Bubeck et al., 2023).”

“One promising avenue is automating the process of writing evaluations using AI systems (Perez et al., 2022b) (see also Pan et al., 2023).”

“However, sometimes specific capabilities will emerge only at greater scale, which makes this analysis much harder (Ganguli et al., 2022); other capabilities display U-shaped scaling (Wei et al., 2022a).”

“Humans could then intentionally misuse these capabilities (Brundage et al., 2018), e.g. for assistance in disinformation campaigns, cyberattacks, or terrorism.”

“model evaluation by an independent, external auditor for the purpose of providing a judgement — or input to a judgement — about the safety of deploying a model (or training a new one) (ARC Evals, 2023; Mökander et al., 2023; Raji et al., 2022b).”

“One of the most severe threats of using LLMs is their ability to promote certain narratives persuasively [42].”

“Intentionally training models with dangerous capabilities, even for research purposes, raises ethical and security concerns akin to “gain-of-function” research in virology, necessitating stringent risk assess - ments and security protocols [24].”

“This process typically involves specifying a set of model evaluations designed to detect dangerous capabilities related to the identified threats [41, 50].”

“…make decisions, or adjust outputs in real-time, ensuring that the AI remains aligned with human values and goals (Mullainathan & Obermeyer, 2017; Shevlane et al., 2023); f) alternative energy sources / quantum computing – this refers to the exploration and use of renewable energy sources, like…”

“Among their most severe threats are manipulation and AI deception [47].”

“Model Evaluations for Dangerous Capabilities (particularly promising) : Benchmarks for dangerous capabilities could be designed by developers, independent third-parties, or government organizations [Shevlane et al., 2023].”

“Overall, safety evaluations have been of high importance as persuasion and deception have been noted as major risks posed by language models (Shevlane et al., 2023; Hendrycks et al., 2023; Weidinger et al., 2022; Burtell & Woodside, 2023).”

“Current security measures include various model evaluations focused on gender and racial bias, truthfulness, toxicity, and the presence of copyrighted material, amongst others [12].”

“However, these capabilities raise significant safety and security concerns, including copyright violations (Karamolegkou et al., 2023), harmful content generation, and retention of dangerous knowledge (e.g., bioweapon instructions) (Shevlane et al., 2023).”

“Dangerous capability evaluations attempt to estimate the proficiency of frontier models on tasks where language models could unlock large scale harm, for example, cyber-offense, persuasion, bio-engineering, and self-replication [24, 50, 51].”

“Beyond the idea of ‘dangerous capabilities’ (Shevlane et al., 2023), it is now recognised that a sufficient level of capabilities and certain propensities, together, are risk catalysers (Ouyang et al., 2022; Bai et al., 2022; Ganguli et al., 2022; Gehman et al., 2020).”

“20 OVERTRUST IN AI AND AUTOMATION BIAS AI systems can increasingly generate convincing misin-formation with high confidence; this can manipulate people, create fake news, encourage polarisation, create power imbalances and lock in oppressive systems.”

“Agency is thus a key focus of behavioural evaluations (Shevlane et al., 2023) and governance (Shavit et al.).”

“…(Park et al., 2023; Carroll et al., 2023; Perez et al., 2023; Sharma et al., 2024; Bang et al., 2023; Hubinger et al., 2019; Berglund et al., 2023; Ngo et al., 2024; Shevlane et al., 2023; Shah et al., 2022; Pan et al., 1 Department of Computer Science, University of Wisconsin-Madison, WI, USA.”

“This motivates finding sub-components or proxies of dangerous capabilities that can be measured in small models and extrapolated to larger ones (Shevlane et al., 2023).”

“A model possesses situational awareness (SA) if it has self-knowledge (knowing its identity and facts about itself), can make inferences about its situation, and acts based on this knowledge (Shevlane et al., 2023; Laine et al., 2023; Berglund et al., 2023; Laine et al., 2024).”

“Deception is a behavior that stimulates research interests from both cognitive and brain sciences [Ganis and Keenan, 2009] and AI researchers [Hagendorff, 2024, Shevlane et al., 2023, Williamson and Prybutok, 2024].”

“This is because the objective function can be written as the average of f ( r i ) , and the only modification to the dynamics would be replacing the σ ( − r i ) factor in Equation (3) with − f ′ ( r i ) .”

“…of AI systems that advance the frontier of AI technology (Alaga & Schuett, 2023; Anderljung et al., 2023; DSIT, 2023b), especially systems that may pose catastrophic risks (Anthropic, 2023; OpenAI, 2023; Shevlane et al., 2023), noting that AI takeover is one of many potential AI catastrophe events.”

“…failure modes of AI — such as biased behavior (Santurkar et al. 2023), reward hacking (Pan, Bhatia, and Steinhardt 2022), and even existential threats (Shevlane et al. 2023) — have been framed as stemming from misalignment between AI’s goals and the values and preferences of its stakeholders.”

“Prior studies on LLM self-replication risks have mainly focused on measuring the capability (verbal-ized success rate) of self-replication, either through direct instructions or within synthetic capability benchmarks (Pan et al., 2024; 2025; Kran et al., 2025; Black et al., 2025).”

“Shevlane et al. (2023) describe two kinds of safety evaluations for large-scale AI risks.”

“…troublesome not only because the low accuracy of such systems as mentioned before, but also due to the consequent inadequate response to LLM misuse, which can lead to significant societal harm [Stokel-Walker and Van Noorden, 2023, Porsdam Mann et al., 2023, Shevlane et al., 2023, Wu et al., 2025].”

“…2022), (2) generation of toxic & harmful speech (Pavlopoulos et al., 2020; Venkit et al., 2023; Deshpande et al., 2023), (3) generation of hallucinated text (Zhou et al., 2021; Ji et al., 2023), (4) generation of misinformation (Jawahar et al., 2020; Pan et al., 2023; Shevlane et al., 2023), etc.”

“In parallel, broader risk analyses [11, 26, 60] have raised concerns about model-driven manipulation at scale but have not systematically examined how distinct deceptive strategies interact with individual user traits to shape susceptibility.”

“…attempt naturalistic evaluation: OpenDeception finds in real-world scenarios, many models have deceptive intentions spelled out in their chain of thought (Yao et al. 2024 (Anthropic 2023), deception capabilities advance faster than detection methods, necessitating careful framework design.”

“While current safety evaluation paradigms in AI often emphasize model capabilities [Shevlane et al., 2023], human psychology teaches a different lesson: harmful or antisocial behaviors are often better predicted by personality than by intelligence.”

“Unregulated advancement of large language models (LLMs) poses extreme societal risks, such as automated warfare, societal inequalities, and misinformation (Bengio et al., 2024; Shevlane et al., 2023; Anwar et al., 2024; Chen & Shu, 2024).”

“…when available. as mandatory risk assessments before AI deployment by third parties with comprehensive access (Casper et al. 2024; Raji et al. 2022; Shevlane et al. 2023), clarifying liability for AI harms, compulsory reporting of safety cases 29 , and know-your-customer requirements (Egan & Heim…”

“In other words, while The Safety Conception may capture something close to the manifest concept of AI safety among AI safety researchers, it does not capture the concept of AI safety that is operative among AI safety researchers or AI safety governance organizations.”

“Some have employed the term in a more restricted sense, to refer specifically to a model’s knowledge of whether at any given time it is deployed or undergoing testing (e.g. to test its truthfulness or harmlessness) [105].”

“Sandbagging is a meta-problem for evaluations, but our work can plausibly be as a dangerous capability evaluation itself (Shevlane et al., 2023).”

“Establishing safety certification standards that specifically target agentic systems in ambiguous environments (Shevlane et al., 2023) is essential for mitigating risks from unintentional transfer or deliberate poisoning.”

“Moreover, our work relates to existing efforts on extreme risks and model evaluations for AI agents [31, 18].”

“More than a topic of merely theoretical interest, however, the availability of black-box access to the latest AI models and high-fidelity simulators could lead to simulation being a key tool in the safe and beneficial deployment of advanced AI agents [21].”

“Future systems may show more dangerous contingency capabilities, such as the ability to conduct offensive cyber operations, manipulate people through dialogue, or provide actionable instructions to commit terror (Shevlane et al., 2023).”

“The former involves learning from feedback [6], [11] or adapting to distributional shifts [26], [27], while the latter ensures alignment through evaluation and assurance [28], [29].”

“Specifically, it means an agent’s own identity, its stage (e.g., testing, training), and its impact on the world (Shevlane et al., 2023; Laine et al., 2023; Berglund et al., 2023; Laine et al.).”

“Work on model evaluation for extreme risks studies how evaluations for dangerous capabilities and alignment can inform responsible training, deployment, transparency, and security decisions (Shevlane et al., 2023).”

“…problem due to the potential harms that can arise in large models (Park et al., 2023; Carroll et al., 2023; Perez et al., 2022; Sharma et al., 2023; Bang et al., 2023; Hubinger et al., 2019; Berglund et al., 2023; Ngo et al., 2022; Shevlane et al., 2023; Shah et al., 2022; Pan et al., 2022).”

“Competent (capability) • Capability evaluations [67] Incompetent (failure) • Create trustworthy AI [68]”

“• Define capabilities thresholds : Define measurable AI capability levels (e.g., autonomous replication) that trigger specific required responses (e.g., halt development, implement security measures) (Karnofsky, 2024; Koessler et al., 2024; Shevlane et al., 2023; Raman et al., 2025).”

“Despite such breakthroughs, LLMs might bring potential social risks (Gehman et al., 2020; Wang et al., 2023e; Esiobu et al., 2023; Tao et al., 2024), raising significant societal concerns (Bommasani et al., 2022; Kaddour et al., 2023; Shevlane et al., 2023).”

“Much of the prior literature on AI security and LLM-related hazard analysis focuses on humans using LLMs with malicious intentions [14, 20].”

“In Shevlane et al. (2023), one can find an introduction to model evaluations.”

“Shevlane et al. (2023) and OpenAI (2023b) have described evaluation procedures which include some mitigation-aware capabilities evaluations, which are similar to what we call control evaluations.”

“This gap matters for scalable oversight: termination is part of the control system, and the evidence behind a terminal decision may feed later assurance arguments [Shevlane et al., 2023, Buhl et al., 2024].”

“…Touvron et al., 2023; Bai et al., 2023; Team et al., 2023), concerns grow about their potential to generate harmful content, amplify misinformation, or facilitate high-risk activities (Duffourc & Gerke, 2023; Tredinnick & Laybats, 2023; Shevlane et al., 2023; Zhuo et al., 2023; Huang et al., 2024).”

“Thus, factual gaps are often identified and fixed before deployment through extensive risk assessment manually [9, 10].”

“Advances in AI capabilities bring corresponding increases in potential for harm, including from cyber attacks, biorisks, mass persuasion, and fraud (Bengio, Hinton, et al., 2024; Anderljung et al., 2023; Shevlane et al., 2023).”

“Safety risks and potential dangers arising from the deployment of LLMs are increasingly studied in the literature (Phuong et al., 2024; Shevlane et al., 2023; Bengio et al., 2023; Bowman, 2023).”

“Examples include filtering child sexual abuse material (CSAM) from training data (e.g., Thiel, 2023), fine-tuning models to refuse criminal requests (e.g., Yuan et al., 2024), and testing for national security risks (e.g., Shevlane et al., 2023).”

“Additionally, some datasets include malicious text and code related to cyberattacks, posing risks of misuse by humans or the AI itself (Bender et al., 2021; Bai et al., 2022; Ji et al., 2023; Shevlane et al., 2023).”

“Some capabilities are more likely to blur the civilian-military distinction because of their usefulness in a variety of contexts (Shevlane et al. 2023).”

“This is why Shevlane et al. (2023) foresee automation of AI research as an extreme risk.”

“In the long-term, AI might develop capabilities that present novel existential threats, creating “catastrophic” consequences for society such as chemical warfare and environmental disaster (Hendrycks et al., 2023; Shevlane et al., 2023; Matteucci et al., 2023).”

“…troublesome not only because the low accuracy of such systems as mentioned before, but also due to the consequent inadequate response to LLM misuse, which can lead to significant societal harm [Stokel-Walker and Van Noorden, 2023, Porsdam Mann et al., 2023, Shevlane et al., 2023, Wu et al., 2025].”

“…evaluation of large language models has become a central pillar of AI safety assurance, with major laboratories, regulatory bodies, and independent auditors conducting increasingly comprehensive testing campaigns to characterise model capabilities and risks prior to deployment (see, e.g., [32, 3]).”

“Following risk assessment frameworks in which a threat materializes only when multiple necessary conditions are jointly satisfied (Reason, 2000; Shevlane et al., 2023), we define a sequence as posing predicted functional risk only when all stages are passed: the model complies, the output encodes a…”

“Behavioral red-teaming, while valuable, faces fundamental limitations: auditors cannot enumerate all possible failure modes, and models may behave differently under evaluation than deployment (Casper et al., 2024; Shevlane et al., 2023).”

“…assurance methods (behavioural evaluations, red-teaming, system cards, conformity assessments) were developed to characterise observable model output and process compliance; they are not designed to establish claims about latent representations or long-horizon agentic behaviour [4, 5, 6].”

“Gating is adopted as a dual-use mitigation step consistent with the extreme-risk evaluation framework of Shevlane et al. [54], and the artifact continues to function as a standardized evaluation substrate for downstream safety work under that access policy.”

“The rapid advancement of large language models (LLMs) necessitates rigorous safety protocols to mitigate unintended consequences and potential misuse (Kim et al., 2022; Bianchi et al., 2023; Grattafiori et al., 2024; Bai et al., 2022b; Shevlane et al., 2023; Zou et al., 2023).”

“Developing such systems is crucial for minimizing the risks of LLM misuse and ensuring responsible AI governance in the age of advanced language models [34, 35].”

“As AI systems grow increasingly capable and influential, their potential impact on individuals and society amplifies the necessity of aligning their actions with desirable outcomes [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11].”

“Such risky behaviors would emerge [41, 81, 82] with increasing U comp. , which have been observed like cyber-attacks [83], deception [84, 85] and scheming [86].”

“Their research emphasizes the ability of general-purpose AI systems to produce both beneficial and potentially harmful capabilities, necessitating careful evaluation and responsible decision-making for model training, deployment, and security [38].”

“Model training: This is the stage where the algorithms are trained on the data to learn the optimal parameters or weights that minimize the error or loss function [18].”

“Another example also applies to Google (Deepmind) [36] is a work on a framework for evaluating the extreme risks of multipurpose AI systems [37].”

“For example Google [Shevlane et al., 2023, Phuong et al., 2024], OpenAI [OpenAI, 2023], and Anthropic [Anthropic, 2023] have public policies for how they assess and evaluate the misuse potential of individual models.”

“Note that all of these evaluations measure the capability of an AI system to perform scheming reasoning, but not its propensity to do so (Shevlane et al., 2023).”

“When a change is approved but wrong, rollback should still work under stress [9, 40, 45].”

“A common thread across these efforts is the identification of dangerous capabilities that require dedicated evaluation and mitigation once a model crosses a pre-defined risk threshold [26, 27].”

“Governments, civil society organisations, and industry have largely relied on pre-deployment mechanisms, such as model evaluations and risk assessments, to anticipate AI-related harms [50].”

“Dangerous-capability and autonomous-task evaluations (Kinniment et al., 2023; Shevlane et al., 2023; Phuong et al., 2024) introduce limited adaptation via counterparties, but these are typically scripted, so the reported behavior does not select the same equilibria likely to be selected in…”

“…approaches, including structured threat modelling (Stix et al., 2025; Murray et al., 2025), agentic pre-deployment evaluations (Bengio et al., 2024; Shevlane et al., 2023; Meinke et al., 2024) and realistic honeypots (Shah et al., 2025; Balesni et al., 2024), post-deployment monitoring and…”

“…2020; Ngo et al., 2024; Carlsmith, 2022; Turner et al., 2021), empirical evaluation (Hendrycks et al., 2021; Perez et al., 2022; Pan et al., 2023; Shevlane et al., 2023), and risk taxonomy and governance (Weidinger et al., 2021; 2022; National Institute of Standards and Technology, 2023); every…”