Model evaluation for extreme risks

“For example, after GPT-3 had already existed for many months, researchers demonstrated that chain-of-thought prompting could significantly increase performance (Wei et al., 2022c).”

“(See related: Mitchell et al., 2018) 3.”

“Developers could have multiple organisational layers of safety evaluation, such as by establishing an internal safety evaluation function that is independent of the teams primarily responsible for building the models, reporting directly to organisational leaders (see Raji et al., 2020).”

“Frontier AI developers are making rapid progress in developing increasingly capable general-purpose models (Bubeck et al., 2023).”

“One promising avenue is automating the process of writing evaluations using AI systems (Perez et al., 2022b) (see also Pan et al., 2023).”

“However, sometimes specific capabilities will emerge only at greater scale, which makes this analysis much harder (Ganguli et al., 2022); other capabilities display U-shaped scaling (Wei et al., 2022a).”

“Humans could then intentionally misuse these capabilities (Brundage et al., 2018), e.g. for assistance in disinformation campaigns, cyberattacks, or terrorism.”

“model evaluation by an independent, external auditor for the purpose of providing a judgement — or input to a judgement — about the safety of deploying a model (or training a new one) (ARC Evals, 2023; Mökander et al., 2023; Raji et al., 2022b).”

“Recent governance work has focused on evaluations, model release, safety cases, data governance, and broader public-safety regulation [1, 2, 4, 17, 19, 36].”

“…as autonomous agents in charge of managing infrastructure, making decisions, and controlling systems, numerous research and regulations [e.g., Shevlane et al., 2023] call for additional evaluations with regard to the propensities of the AI systems (which are distinct from the capabilities of…”

“Such measures could include reporting requirements, or subjecting the model to further red teaming and safety testing (Shevlane et al. 2023; Kinniment et al. 2023; Phuong et al. 2024).”

“…specifically designed to probe for potentially harmful abilities, such as persuasiveness, deception capabilities, or proficiency at tasks with dual-use potential, with flagged systems being prioritized for more extensive review and/or access restrictions (Shevlane et al., 2023; Hooker, 2024).”

“Rapid progress in the capabilities of language models has led to a surge of interest in their alignment and evaluation, aiming to ensure that they are safe, reliable, and beneficial (Shevlane et al., 2023; Ji et al., 2023).”

“Runtime monitoring and safety layers have been proposed in autonomous and agentic systems to constrain behavior during deployment [1, 15].”

“In parallel, red-teaming practices and safety evaluations remain model-centric and rarely examine how a human + AI pair could subvert non-AI workflows such as DOT QA data chains ( 45 , 46 ).”

“Prior work evaluates AI systems for dangerous capabilities and deceptive behaviors [11], with companies documenting cases of model deception [12].”

“Recent taxonomies and risk surveys characterize such hazards and recommend layered safeguards with continuous evaluation (Weidinger et al., 2022; 2021; Shevlane et al., 2023).”

“The compositional space creates 36,585+ pairwise combinations, each potentially weaponizable while appearing legitimate to individual service monitors. outcome (Hou, et al., 2025; Huang, et al. 2025; Shevlane, et al., 2023; Guo, et al., 2025).”

“Moreover, evaluations could be run during as well as after frontier training runs to ensure consistent application of safety standards and identify possible threats at an earlier stage (METR, 2023; Shevlane et al., 2023).”

“In regulatory contexts, they help to identify areas where intervention may be needed and inform decisions by providing evidence about system capabilities and risks (Reuel et al., 2025; Shevlane et al., 2023; Paskov et al., 2025).”

“Some works have focused on evaluation frameworks for catastrophic risks [8, 41] and provided desirable qualities of evaluations and evaluation suites that are specifically relevant to the sub-field of extreme risk evaluations.”

“Although experts disagree on the extent and relative severity of different AI threats, one thing they almost all agree on is the need for independent external scrutiny of consequential AI systems [15, 17, 3, 10].”

“Advanced monitoring, responsive governance strategies, implementing regular audits, developing domain-specific models to manage threats [59,67,68].”

“[10] Given the potential of advanced AI systems to rapidly improve during training, this method is insufficient.”

“More-over, hybrid frameworks that incorporate both contextual features and quantitative values have gained traction in recent LLM evaluation literature as a means to reflect real-world performance under uncertain or task-specific conditions [36], [37].”

“…hand, harmlessness-related benchmarks assess the safety and potential risks of LLMs, including issues like social bias (Parrish et al., 2021; Zhao et al., 2023), general safety considerations (Inan et al., 2023; Wang et al., 2023a; Tedeschi et al., 2024), and extreme risks (Shevlane et al., 2023).”

“In both cases, some of the specified values are disregarded. nAI, 2024; 2025; Meta, 2025; Gemini et al., 2024), but also introduced social concerns such as generating hate speech and reinforcing biases (Bommasani et al., 2021; Shevlane et al., 2023).”

“Before an AI agent is deployed, its developer may estimate the probability that it could complete a dangerous multi-step task, such as setting up its own infrastructure or acquiring resources (Shevlane et al., 2023; Phuong et al., 2024).”

“Alignment evaluations (evals) aim to assess an LLM’s propensity to engage in misaligned behaviour [2].”

“• Dangerous-capability evaluations [23], [24] that probe uplift in biological, chemical, and cyber domains, persuasion, self-proliferation, and autonomous replication.”

“If scaling broadens the semantic knowledge of image-only pretrained encoders, evaluations should examine both emerging capabilities and their downstream use (Shevlane et al., 2023).”

“By reducing safety evaluation to a single pass/fail label at the end of an interaction, existing works overlook the temporal dynamics of risk evolution , missing the optimal intervention window to prevent irreversible harm (Ghosh et al., 2025; Shevlane et al., 2023).”

“Conditionally staged deployment, in which advancing requires affirmative evidence of safety and the authority to halt is structurally independent from the authority to ship, confronts the same mechanism that the Challenger pre-launch review failed to address [54, 61].”

“Evaluations of dangerous capability and robust refusal are policy-relevant even when they do not track general benchmark averages [33, 50].”

“On the alignment side, RLHF [24] and Constitutional AI [25] focus on text generation safety rather than action-level security in interactive environments [26, 27].”

“The dominance of information disclosure and prompt injection threats identified through STRIDE-based threat modeling suggests that security risks in AI-enabled websites are predominantly semantic and contextual, rather than purely structural, as highlighted in recent AI security literature [29].”

“Our findings have implications for ongoing discussions concerning transparency, reproducibility, accountability, and governance in AI research ( 7 , 17 , 18 ).”

“Safety assessments conducted prior to the deployment of an artificial intelligence (AI) system, such as model evaluations and red-teaming, allow testing against anticipated failure modes under controlled settings (Shevlane et al., 2023).”

“Current frontier-AI governance work, including dangerous-capability evaluations (Shevlane et al., 2023), responsible scaling policies (Anthropic, 2026; Ope-nAI, 2025), and structured frontier-regulation proposals (Anderljung et al., 2023), maps cleanly onto the micro-prudential side: each targets…”

“…already emerging in practice (risk-based scoping in the NIST AI RMF [38], management-system requirements in ISO/IEC 42001 [39], dangerous-capability evaluations [42], transparency indicators [43], and the EU AI Act [40]), so measurement and mutual recognition are feasible rather than hypothetical.”

“…useful information on discrete harms to certain rights, such as privacy violations or discriminatory bias rates, the most serious risks may arise from correlated, cross-domain harms that create vulnerability in sociopolitical and economic systems (Dafoe 2018; Shevlane et al. 2023; Kasirzadeh 2025).”

“Furthermore, there is a trade-off related to the intelligence of the red teaming agent itself [23].”

“The lack of formal verification and principled security guarantees remains a significant challenge, emphasizing the need for rigorous design methodologies for LLM systems.”

“In Model Evaluation for Extreme Risks [Shevlane et al., 2023], the authors propose that evaluations should be central to determining whether AIs are dangerous and aligned.”

“…and autonomy, so too has the focus on safety infrastructure, emphasizing human or AI feedback in model training (Christiano et al., 2017; Guan et al., 2024; Ouyang et al., 2022), robust evaluations (Shevlane et al., 2023), and the implementation of fundamental value systems (Bai et al., 2022).”

“First, not all sciences contain mixed claims (while all sciences may be value-laden), so mixed claims carve out a special class of sciences.”

“Gaining deeper insight into the inner workings of LLMs could bring us closer to solving many currently open problems, such as generating faithful explanations for their outputs (Turpin et al., 2023) or accurately evaluating their true capabilities (Shevlane et al., 2023; van der Weij et al., 2024).”

“While traditional QA benchmarks have been instrumental in assessing models across a wide range of benign domains, there is a growing need to better understand capabilities that relate to sensitive topics, such as bioterrorism or cyberwarfare [14].”

“For example, widespread use of large language models brings a wide range of societal risks (Weidinger et al., 2022; Shevlane et al., 2023), and NNs are increasingly used in high-stakes application areas such as healthcare (Rajpurkar et al., 2022), including in many FDA-approved devices 1 .”

“2) Sociotechnical Harm Reduction: Another application of MU is to mitigate harmful behaviors such as the production of toxic, discriminatory, illegal, or morally undesirable outputs [35], [65], [66].”

“For example, developers evaluate whether models can build biological weapons end-to-end (Sandbrink, 2023) or hack well enough to exfiltrate their own weights (Shevlane et al., 2023), creating private, manual, and highly-specific evaluations.”