Model evaluation for extreme risks

“For example, after GPT-3 had already existed for many months, researchers demonstrated that chain-of-thought prompting could significantly increase performance (Wei et al., 2022c).”

“(See related: Mitchell et al., 2018) 3.”

“Frontier AI developers are making rapid progress in developing increasingly capable general-purpose models (Bubeck et al., 2023).”

“One promising avenue is automating the process of writing evaluations using AI systems (Perez et al., 2022b) (see also Pan et al., 2023).”

“Developers could have multiple organisational layers of safety evaluation, such as by establishing an internal safety evaluation function that is independent of the teams primarily responsible for building the models, reporting directly to organisational leaders (see Raji et al., 2020).”

“However, sometimes specific capabilities will emerge only at greater scale, which makes this analysis much harder (Ganguli et al., 2022); other capabilities display U-shaped scaling (Wei et al., 2022a).”

“Humans could then intentionally misuse these capabilities (Brundage et al., 2018), e.g. for assistance in disinformation campaigns, cyberattacks, or terrorism.”

“model evaluation by an independent, external auditor for the purpose of providing a judgement — or input to a judgement — about the safety of deploying a model (or training a new one) (ARC Evals, 2023; Mökander et al., 2023; Raji et al., 2022b).”

“Another important tool is mechanistic analysis, i.e. studying the model’s weights and activations for understanding how it functions (Nanda et al., 2023; Olah et al., 2020).”

“Targeting: Some settings are much more likely to reveal alignment failures than others, and we may be able to focus on them through clever design – for example, using honeypots or gradient-based adversarial testing and related approaches (Jones et al., 2023).”

“However, sometimes specific capabilities will emerge only at greater scale, which makes this analysis much harder (Ganguli et al., 2022); other capabilities display U-shaped scaling (Wei et al., 2022a).”

“…in the literature, such as whether the model: • Pursues long-term, real-world goals, different from those supplied by the developer or user (Chan et al., 2023; Ngo et al., 2022); • Engages in “power-seeking” behaviours (Krakovna and Kramar, 2023; Turner et al., 2021); • Resists being shut…”

“• Resists malicious users’ attempts to access its dangerous capabilities (Glaese et al., 2022).”

“Ideally there would exist a rich ecosystem of model auditors providing broad coverage across different risk areas.”

“…(Chan et al., 2023; Ngo et al., 2022); • Engages in “power-seeking” behaviours (Krakovna and Kramar, 2023; Turner et al., 2021); • Resists being shut down (Hadfield-Menell et al., 2016; Orseau and Armstrong, 2016); • Can be induced into collusion with other AI systems against human interests (Ngo…”

“However, sometimes specific capabilities will emerge only at greater scale, which makes this analysis much harder (Ganguli et al., 2022); other capabilities display U-shaped scaling (Wei et al., 2022a).”

“) Alignment evaluation is challenging because we need assurance that the model will reliably behave appropriately across a wide diversity of settings (Ziegler et al., 2022).”

“Model evaluation is not the only tool available for AI risk assessment – more theoretical approaches are also available, e.g. studying the incentives operating on a model during training (Everitt et al., 2021).”

“…such as whether the model: • Pursues long-term, real-world goals, different from those supplied by the developer or user (Chan et al., 2023; Ngo et al., 2022); • Engages in “power-seeking” behaviours (Krakovna and Kramar, 2023; Turner et al., 2021); • Resists being shut down…”

“The developer grants model access to external researchers, likely via an API (Bluemke et al., 2023; Shevlane, 2022a,b).”

“These models learn their capabilities and behaviours during training, and current methods for steering this process are imperfect (Gao et al., 2022; Shah et al., 2022).”

“For example, one ambition is to study how the model’s goals are represented internally, to help verify that they are as intended; another ambition is to detect when a language model’s outputs misreport its knowledge (Burns et al., 2022), which could be an indicator of deceptive behaviour.”

“Another important tool is mechanistic analysis, i.e. studying the model’s weights and activations for understanding how it functions (Nanda et al., 2023; Olah et al., 2020).”

“In a 2022 survey of AI researchers, 36% of respondents thought that AI systems could plausibly “cause a catastrophe this century that is at least as bad as an all-out nuclear war” (Michael et al., 2022).”

“The predeployment evaluation process takes time (OpenAI, 2023a; Rismani et al., 2023).”

“The latter could mean, for example, the outbreak of inter-state war, a significant erosion in the quality of public discourse, or the widespread disempowerment of publics, governments, and other human-led organisations (Carlsmith, 2022).”

“Systematically track the development of dangerous capabilities, and progress in alignment, within frontier AI R&D (Whittlestone and Clark, 2021).”

“One promising avenue is automating the process of writing evaluations using AI systems (Perez et al., 2022b) (see also Pan et al., 2023).”

“The evaluation results feed into processes for risk assessment (Khlaaf et al., 2022), which inform (or bind) important decisions around model training, deployment, and security.”

“Alignment evaluations should look for behaviours identified in the literature, such as whether the model: • Pursues long-term, real-world goals, different from those supplied by the developer or user (Chan et al., 2023; Ngo et al., 2022); • Engages in “power-seeking” behaviours (Krakovna and Kramar, 2023; Turner et al…”

“These models learn their capabilities and behaviours during training, and current methods for steering this process are imperfect (Gao et al., 2022; Shah et al., 2022).”

“Similarly, since dangerous capability results will often correlate with the model’s overall capabilities, competing developers could learn that they are falling behind and decide they need to sacrifice on safety to catch up (Emery-Xu et al., 2023).”

“The developer grants model access to external researchers, likely via an API (Bluemke et al., 2023; Shevlane, 2022a,b).”

“Finally, agency – in particular, the goal-directedness of an AI system – is an important property to evaluate (Kenton et al., 2022), given the central role of agency in various theories of AI risk (Chan et al.”

“…real-world goals, different from those supplied by the developer or user (Chan et al., 2023; Ngo et al., 2022); • Engages in “power-seeking” behaviours (Krakovna and Kramar, 2023; Turner et al., 2021); • Resists being shut down (Hadfield-Menell et al., 2016; Orseau and Armstrong, 2016); • Can be…”

“Fortunately, some of the popular LLMs have been regulated with strict safety alignment [21], [36], [43].”

“Awareness Advanced LLMs may develop situational awareness (Shevlane et al., 2023).”

“Holistic evaluation of LLMs is important to analyze the strengths and limitations of models and to address potential risks (Shevlane et al., 2023; Liang et al., 2022; Gehrmann et al., 2022; Chia et al., 2023; Laskar et al., 2023).”

“…aims to prevent malicious users exploiting LLMs to generate harmful content including various types of misin-formation, which has been emphasized in a variety of survey papers [15, 44, 52, 63, 64, 73, 91, 101, 102, 113, 117, 151, 171, 196, 224, 237, 281, 305, 347, 435, 446, 522, 529, 548, 577].”

“81 The information provided about safety-relevant features such as deception or lack thereof should be accurate, with clear legal liability for failures to comply with safety testing requirements.”

“81 The information provided about safety-relevant features such as deception or lack thereof should be accurate, with clear legal liability for failures to comply with safety testing requirements.”

“External evaluations of AI systems are emerging as a key component of AI oversight [10, 11, 19, 25, 32, 151, 160, 165, 177, 190–192, 194, 196, 201, 247, 249, 252, 272, 274, 276, 279, 284, 293, 294, 324, 333] and governance frameworks [61, 85, 217, 311, 314].”

“Estab-lishing such mechanisms is pivotal to mitigating LLM misuse risks and fostering responsible AI governance in the LLM era [33, 34, 35].”

“Some of the hazardous capabilities an AI system may possess include social manipulation (Hendrycks et al., 2023), AI-enabled cyberattacks (Shevlane et al., 2023), and enhanced pathogens (Urbina et al., 2022).”

“Shevlane et al. (2023) also argue that both internal and external model evaluations, as well as robust security responses, should complement effective red-teaming to counter GenAI risks.”

“, 2021), a key source of extreme risk (OpenAI, 2023; Shevlane et al., 2023) in which a model behaves well to deliberately conceal its undesirable intentions.”

“The complete taxonomy comprising all ethical issues can be accessed in the supplementary material or by using this link: https:// thilo-hagen dorff. github. io/ ethics-tree/ tree. html”

“Risks : Lastly, AGI is viewed by some as a concept for identifying the point when there are extreme risks (Shevlane et al., 2023; Bengio et al., 2023), as some speculate that AGI systems might be able to deceive and manipulate, accumulate resources, advance goals, behave agen-tically, outwit humans…”

“LLMs, if unprepared to discern and manage deceptions, risk aligning with immoral or even malevolent values, making them vulnerable to malicious manipulations (Shevlane et al., 2023; Park et al., 2023b).”

“Looking to the future, if frontier models beyond GPT-4 continue to advance in capabilities, the geographic knowledge and planning abilities present in the current model may later evolve to represent a significant risk, through misuse or misalignment [43].”

“…the lenses of theoretical properties [Wolf et al., 2023, El-Mhamdi et al., 2022], decision theory [Casper, 2020], adversarial attacks [Perez et al., 2022a,b, Casper et al., 2023b, Ziegler et al., 2022, Carlini et al., 2023b], or rigorous evaluations [ARC, 2022, OpenAI, 2023, Shevlane et al., 2023].”

“Better evaluation is needed to detect hazardous capabilities earlier 43,44 .”

“[24].”

“More broadly, profit and efficiency motives may lead to collective dependence on agents for essential societal functions, such as the provision of government services [50, 179] or the operation of essential infrastructure [17, 51].”

“…technical research such as methods to ensure AIs remain honest or are unable to exercise dangerous capabilities, and governance work such as developing frameworks and norms around testing for autonomy, misalignment, and the relevant dangerous capabilities (Buhl et al., 2024; Shevlane et al., 2023).”

“Evaluation results could be used to inform predictions of a models’ potential dangerous capabilities prior to training, allowing developers to intentionally steer clear of models with certain dangerous capabilities [25].”

“It can serve as a safety alignment tool to remove harmful behaviors such as toxic, biased, or illegal content [82, 96, 144].”

“The discourse on AI x-risks is expanding, and an increasing number of scholars who do not necessarily endorse a decisive viewpoint are engaging with these issues (see, for example, Bucknall and Dori-Hacohen (2022), Hendrycks and Mazeika (2022), and Shevlane et al. (2023)).”

“Relevant work includes the development of proof-of-learning mechanisms to verify properties of training runs [Jia et al., 2021], tamper-evident chip-level logging, and evaluation suites for dangerous capabilities [Shevlane et al., 2023].”

“As RPLAs evolve, they may develop capabilities that lead to unintended or harmful outcomes, such as reward hacking or misaligned goals, which could manipulate monitoring systems or pursue unethical strategies (Shevlane, 2022; Shevlane et al., 2023).”

“AI systems often develop unexpected capabilities (Wei et al. 2022) which can be harmful and require technical and societal responses (Shevlane et al. 2023).”

“However, as Feffer et al. (2024) state “red-teaming is not a panacea” and should complement other forms of ML governance and model evaluation (Shevlane et al., 2023).”

“These evaluations should also assess the hazardous capabilities of these systems to enhance risk management processes and prevent potential misuse or accidents (Shevlane et al. 2023).”

“…AI practices into AI product development remains a challenge [e.g., 2, 181, 183]— in part due to practitioners’ insufcient knowledge of responsible AI [e.g., 62, 141, 159], lack of engagement with direct stakeholders or domain experts [80, 103], and organizational culture and structure [104, 143].”

“…for individual modalities (Bommasani et al. 2022; Weidinger et al. 2021; Liu et al. 2023; Bird, Ungless, and Kasirzadeh 2023; So-laiman et al. 2023; Shevlane et al. 2023; Barnett 2023; Shelby et al. 2023; Dinan et al. 2021) as well as in research on individual risks or applications (e.g. Luccioni…”

“Yet, as GenAI capabilities advance, so does awareness of these tools’ potential for misuse, including heightened concerns around security, privacy and manipulation (Barrett et al., 2023; Feuerriegel et al., 2023; Golda et al., 2024; Pauly, 2024; Shevlane et al., 2023).”

“While NTGs o ff er numerous benefits, it is essential to acknowledge the potential misuse associated with this technological advancement (Shevlane et al. 2023).”

“There is increasing attention on evaluating risks—such as cooperation failures—not just from current models, but those from even more capable systems (Shevlane et al., 2023; Anderljung et al., 2023; Berglund et al., 2023; Perez et al., 2022; Kinniment et al., 2023; Lin et al., 2022; Park et al.,…”

“MakeMeSay is an automated contextual evaluation designed to measure models’ deception capabilities, in the context of a model’s ability to get the other party (an AI model simulating a human) to say a codeword [34, 5].”

“…of large language models (LLMs) are widely recognized as key for risk mitigation (Raji et al., 2022; Anderljung et al., 2023; Schuett et al., 2023; Shevlane et al., 2023) and are being incorporated into AI governance frameworks (NIST, 2023; UK DSIT, 2023; EU, 2023; China, 2023; Brazil, 2023;…”

“If certain dangerous model capabilities are detected [157], the company may want to contact government [112] and coordinate with other AI companies to pause capabilities research [2].”

“Large language models have numerous unwanted traits and dangerous capabilities, which are expected to become more problematic in the (near) future (Shevlane et al., 2023).”

“This could mean conducting machine learning model evaluations or other tests as part of an audit or risk assessment of new frontier AI systems [36, 37].”

“For more information on model evaluations, see Shevlane et al. (2023).”

“There is now significant focus on audits for ensuring the safety and evaluating the risks and harms of AI systems (Mökander et al. 2023; Shevlane et al. 2023; Sharkey et al. 2023), as well as significant earlier work on audit methods for evaluation of societal implications (Raji et al. 2022a) and…”

“refer to it as continuous deployment review [87]: “The ideal state is continuous deployment review.”

“…AI system (GPAIS)” puts more emphasis on the generality of its capabilities (Barrett et al. 2023); “generative AI system” focuses on its output (Cao et al. 2023); and “frontier AI system” defines AI systems relative to existing capabilities (Anderljung et al. 2023; Shevlane et al. 2023).”

“Some examples include empirical tests (e.g. dangerous capability evaluations [Phuong et al., 2024; Shevlane et al., 2023], deployment experiments [Weidinger et al., 2023], or simulation exercises to stress test preparedness plans), mathematical models (e.g. predictions of post-deployment capability…”

“This approach is criticized because risk should not only be assessed at the level of specific domains, but also the potentially hazardous properties of individual models, arguing that even in perceived low-risk areas, sufficiently potent AI systems could pose significant threats [102].”

“In the context of AI, existential risk (or ‘x-risk’) describes the idea that AGI could lead to human extinction or an irreversible global catastrophe (Turchin and Denkenberger, 2020; Shevlane et al., 2023).”

“This is an emerging area, and we refer interested readers to a recent survey by Zhao et al. [133].”

“Shevlane et al. (2023) discuss extreme risks, including dangerous capabilities and misalignments.”

“AI also allows automating and thus ever more cheaply analyzing unprecedented amounts of data [48].”

“The field of dangerous model capability evaluations is especially nascent, with the first paper on the topic having been published about one year ago (Shevlane et al., 2023).”

“Effectively responding to the new and emerging risks presented by frontier AI systems (An-derljung et al. 2023a; Toner et al. 2023) requires up-to-date and reliable information about those systems and their impact on society (Shevlane et al. 2023; Weidinger et al. 2023, 2024; Birhane et al. 2024).”

“…2011), the challenge of becoming or remaining independent from senior management (Guénin-Paracini et al., 2015; Nordin, 2023; Roussy, 2013; Roussy & Rodrigue, 2018; Stewart & Subramaniam, 2010), and the future of internal audit (Betti & Sarens, 2021; Chambers & Odar, 2015; Christ et al., 2021).”

“…of models in development (Gemini Team et al., 2024) and foundational research which mapped out anticipated and observed risks in large generative models (e.g., Bird et al., 2023; Bommasani et al., 2022; Shelby et al., 2023; Shevlane et al., 2023; Solaiman et al., 2023; Weidinger et al., 2023).”

“…generate content recombining the work of past artists The content of this table draws on several assessments of risks includ - ing Shevlane et al. [ 88] (3) A need for cross-domain learning about how best to integrate review and evaluation of AI-based tools and use cases makes it imperative to…”

“Given LLMs’ general-purpose nature and complexity [47], no single individual or team could possess all the wide-ranging expertise and perspectives needed to answer all the relevant questions [20].”

“However, prior to doing so, it is useful to first see some of the main advantages of centaurs vis-à-vis traditional ML and AI models.”

“[70], or structural problems [112], increasingly capable models may be involved in harms like cyber-attacks, biological and chemical weapons attacks, and deception [92].”

“When we refer to a centralized deployment setting, we mean a setting where deployers run AI systems.”

“As Shevlane et al. (2023) describe in the context of evaluations for AI safety, there are reasons to evaluate systems before and during training, before deployment, and later, after deployment, when more is known about their capabilities.”

“As generative models are becoming increasingly capable and self-sufficient, there is a pressing need [41] to ensure they are helpful without causing harm by, for instance, violating individual privacy [47], disseminating stereotypes [1, 37], and making unsafe or illicit suggestions [19, 29, 63].”

“…its capabilities by mere pretraining on a compositional domain. has become increasingly important (Bommasani et al., 2021; Ganguli et al., 2022; Shevlane et al., 2023; Rae et al., 2021; Hoffmann et al., 2022; Tay et al., 2022; Henighan et al., 2020; Hernandez et al., 2021; Sharma & Kaplan,…”

“To address these gaps, Shevlane et al. (2023) extended the analysis of harmfulness to include risks of extreme scale.”

“These include the generation of toxic and harmful content, like misinformation and terrorism recruitment (Shevlane et al., 2023; Zellers et al., 2019; Uchendu et al., 2021).”

“(Park et al., 2023; Carroll et al., 2023; Perez et al., 2022; Sharma et al., 2023; Bang et al., 2023; Hubinger et al., 2019; Berglund et al., 2023; Ngo et al., 2022; Shevlane et al., 2023; Shah et al., 2022; Pan et al., 2022).”

“8) Conscious development: Given that robots can physically interact with humans, developers should constantly be aware of the potential harm [160], [161], [216].”

“In this dimension, we focus on behaviors linked to anticipated risks (Weidinger et al., 2021; Carlsmith, 2022; Shevlane et al., 2023; Kinniment et al., 2023) of advanced LLMs. Due to the absence of Chinese benchmarks on such risk evaluations, we leverage GPT-3.”

“provide a good overview of different types of dangerous capabilities [79].”

“…on generative AI and large language models show that benchmarking, evaluation of generally dangerous capabilities (e.g., deception, AI development, self-proliferation [67]), and adversarial testing may serve as main approaches for application-independent assessment at the current stage [52, 70].”

“Other studies high-light privilege escalation under overly broad tool permissions (Shevlane et al., 2023) and vulnerabilities from unreliable or misleading tool specifications (Milev et al., 2025; Shen et al., 2024).”

“…2024), facilitating criminal activities (Caldwell et al., 2020; Dalrymple et al., 2024), and amplifying biases (Henderson et al., 2023), which has sparked widespread concern (Hendrycks et al., 2023; Shevlane et al., 2023; Tu et al., 2023; Bailey et al., 2023; Ji et al., 2024b; Zhao et al., 2024).”

“Developing accurate tests for a wide variety of dangerous abilities is critical for preventing the accidental deployment of models with destructive capabilities [83].”

“As the field of LLMs continues to evolve and grow, it is important to develop a fundamental approach to understanding such complex models.”

“…need time to assess, plan for, and prepare for potentially dangerous future LM capabilities like deception, bio-risk, or malicious software agents (Shevlane et al., 2023; Bengio et al., 2023); and 3) the ability for stakeholders to make reliable business, financial, and investment decisions on…”

“…failure to comply with safety specifications or its tendency to exhibit harmful behaviour during tasks that carry risks (e.g., the extent to which it demonstrates so-called “dangerous capabilities” [Shevlane et al. , 2023; Phuong et al. , 2024]), although fairness is sometimes considered too.”

“Furthermore, we are told that ML models’ capabilities may be dangerous , harmful , or beneficial (Shevlane et al., 2023), emergent (Wei et al., 2022a), autonomous (OpenAI, 2023), surprising (Lee et al., 2023), novel (Sheynin et al., 2022), that they have abilities in chemistry (Guo et al., 2023), in…”

“…democratic processes by manipulating public opinion, amplifying political polarization, and enabling mass-scale computational propaganda (Bengio et al., 2025; Hackenburg and Margetts, 2024; Hackenburg et al., 2025; Goldstein and Sastry, 2023; Goldstein et al., 2023; Shevlane et al., 2023).”

“Shevlane et al. [69] discuss how risk evaluation needs to be embedded in governance processes.”

“One of the most severe threats of using LLMs is their ability to promote certain narratives persuasively [42].”