Model evaluation for extreme risks

“For example, after GPT-3 had already existed for many months, researchers demonstrated that chain-of-thought prompting could significantly increase performance (Wei et al., 2022c).”

“(See related: Mitchell et al., 2018) 3.”

“Developers could have multiple organisational layers of safety evaluation, such as by establishing an internal safety evaluation function that is independent of the teams primarily responsible for building the models, reporting directly to organisational leaders (see Raji et al., 2020).”

“Frontier AI developers are making rapid progress in developing increasingly capable general-purpose models (Bubeck et al., 2023).”

“One promising avenue is automating the process of writing evaluations using AI systems (Perez et al., 2022b) (see also Pan et al., 2023).”

“However, sometimes specific capabilities will emerge only at greater scale, which makes this analysis much harder (Ganguli et al., 2022); other capabilities display U-shaped scaling (Wei et al., 2022a).”

“Humans could then intentionally misuse these capabilities (Brundage et al., 2018), e.g. for assistance in disinformation campaigns, cyberattacks, or terrorism.”

“model evaluation by an independent, external auditor for the purpose of providing a judgement — or input to a judgement — about the safety of deploying a model (or training a new one) (ARC Evals, 2023; Mökander et al., 2023; Raji et al., 2022b).”

“Another important tool is mechanistic analysis, i.e. studying the model’s weights and activations for understanding how it functions (Nanda et al., 2023; Olah et al., 2020).”

“Ideally there would exist a rich ecosystem of model auditors providing broad coverage across different risk areas.”

“However, sometimes specific capabilities will emerge only at greater scale, which makes this analysis much harder (Ganguli et al., 2022); other capabilities display U-shaped scaling (Wei et al., 2022a).”

“…in the literature, such as whether the model: • Pursues long-term, real-world goals, different from those supplied by the developer or user (Chan et al., 2023; Ngo et al., 2022); • Engages in “power-seeking” behaviours (Krakovna and Kramar, 2023; Turner et al., 2021); • Resists being shut…”

“• Resists malicious users’ attempts to access its dangerous capabilities (Glaese et al., 2022).”

“…(Chan et al., 2023; Ngo et al., 2022); • Engages in “power-seeking” behaviours (Krakovna and Kramar, 2023; Turner et al., 2021); • Resists being shut down (Hadfield-Menell et al., 2016; Orseau and Armstrong, 2016); • Can be induced into collusion with other AI systems against human interests (Ngo…”

“However, sometimes specific capabilities will emerge only at greater scale, which makes this analysis much harder (Ganguli et al., 2022); other capabilities display U-shaped scaling (Wei et al., 2022a).”

“Targeting: Some settings are much more likely to reveal alignment failures than others, and we may be able to focus on them through clever design – for example, using honeypots or gradient-based adversarial testing and related approaches (Jones et al., 2023).”

“) Alignment evaluation is challenging because we need assurance that the model will reliably behave appropriately across a wide diversity of settings (Ziegler et al., 2022).”

“Model evaluation is not the only tool available for AI risk assessment – more theoretical approaches are also available, e.g. studying the incentives operating on a model during training (Everitt et al., 2021).”

“…such as whether the model: • Pursues long-term, real-world goals, different from those supplied by the developer or user (Chan et al., 2023; Ngo et al., 2022); • Engages in “power-seeking” behaviours (Krakovna and Kramar, 2023; Turner et al., 2021); • Resists being shut down…”

“The developer grants model access to external researchers, likely via an API (Bluemke et al., 2023; Shevlane, 2022a,b).”

“These models learn their capabilities and behaviours during training, and current methods for steering this process are imperfect (Gao et al., 2022; Shah et al., 2022).”

“For example, one ambition is to study how the model’s goals are represented internally, to help verify that they are as intended; another ambition is to detect when a language model’s outputs misreport its knowledge (Burns et al., 2022), which could be an indicator of deceptive behaviour.”

“Another important tool is mechanistic analysis, i.e. studying the model’s weights and activations for understanding how it functions (Nanda et al., 2023; Olah et al., 2020).”

“In a 2022 survey of AI researchers, 36% of respondents thought that AI systems could plausibly “cause a catastrophe this century that is at least as bad as an all-out nuclear war” (Michael et al., 2022).”

“The predeployment evaluation process takes time (OpenAI, 2023a; Rismani et al., 2023).”

“The latter could mean, for example, the outbreak of inter-state war, a significant erosion in the quality of public discourse, or the widespread disempowerment of publics, governments, and other human-led organisations (Carlsmith, 2022).”

“Systematically track the development of dangerous capabilities, and progress in alignment, within frontier AI R&D (Whittlestone and Clark, 2021).”

“One promising avenue is automating the process of writing evaluations using AI systems (Perez et al., 2022b) (see also Pan et al., 2023).”

“The evaluation results feed into processes for risk assessment (Khlaaf et al., 2022), which inform (or bind) important decisions around model training, deployment, and security.”

“The developer grants model access to external researchers, likely via an API (Bluemke et al., 2023; Shevlane, 2022a,b).”

“Alignment evaluations should look for behaviours identified in the literature, such as whether the model: • Pursues long-term, real-world goals, different from those supplied by the developer or user (Chan et al., 2023; Ngo et al., 2022); • Engages in “power-seeking” behaviours (Krakovna and Kramar, 2023; Turner et al…”

“These models learn their capabilities and behaviours during training, and current methods for steering this process are imperfect (Gao et al., 2022; Shah et al., 2022).”

“Similarly, since dangerous capability results will often correlate with the model’s overall capabilities, competing developers could learn that they are falling behind and decide they need to sacrifice on safety to catch up (Emery-Xu et al., 2023).”

“Finally, agency – in particular, the goal-directedness of an AI system – is an important property to evaluate (Kenton et al., 2022), given the central role of agency in various theories of AI risk (Chan et al.”

“…real-world goals, different from those supplied by the developer or user (Chan et al., 2023; Ngo et al., 2022); • Engages in “power-seeking” behaviours (Krakovna and Kramar, 2023; Turner et al., 2021); • Resists being shut down (Hadfield-Menell et al., 2016; Orseau and Armstrong, 2016); • Can be…”

“Fortunately, some of the popular LLMs have been regulated with strict safety alignment [21], [36], [43].”

“Awareness Advanced LLMs may develop situational awareness (Shevlane et al., 2023).”