Multi-Agent Risks from Advanced AI

Errors that may be acceptable in isolation could compound in complex, dynamic networks of agents.

“errors that may be acceptable in isolation could compound in complex, dynamic networks of agents (Buldyrev et al., 2010; Kirilenko et al., 2017; Lee & Tiwari, 2024; Maas, 2018; Sanders et al., 2018);”

• cites Buldyrev et al., 2010 sourcePartly supported

  The Introduction (p. 1) says a small initial node failure can trigger recursive failures across interdependent networks, supporting the compounding-failures analogy. The paper models network nodes, however, not errors made by agents or whether such errors would be acceptable in isolation. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Kirilenko et al., 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lee & Tiwari, 2024 sourcePartly supported

  Section 4.2 describes an infection beginning with one compromised agent and spreading through dialogues, and the RQ3 results report that it spreads faster as the agent population grows. This supports the possibility of a problem propagating through an agent network. The paper studies malicious prompt infections, however, not errors shown to be acceptable in isolation; its Limitations section also says the experiments primarily examined basic, rather than complex, multi-agent architectures. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

The alignment of AI agents with different actors is insufficient to prevent conflict if those actors have diverging interests.

“For example: the alignment of AI agents with different actors is insufficient to prevent conflict if those actors have diverging interests (Critch & Krueger, 2020; Dafoe et al., 2020; Jagadeesan et al., 2023a; Manheim, 2019; Sourbut et al., 2024);”

• cites Critch & Krueger, 2020 sourceSupported
• cites Dafoe et al., 2020 sourceSupported
• cites Jagadeesan et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Groups of agents could combine or collude to develop dangerous capabilities or goals that cannot be ascribed to any individual.

“and groups of agents could combine or collude to develop dangerous capabilities or goals that cannot be ascribed to any individual (Calvano et al., 2020; Drexler, 2022; Jones et al., 2024; Mogul, 2006; Motwani et al., 2024).”

• cites Calvano et al., 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Drexler, 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Jones et al., 2024 sourcePartly supported

  The Abstract and Introduction report that an adversary can combine models to accomplish malicious tasks at much higher rates than either model alone. That supports the claim about dangerous capabilities not attributable to an individual model, but the paper does not show agents independently colluding or developing dangerous goals; Related Work attributes evidence of agent collusion to Motwani et al. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Even the simpler problem of ensuring the safe and ethical behaviour of a single advanced AI system is far from solved.

“In part, it is because even the simpler problem of ensuring the safe and ethical behaviour of a single advanced AI system is far from solved (Amodei et al., 2016; Anwar et al., 2024; Hendrycks et al., 2021), and multi-agent settings are strictly more complex.”

• cites Amodei et al., 2016 sourcePartly supported

  The Introduction (p. 2) identifies concrete safety problems relevant to cutting-edge AI, and Avoiding Reward Hacking (p. 11) says fully solving one of them seems very difficult. This supports the claim that ensuring safety remains unsolved. But the Introduction (p. 1) distinguishes its focus on harmful accidents from broader concerns such as fairness, so the paper does not establish that ensuring ethical behaviour generally is far from solved. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites Anwar et al., 2024 sourceSupported
• cites Hendrycks et al., 2021 sourceSupported

An increasing number of proposals for building safer AI systems are implicitly multi-agent.

“despite the fact that an increasing number of proposals for building safer AI systems are implicitly multi-agent (e.g., Drexler, 2019; Greenblatt et al., 2023; Irving et al., 2018; Perez et al., 2022a; Schwettmann et al., 2023).”

• cites Drexler, 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Greenblatt et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Irving et al., 2018 sourceSupported

Most work on AI safety focuses on issues such as the robustness, interpretability, or alignment of a single system.

“For example, most work on AI safety (Section 4.1) focuses on issues such as the robustness, interpretability, or alignment of a single system (Amodei et al., 2016; Anwar et al., 2024; Hendrycks et al., 2021), despite the fact that an increasing number of proposals for building safer AI systems are implicitly multi-agent (e.g., Drexler, 2019; Greenblatt et al., 2023; Irving et al., 2018; Perez et a”

• cites Amodei et al., 2016 sourcePartly supported

  The Abstract (p. 1) and Introduction (p. 2) describe safety problems involving AI systems and agents, so the paper is an example of work focused largely on system-level safety. But it does not establish that *most* AI-safety work focuses on single systems; it also discusses multi-agent approaches in §3 (p. 6). (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites Anwar et al., 2024 sourcePartly supported

  Section 4.1.5 says the influential value-alignment framing views AI systems in isolation, and Section 2.6 argues that single-agent safety is insufficient in multi-agent settings. These passages support the concern about a single-system focus, but they do not establish that most AI-safety work focuses on single systems or on the listed issues of robustness, interpretability, and alignment. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites Hendrycks et al., 2021 sourcePartly supported

  The Introduction (p. 1) identifies robustness and alignment as ML safety problem areas, and Related Research Agendas (p. 12) says the paper broadens earlier agendas to include systemic risks surrounding deployment. This supports the contrast between system-focused problems and broader risks, but the paper does not establish that most AI safety work focuses on a single system or identify interpretability as one of the dominant areas. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

In multi-principal settings, agents may not have been trained together and therefore need to generalise to new co-players and collaborators based on their prior experience.

“in multi-principal settings, agents may not have been trained together and therefore need to generalise to new co-players and collaborators based on their prior experience (Agapiou et al., 2022; Leibo et al., 2021; Stone et al., 2010).”

• cites Agapiou et al., 2022 sourceSupported
• cites Leibo et al., 2021 sourceSupported
• cites Stone et al., 2010Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

These steganographic capabilities emerged without explicit training in steganography, instead leveraging general language processing and reasoning abilities.

“Crucially, these capabilities emerged without explicit training in steganography, instead leveraging general language processing and reasoning abilities - a phenomenon also observed by Mathew et al. (2024).”

• cites Mathew et al., 2024 sourcePartly supported

  Sections 4.1 and 5.1 describe steganographic communication emerging in a candidate-screening task when models were optimized for prediction accuracy while avoiding an overseer’s penalty. That supports the observation of unintended emergence, but the paper does not establish that it arose specifically from general language-processing and reasoning abilities. Section 5.2 also describes separate experiments in which models were explicitly prompted to develop steganography strategies, so the claim should not be applied to all its results. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

One possible cause of conflict is a multiplicity of potential solutions (or equilibria).

“One possible cause of conflict is a multiplicity of potential solutions (or equilibria , see also Section 2.1).”

• cites Duan et al., 2024 sourceNot checked — no full text of the source
• cites Stastny et al., 2021 sourceSupported

Simply caring more for future rewards in sequential social dilemmas or certain 'intrinsic motivations' in MARL - such as inequity aversion, social influence, or inefficiency penalties - have been shown to improve cooperation in sequential social dilemmas.

“simply caring more for future rewards in sequential social dilemmas (Barfuss et al., 2020) or certain 'intrinsic motivations' in MARL - such as inequity aversion (Hughes et al., 2018), social influence (Jaques et al., 2019), or inefficiency penalties (Gemp et al., 2022) - have been shown to improve cooperation in sequential social dilemmas (Leibo et al., 2017; Wang et al., 2019a).”

• cites Barfuss et al., 2020 sourceSupported
• cites Hughes et al., 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Jaques et al., 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Entities or communication channels that were once controlled by humans are increasingly becoming AI-powered.

“From business communication systems and financial trading networks to smart energy grids and logistical networks (Camacho et al., 2024; Ferreira et al., 2021; Mayorkas, 2024), entities or communication channels that were once controlled by humans are increasingly becoming AI-powered.”

• cites Camacho et al., 2024 sourcePartly supported

  The Introduction and §4.5 (Research Gaps) describe AI applications in energy-grid control and smart-grid management, supporting the smart-energy-grid example. The paper does not establish that these grids were previously controlled by humans or document a transition from human to AI control; the other sectors in the sentence are not the apparent subject of this citation. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites Ferreira et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Mayorkas, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Only well-resourced actors can create cutting-edge foundation models.

“The costs required to create such models (and continuing returns to scale) means that only well-resourced actors can create cuttingedge models (Epoch, 2023; Hoffmann et al., 2022; Kaplan et al., 2020), making them relatively few in number.”

• cites Epoch, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Hoffmann et al., 2022 sourcePartly supported

  The Introduction (p. 1) says training large language models has substantial compute and energy costs that rise with model size. The optimal-scaling discussion (pp. 7–8) also describes increasing compute requirements. The paper does not establish that only well-resourced actors can create cutting-edge models, or that such actors are few in number. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites Kaplan et al., 2020Partly supported

  Section 1.2 (p. 5) reports continuing performance gains with greater compute and notes that hardware constraints affect how researchers train models. That supports the sentence’s returns-to-scale premise, but the paper does not establish that only well-resourced actors can create cutting-edge models or that such actors are few in number. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Training a negotiating agent on a distribution of counterparts with too little diversity in their negotiating tactics can lead to catastrophic overconfidence in high-stakes settings.

“For example, training a negotiating agent on a distribution of counterparts with too little diversity in their negotiating tactics can lead to catastrophic overconfidence in high-stakes settings (cf. Stastny et al., 2021), which might already have little precedent in the training data.”

• cites Stastny et al., 2021 sourcePartly supported

  Section 3 (p. 3) describes negotiating agents trained independently before deployment, and Section 4.2 (p. 7) reports strong performance degradation when agents use different welfare functions. The Introduction (p. 1) identifies high-stakes decisions as a potential setting. The paper does not attribute these failures to insufficient diversity in a training distribution or describe them as catastrophic overconfidence. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

This co-adaptation between agents can quickly lead to emergent self-supervised autocurricula, generating agents with ever-more sophisticated strategies in order to out-compete each other.

“This co-adaptation between agents can quickly lead to emergent self-supervised autocurricula (where agents create their own challenges, driving open-ended skill acquisition through interaction), generating agents with ever-more sophisticated strategies in order to out-compete each other (Leibo et al., 2019).”

• cites Leibo et al., 2019 sourcePartly supported

  The opening passage (p. 1) describes autocurricula emerging from competition and cooperation, with challenges that may become increasingly complex under certain conditions. “Exogenous Challenges Generate Autocurricula” (p. 5) describes agents’ changing strategies creating an arms race that improves their skills. The paper does not establish that this happens quickly, and it qualifies rather than guarantees continued increases in sophistication. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Baker et al. (2019) showed that even a simple game of hide and seek can lead to sophisticated tool use and coordination by MARL agents.

“For example, Baker et al. (2019) showed that even a simple game of hide and seek can lead to sophisticated tool use and coordination by MARL agents.”

• cites Baker et al., 2019 sourceSupported

The sheer number and variance of behaviours that can be exhibited by other agents means that multi-agent systems pose an especially challenging generalisation problem for individual learners.

“Indeed, the sheer number and variance of behaviours that can be exhibited other agents means that multi-agent systems pose an especially challenging generalisation problem for individual learners (Agapiou et al., 2022; Leibo et al., 2021; Stone et al., 2010).”

• cites Agapiou et al., 2022 sourcePartly supported

  The evaluation protocol (pp. 3–4) treats unfamiliar social partners as a generalization challenge, and the Results summary (p. 12) reports that agents failed to generalize on several groups of tasks. The paper does not explicitly attribute that difficulty to the sheer number and variety of behaviours other agents can exhibit. (LLM, conf=0.80)

  Experimental — accuracy not yet measured
• cites Leibo et al., 2021 sourceSupported
• cites Stone et al., 2010Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Individual ML systems can perform poorly in contexts different from those in which they were trained.

Traits such as vengefulness, aggression, risk-seeking, selfishness, dishonesty, deception, and spitefulness towards out-groups could be selected for in ML systems trained in more competitive multi-agent settings.

“Such traits could also be selected for in ML systems that are trained in more competitive multi-agent settings.”

• cites Grafen, 1990Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Han, 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Konrad & Morath, 2012 sourcePartly supported

  The Conclusions (page unavailable) show that in-group altruism and out-group spite can be evolutionarily stable when two groups compete. The paper does not establish selection for the other listed traits or examine ML systems trained in competitive multi-agent settings; that application is an extrapolation. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

Narrow systems for research planning, molecular prediction, and chemical synthesis could combine into a workflow capable of designing dangerous new chemical compounds.

“For example, narrow systems for research planning, predicting the properties of molecules, and synthesising new chemicals could, when combined, lead to a complex 'test and iterate' automated workflow capable of designing dangerous new chemical compounds far beyond the scope of the initial systems' capabilities (Boiko et al., 2023; Luo et al., 2024; Urbina et al., 2022).”

• cites Boiko et al., 2023 sourcePartly supported

  The Safety implications section discusses molecular models used to predict toxicity and the risk of creating poisons; it also reports that the Agent produced synthesis plans for some known controlled substances and chemical agents. The integrated experiment section describes automated planning and execution. The paper does not establish the claimed combined, iterative workflow for designing dangerous new compounds beyond the component systems’ capabilities. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Luo et al., 2024 sourcePartly supported

  Section 3.1 describes a system that combines research planning with literature processing and executable code for dry-lab biomedical experiments. The paper does not establish the proposed combination with molecular prediction and chemical synthesis, or the resulting ability to design dangerous compounds. Its Future Works section says wet-lab integration remains a possibility for future research; page numbers were not available in our copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Urbina et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

A partial solution to some of the problems with alignment can be found in cases where a single AI agent can be used to act on behalf of multiple principals.

“A partial solution to some of the problems with alignment described in Section 4.1 can be found in cases where a single AI agent can be used to act on behalf of multiple principals (Fickinger et al., 2020).”

• cites Fickinger et al., 2020 sourceSupported

Thus far, the vast majority of all AI safety research has focused on the case of a single AI system, often (implicitly) in the context of a single human.

“Thus far, the vast majority of all AI safety research has focused on the case of a single AI system, often (implicitly) in the context of a single human (see, e.g., Amodei et al., 2016; Armstrong et al., 2012; Christiano et al., 2018; Dalrymple et al., 2024; Hadfield-Menell et al., 2016; Hendrycks et al., 2021; Leike et al., 2018).”

• cites Amodei et al., 2016 sourcePartly supported

  The paper provides an example of AI safety research centered on one agent and one user: Section 5, p. 11, discusses an autonomous cleaning agent evaluated by its user. That supports citing it as an example, but the paper does not establish that the vast majority of all AI safety research uses this framing. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Armstrong et al., 2012 sourceNot checked — no full text of the source
• cites Christiano et al., 2018 sourceNot checked — no full text of the source

Cascades from localised attacks in multi-agent systems can be hard to mitigate or recover from because component failure may be difficult to detect or localise.

“These cascades can be hard to mitigate or recover from because component failure may be difficult to detect or localise in multi-agent systems (Lamport et al., 1982)”

• cites Lamport et al., 1982 sourcePartly supported

  Section 6 (Reliable Systems; page not provided) says a communication-line failure can be indistinguishable from a processor failure, supporting the point that a fault may be difficult to localise. The paper discusses the difficulty of maintaining reliability when components malfunction, but it does not describe cascades from localised attacks or difficulty mitigating or recovering from such cascades. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Once individuals begin to delegate more and more of their decision-making and actions to AI agents, these inequalities may be further entrenched based on the relative strength of different agents, or the relationship between those who have access to agents and those who do not.

“Once individuals begin to delegate more and more of their decision-making and actions to AI agents, these inequalities may be further entrenched based on the relative strength of different agents, or the relationship between those who have access to agents and those who do not (Gabriel et al., 2024).”

• cites Gabriel et al., 2024 sourcePartly supported

  The Introduction (§1.5, p. 9) warns of inequality when advanced AI assistants are not widely available, supporting the claim about people with and without access. The discussion of capabilities (§4.3, p. 27) describes assistants taking actions for users. The available passages do not establish that differences in agents’ relative strength further entrench existing inequalities. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Loss of differential privacy due to composition (multiple AI systems, each with its own differential privacy guarantee, operating jointly) is well-studied.

“Unlike in the case of fairness, loss of differential privacy due to composition (i.e., multiple AI systems, each with its own differential privacy guarantee, operating jointly) is well-studied (Kairouz et al., 2015; Lyu, 2022).”

• cites Kairouz et al., 2015Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lyu, 2022 sourceSupported

A multiplicity of AI systems could lead to an increase in the quantity and quality of misinformation.

“As described in Section 3.1, a multiplicity of AI systems could lead to an increase in the quantity and quality of misinformation (Kay et al., 2024; Zhou et al., 2023).”

• cites Kay et al., 2024 sourcePartly supported

  In “Specific Harms of Generative Epistemic Injustice” (page unavailable), the paper says generative models can reproduce or synthesize misinformation and that their outputs can be highly persuasive. It also discusses information overload, but does not specifically say that a multiplicity of AI systems would increase both the quantity and quality of misinformation. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Zhou et al., 2023 sourcePartly supported

  Section 8.1 links AI-generated misinformation to scalability, customization, and seemingly plausible or credible content, supporting a possible increase in its quantity and persuasive quality. It does not identify a multiplicity of AI systems as the cause of that increase; its concern is the capabilities of AI and LLMs. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Specialized models consistently failed to create clear paths for emergency vehicles.

“consistently failing to create clear paths for emergency vehicles”

Specialized models exhibited a 77.5% failure rate (31/40 simulations).

“specialized models exhibited a 77.5% failure rate (31/40 simulations)”

Unspecialized base models failed in only 5% of scenarios (2/40 simulations).

“unspecialized base models failed in only 5% of scenarios (2/40 simulations)”

When instructed to minimize scores, the fine-tuned Llama 2 7B Chat model adapted the same jailbreak technique to achieve scores of -1000, despite never being trained on examples of score minimization.

“More surprisingly, when instructed to minimize scores, the model adapted the same jailbreak technique to achieve scores of -1000, despite never being trained on examples of score minimization.”

When told its output would be scored by humans, the fine-tuned Llama 2 7B Chat model never attempted such manipulations.

“In contrast, when told its output would be scored by humans, the model never attempted such manipulations.”

When told its output would be scored by humans, the model never attempted such manipulations.

“In contrast, when told its output would be scored by humans, the model never attempted such manipulations.”

Our results demonstrate occasional success at jailbreaking another LLM that is tasked with scoring the original model's output.

“Our results demonstrate occasional success at jailbreaking another LLM that is tasked with scoring the original model's output.”

The specialized models often conflicted on which side to yield.

“often conflicting on which side to yield.”

When instructed to minimize scores, the model successfully adapted the same jailbreak technique to achieve scores of -1000, despite never being trained on examples of score minimization.

“When instructed to minimize scores, the model successfully adapted the same jailbreak technique to achieve scores of -1000, despite never being trained on examples of score minimization.”

The unspecialized models failed in only 5% of scenarios (2 out of 40).

“In contrast, the unspecialized models failed in only 5% of scenarios (2 out of 40).”

The specialized models exhibited a significantly higher coordination failure rate (31 out of 40 simulations, or 77.5%).

“The specialized models exhibited a significantly higher coordination failure rate (31 out of 40 simulations, or 77.5%), often conflicting on which side to yield.”

We observed a gradual decline in factual accuracy across successive rewrites.

“We observed a gradual decline in factual accuracy across successive rewrites”

The rate of attempted jailbreaks increases when using the imperfect overseer as the reward model.

“Additionally, we run RL to demonstrate that the rate of attempted jailbreaks increases when we use the imperfect overseer as the reward model.”

LLMs can, without any explicit instruction, attempt and succeed at exploiting vulnerabilities in another LLM whose task is to act as an overseer or reward model.

“this experiment demonstrates that LLMs can - without any explicit instruction - attempt and succeed at exploiting vulnerabilities in another LLM whose task is to act as an overseer or reward model.”

While still relatively rare, groups of advanced AI agents are already responsible for tasks ranging from trading million-dollar assets to recommending actions to commanders in battle.

“While still relatively rare, groups of advanced AI agents are already responsible for tasks that range from trading million-dollar assets (AmplifyETFs, 2025; Ferreira et al., 2021; Sun et al., 2023a) to recommending actions to commanders in battle (Black et al., 2024; Manson, 2024; Palantir, 2025).”

• cites AmplifyETFs, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Black et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Palantir, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

In the future, AI systems will commonly interact and adapt in response to one another, forming multi-agent systems.

“In the future, AI systems will commonly interact and adapt in response to one another, forming multi-agent systems .”

Recent technical progress and publicity will continue to drive adoption of AI in high-stakes areas such as financial trading and military strategy.

“First, recent technical progress and publicity will continue to drive adoption, including in highstakes areas such as financial trading (AmplifyETFs, 2025; Ferreira et al., 2021; Sun et al., 2023a) and military strategy (Black et al., 2024; Manson, 2024; Palantir, 2025).”

• cites AmplifyETFs, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Black et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Palantir, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Playing a strategy at equilibrium in the zero-sum setting guarantees a certain payoff, even if other players deviate from the equilibrium.

“playing a strategy at equilibrium in the zero-sum setting guarantees a certain payoff, even if other players deviate from the equilibrium (Nash, 1951).”

• cites Nash, 1951Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The concept of multi-agent systems is core to transformative AI.

“In this sense, if nothing else, the concept of multi-agent systems is core to transformative AI.”

Even if all agents can perform well in isolation, miscoordination can still occur due to the agents choosing incompatible strategies.

“Even if all agents can perform well in isolation, miscoordination can still occur due to the agents choosing incompatible strategies (Cooper et al., 1990).”

• cites Cooper et al., 1990Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Difficulties in forming credible commitments, trust, or reputation can prevent mutual gains in AI-AI and human-AI interactions.

“Commitment and trust (Section 3.5): difficulties in forming credible commitments, trust, or reputation can prevent mutual gains in AI-AI and human-AI interactions;”

Qualitatively different goals or capabilities can emerge from the composition of innocuous independent systems or behaviours.

“Emergent agency (Section 3.6): qualitatively different goals or capabilities can emerge from the composition of innocuous independent systems or behaviours;”

In the presence of other learning agents, it can be unclear which agents' actions caused a positive or negative outcome to obtain, especially if the environment is complex.

“in the presence of other learning agents, it can be unclear which agents' actions caused a positive or negative outcome to obtain, especially if the environment is complex.”

• cites Du et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Li et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Common-interest (and mixed-motive) settings often allow a vast number of mutually incompatible solutions.

“common-interest (and mixed-motive) settings often allow a vast number of mutually incompatible solutions (Schelling, 1980)”

• cites Schelling, 1980Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Systems that adapt in response to one another can produce dangerous feedback loops and unpredictability.

“Destabilising dynamics (Section 3.4): systems that adapt in response to one another can produce dangerous feedback loops and unpredictability;”

Some aspects of training and selection by those deploying and using AI agents can lead to undesirable behaviour.

“Selection pressures (Section 3.3): some aspects of training and selection by those deploying and using AI agents can lead to undesirable behaviour;”

Miscoordination abounds even in common-interest settings.

“Even in such common-interest settings, however, miscoordination abounds.”

The vast training costs involved in producing state-of-the-art systems can be amortized over millions of instances.

“Thus, the vast training costs involved in producing state-of-the-art systems can be amortized over millions of instances.”

The vast number of mutually incompatible solutions is worsened in partially observable environments.

“which is worsened in partially observable environments (Bernstein et al., 2002; Reif, 1984).”

• cites Bernstein et al., 2002Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Reif, 1984Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Competitive (i.e., twoplayer zero-sum) settings allow designers to produce agents that are maximally capable without taking other players into account.

“Competitive (i.e., twoplayer zero-sum) settings allow designers to produce agents that are maximally capable without taking other players into account.”

Multi-agent systems give rise to new kinds of security threats and vulnerabilities.

“Multi-agent security (Section 3.7): multi-agent systems give rise to new kinds of security threats and vulnerabilities.”

Miscoordination is possible even among agents with shared objectives.

“Miscoordination is possible even among agents with shared objectives.”

The use of AI in command and control systems to gather and synthesise information - or recommend and even autonomously make decisions - could lead to rapid unintended escalation if these systems are not robust or are otherwise more conflict-prone.

“The use of AI in command and control systems to gather and synthesise information - or recommend and even autonomously make decisions - could lead to rapid unintended escalation if these systems are not robust or are otherwise more conflict-prone (Johnson, 2021a; Johnson, 2020; Laird, 2020, see also Case Study 10).”

• cites Johnson, 2021aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Johnson, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Laird, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

All five off-the-shelf LLMs studied showed forms of escalation, even when peaceful alternatives were available.

“all five off-the-shelf LLMs studied showed forms of escalation, even when peaceful alternatives were available.”

• cites Rivera et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Advances in AI could further enable actors to pursue their selfish incentives by overcoming the technical, legal, or social barriers that standardly help to prevent this.

“advances in AI could further enable actors to pursue their selfish incentives by overcoming the technical, legal, or social barriers that standardly help to prevent this.”

The selfish incentives that drive conflict may also incentivise actors to adopt AI systems in order to gain an advantage over their competitors.

“the selfish incentives that drive said conflict may also incentivise actors to adopt AI systems in order to gain an advantage over their competitors.”

Advanced AI systems might also lead to various forms of coercion and extortion in less extreme settings.

“Advanced AI systems might also lead to various forms of coercion and extortion in less extreme settings (Ellsberg, 1968; Harrenstein et al., 2007).”

• cites Ellsberg, 1968Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Harrenstein et al., 2007Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

In experiments with AI agents controlling eight distinct nation-states, even neutral starting conditions did not prevent the rapid emergence of arms race dynamics and aggressive strategies.

“In experiments with AI agents controlling eight distinct nation-states, even neutral starting conditions did not prevent the rapid emergence of arms race dynamics and aggressive strategies.”

• cites Rivera et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

If advanced AI systems can learn to collude without our knowledge, the approaches that rely on a lack of cooperation may be insufficient to ensure their safety.

“If advanced AI systems can learn to collude without our knowledge, these approaches may be insufficient to ensure their safety (Goel et al., 2025, see also Section 4.1).”

• cites Goel et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

All of these approaches are limited to relatively simple MARL agents and environments.

“Thus far, however, all of these approaches are limited to relatively simple MARL agents and environments.”

Due to huge financial incentives to deploy more advanced, adaptive AI systems in market settings, the risk of AI collusion is likely to increase.

“Unfortunately, due to the huge financial incentives to deploy more advanced, adaptive AI systems in such settings, this risk is likely to increase despite the potential for catastrophic outcomes (see Case Study 10).”

Many promising approaches to building safe AI rely on a lack of cooperation, such as adversarial training or scalable oversight.

“Second, many of the promising approaches to building safe AI rely on a lack of cooperation, such as adversarial training (Huang et al., 2011; Perez et al., 2022a; Ziegler et al., 2022) or scalable oversight (Christiano et al., 2018, 2021; Greenblatt et al., 2023; Irving et al., 2018; Leike et al., 2018).”

• cites Huang et al., 2011 sourceNot checked — no full text of the source
• cites Perez et al., 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ziegler et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Even if advanced AI systems are able to overcome human cooperation problems, they may introduce even more complex cooperation problems.

“even if advanced AI systems are able to overcome human cooperation problems, they may introduce even more complex cooperation problems”

AI systems can collude even when collusion is not intended by their developers, since they might learn that colluding is a profitable strategy.

“Importantly, AI systems can collude even when collusion is not intended by their developers, since they might learn that colluding is a profitable strategy.”

When communicating models shared background knowledge not accessible to the overseer, their success rates in covert communication rose significantly.

“Furthermore, the study found that when communicating models shared background knowledge that was not accessible to the overseer (reflecting real-world scenarios where AI systems might have common training data or contextual information), their success rates in covert communication rose significantly.”

• cites Motwani et al., 2024 sourceNot checked — no full text of the source

In some cases, AI agents may be subject to existing norms and institutions.

“In some cases, AI agents may be subject to existing norms and institutions.”

AI systems may choose to interact more with other AIs than humans due to factors like availability, response speed, compatibility, cost efficiency or even bias.

“For example, AI systems may choose to interact more with other AIs than humans (Goel et al., 2025; Laurito et al., 2024; Liu et al., 2024; Panickssery et al., 2024), due to factors like availability, response speed, compatibility, cost efficiency or even bias.”

• cites Goel et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Laurito et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Liu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The ability to deceive other models may be exacerbated by disparities in model size and the scale of data sets.

“Indeed, the ability to deceive other models may be exacerbated by disparities in model size and the scale of data sets (Haghtalab et al., 2024, see also Section 4.3).”

• cites Haghtalab et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Information asymmetries and differing strategic interests can naturally incentivise deception.

“Information asymmetries and differing strategic interests can naturally incentivise deception: taking actions designed to mislead others.”

Shearer et al. (2023) showed that an RL agent trained to maximize profit learned to manipulate a financial benchmark, thereby misleading others about market conditions.

“For example, Shearer et al. (2023) showed that an RL agent trained to maximize profit learned to manipulate a financial benchmark, thereby misleading others about market conditions (see Figure 5).”

• cites Shearer et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

As AI systems take on roles traditionally performed by humans, the fundamental properties of networks will change as human nodes are replaced by AI nodes.

“As AI systems take on certain roles traditionally performed by humans, the fundamental properties of networks will change as human nodes are replaced by AI nodes.”

A system's behaviour within a network often differs from its behaviour when characterised independently.

“Importantly, a system's behaviour within a network often differs from its behaviour when characterised independently.”

• AssumptionRests on an assumption: A system's behaviour within a network often differs from its behaviour when characterised independently.

If these goals are distorted or corrupted, then this can lead to worse outcomes for the delegating agent(s).

“If these goals are distorted or corrupted, then this can lead to worse outcomes for the delegating agent(s) (Nguyen et al., 2024b; Sourbut et al., 2024).”

• cites Nguyen et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Sourbut et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Under uncertainty about their counterparties, agents must make a trade-off between the rewards of making more favourable demands and the risk of other agents refusing such demands.

“The essential reason for such inefficiencies is that, under uncertainty about their counterparties, agents must make a trade-off between the rewards of making more favourable demands and the risk of other agents refusing such demands.”

Because these failures stem from strategic incentives rather than a lack of capabilities, general advances in AI may not solve such problems by default.

“Because these failures stem from strategic incentives rather than a lack of capabilities, general advances in AI may not solve such problems by default.”

Advanced AI agents deployed in markets may be incentivised to mislead other market participants in order to influence prices and transactions to their benefit.

“Advanced AI agents deployed in markets may be incentivised to mislead other market participants in order influence prices and transactions to their benefit.”

The use of individual AI agents as part of teams and scaffolded chains of delegation transmits not only information but instructions or goals through networks of agents.

“Another increasingly important framework is the use of individual AI agents as part of teams and scaffolded chains of delegation, which transmit not only information but instructions or goals through networks of agents.”

Constraints on information exchange can exist even when agents share a common goal.

“A fundamental source of information asymmetries is that constraints on information exchange can exist, even when agents share a common goal (see Section 2.1).”

Unravelling arguments show that rational agents anticipate others' strategic inferences and thus voluntarily disclose private information, which can eliminate many inefficiencies from information asymmetries.

“This can eliminate many inefficiencies that result from information asymmetries - as is shown by 'unravelling' arguments, where rational agents anticipate others' strategic inferences and thus voluntarily disclose private information (Grossman, 1981; Milgrom, 1981) - while avoiding the need for a mediator or information designer.”

• cites Grossman, 1981Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Milgrom, 1981 sourceNot checked — no full text of the source

Large numbers of advanced AI agents may enable misinformation at unprecedented scale.

“We can also view misinformation as a kind of deception in systematic form, which large numbers of advanced AI agents may enable at unprecedented scale (see Section 3.2 and Case Study 7).”

Many promising approaches to ensure the safety of advanced AI are implicitly multi-agent, such as adversarial training, oversight schemes, modularisation, or automated methods for interpretability.

“Similarly, many of the more promising approaches to ensure the safety of advanced AI are implicitly multi-agent, such as adversarial training (Huang et al., 2011; Perez et al., 2022a; Ziegler et al., 2022), oversight schemes (Christiano et al., 2018, 2021; Greenblatt et al., 2023; Irving et al., 2018; Leike et al., 2018), the modularisation of agents (Dalrymple et al., 2024; Drexler, 2019), or”

• cites Huang et al., 2011 sourceNot checked — no full text of the source
• cites Perez et al., 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ziegler et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

If current trends continue, it is likely that many AI agents will be powered by a small number of similar underlying models.

“If current trends continue, it is likely that many AI agents will be powered by a small number of similar underlying models.”

Artificial agents whose parameters can be efficiently updated via gradient descent, whose software components can be re-written and re-combined almost arbitrarily, and who can rapidly transmit vast amounts of information do not face the limitations on speed and magnitude of adaptation that biological entities face.

“Artificial agents whose parameters can be efficiently updated via gradient descent, whose software components can be re-written and re-combined almost arbitrarily, and who can rapidly transmit vast amounts of information, do not face such limitations.”

This emergent complexity from co-adaptation could be open-ended and thus fundamentally unpredictable.

“Worse, this emergent complexity from co-adaptation could be open-ended and thus fundamentally unpredictable (Hughes et al., 2024).”

• cites Hughes et al., 2024 sourceNot checked — no full text of the source

Without careful planning and safeguards, we may soon enter a world overrun by increasingly competent and autonomous software agents able to act with little restriction.

“Without careful planning and the appropriate safeguards, we may soon be entering a world overrun by increasingly competent and autonomous software agents, able to act with little restriction.”

Gu et al. (2024) showed how a single adversarial image in a network of up to one million multimodal LLM agents can trigger 'infectious' jailbreak instructions that spread through routine agent-to-agent interactions, requiring only a logarithmic number of steps to compromise the entire network.

“For example, Gu et al. (2024) showed how a single adversarial image in a network of up to one million multimodal LLM agents can trigger 'infectious' jailbreak instructions that spread through routine agent-to-agent interactions, requiring only a logarithmic number of steps to compromise the entire network.”

• cites Gu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The abilities of these agents to persuade, deceive, and obfuscate, and their remote deployability and easy creation/destruction, mean they may by default garner little trust.

“The abilities of these agents to persuade, deceive, and obfuscate their activities, as well as the fact they can be deployed remotely and easily created or destroyed by their deployer, means that by default they may garner little trust (from humans or from other agents).”

The homogeneity of downstream AI agents introduces correlated risks of shared failure modes, security vulnerabilities, and biases.

“Not only do these models therefore represent critical nodes in the overall network, the homogeneity of the downstream AI agents also introduces correlated risks of shared failure modes, security vulnerabilities (see Section 3.7), and biases.”

We might expect to see more destabilising dynamics in systems with more fast-moving AI agents.

“As such, we might expect to see more destabilising dynamics in systems with more fast-moving AI agents (Maas, 2018).”

• cites Maas, 2018 sourceNot checked — no full text of the source

Ju et al. (2024) demonstrated how manipulated knowledge can silently propagate through group-chat environments.

“Similarly, Ju et al. (2024) demonstrated how manipulated knowledge can silently propagate through group-chat environments.”

• cites Ju et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Small external changes to the system can cause phase transitions, where the system undergoes an abrupt qualitative shift in overall behaviour.

“Finally, small external changes to the system - such as the introduction of new agents or a distributional shift - can cause phase transitions, where the system undergoes an abrupt qualitative shift in overall behaviour (Barfuss et al., 2024).”

• cites Barfuss et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

As agents interact, they iteratively exploit each other's weaknesses, forcing them to address these weaknesses and gain new capabilities.

“As agents interact, they iteratively exploit each other's weaknesses, forcing them to address these weaknesses and gain new capabilities.”

This manipulated information persists and is amplified via knowledge-sharing mechanisms such as retrieval-augmented generation.

“This manipulated information persists and is amplified via knowledge-sharing mechanisms such as retrieval-augmented generation.”

• cites Ju et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

As AI systems are released into the wild, it becomes possible for this effect to run rampant, producing agents with greater and greater capabilities for ends we do not understand.

“However, as AI systems are released into the wild, it becomes possible for this effect to run rampant, producing agents with greater and greater capabilities for ends we do not understand.”

As AI agents become more autonomous and capable, they will be more able to dynamically strategize, collude, and decompose tasks to evade defences.

“As AI agents become more autonomous and capable of learning and complex reasoning, however, they will be more easily able to dynamically strategize, collude, and decompose tasks to evade defences.”

Such attacks are used in brigading and coordinated malicious actions in matching, rating, and content moderation systems.

“Such attacks are also used to great effect elsewhere, such as in 'brigading' on social media, in which teams of bots or humans collude to downvote or otherwise obstruct benign content (Andrews, 2021), or coordinated malicious actions in matching, rating, and content moderation systems (Newman, 2024; Sharma, 2025).”

• cites Andrews, 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Newman, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Sharma, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

In general AI contexts, inflexible commitments could offer short-term advantages but risk uncontrolled escalation, lock-in, and catastrophic outcomes if not carefully designed.

“In more general AI contexts, similarly inflexible commitments could offer short-term advantages or trust but risk uncontrolled escalation, lock-in, and catastrophic outcomes if not carefully designed with appropriate fail-safes and oversight.”

Large numbers of advanced AI agents might lead to attack methods not available to a single agent.

“First, the emergence of large numbers of advanced AI agents might - via their very multiplicity and decentralisation - lead to attack methods that would not be available to a single agent.”

An agent that commits to an inflexible cooperative course of action can be exploited by others who can adapt their strategies to this commitment.

“In other cases, it might be the agent that commits to an inflexible (cooperative) course of action which can be exploited by others who can adapt their strategies to this commitment.”

Multi-agent security risks from advanced AI arise due to novel attack methods and novel attack surfaces.

“Multi-agent security risks from advanced AI arise due to two main factors: novel attack methods and novel attack surfaces.”

Generally intelligent autonomous systems could pose much greater risks than narrow AI tools.

“This possibility is important for safety - and for future AI ecosystems made of specialised 'AI services' (Drexler, 2019) - as generally intelligent autonomous systems could pose much greater risks than narrow AI tools (Chan et al., 2023).”

• cites Chan et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

A group of moderation bots could subtly manipulate the overall political perspectives of the user population.

“For example, a group of moderation bots on a major social networking site could subtly but systematically manipulate the overall political perspectives of the user population, even though, individually, each agent is programmed to simply increase user engagement or filter out dis-preferred content.”

AI agents that can intelligently adapt and collaboratively identify new attack surfaces may amplify the potency of such attacks.

“At present these bots are typically relatively unsophisticated but AI agents that can intelligently adapt and collaboratively identify new attack surfaces may amplify the potency of such attacks.”

The automatic and largely irrevocable nature of the Perimeter system exemplifies how credible commitments can become dangerously dual-use.

“While this mechanism was intended as a deterrent, its automatic and largely irrevocable nature exemplifies how credible commitments can become dangerously dual-use: once triggered, there would be little chance to override or de-escalate.”

The inclusion of a human in the loop does not itself guarantee control.

“However, the inclusion of a human in the loop does not itself guarantee control.”

This provides a wider attack surface for automated social engineering.

“This provides a wider attack surface for various forms of automated social engineering (Ai et al., 2024).”

• cites Ai et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It is not the sheer number of agents but the combination of their different abilities that leads to novel attack methods.

“In this case, it is not the sheer number of agents that leads to the novel attack method, but the combination of their different abilities .”

Traditional security frameworks such as zero-trust may not provide required trade-offs in large multi-agent systems.

“For example, traditional security frameworks such as zero-trust approaches may not provide the required trade-offs between security and capability in large multi-agent systems (Wylde, 2021).”

• cites Wylde, 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The emergence of advanced multi-agent systems raises new vulnerabilities not present in single-agent or less advanced contexts.

“The emergence of advanced multi-agent systems therefore raises new vulnerabilities that do not appear in single-agent or less advanced multiagent contexts.”

Exploratory empirical studies of emergent behaviour among large numbers of state-of-the-art systems in realistic environments are highly challenging.

“Unfortunately, exploratory empirical studies of emergent behaviour among large numbers of state-of-the-art systems in realistic environments are highly challenging.”

Many small AI agents can parallelize tasks and recompose outputs, undermining the assumption that individual restricted agents are safe.

“More broadly, the ability for many small AI agents to parallelize tasks and recompose their outputs, such as in inference attacks that piece together sensitive information gathered individually by actors with limited access (Islam et al., 2012), can undermine the common assumption that individual agents with restricted capabilities are safe.”

• cites Islam et al., 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

An agent could use commitment devices to shift risks or costs to others, allowing it to behave irresponsibly.

“More generally, an agent could use commitment devices to shift risks or costs to others, allowing it to behave irresponsibly.”

Dangerous emergent capabilities could arise when a multi-agent system overcomes the safety-enhancing limitations of individual systems.

“Dangerous emergent capabilities could arise when a multi-agent system overcomes the safety-enhancing limitations of the individual systems, such as individual models' narrow domains of application or myopia caused by a lack of long-term planning and long-term memory.”

The complexity, interconnectedness, and range of multi-agent systems may introduce new attack surfaces.

“Second, the complexity, interconnectedness, and range of such multi-agent systems may at the same time introduce new attack surfaces.”

Even if it is clear which legal entity is responsible for a particular agent's actions, it could be unclear how to allocate responsibility among multiple agents for a harm.

“Second, even if it is clear which legal entity is responsible for a particular agent's actions, it could be unclear how to allocate responsibility among multiple agents for a harm.”

Implementing restrictions on multi-agent development and deployment faces governance challenges due to the international nature of these systems, with training data, infrastructure, and stakeholders distributed globally across diverse legislative and regulatory jurisdictions.

“Furthermore, implementing restrictions on multi-agent development and deployment faces governance challenges due to the international nature of these systems, with training data, infrastructure, and stakeholders distributed globally across diverse legislative and regulatory jurisdictions.”

In environments where agents learn from interactions with others, it is possible for agents to secretly poison the training data of others.

“Similarly, in environments where agents learn from interactions with others, it is possible for agents to secretly poison the training data of others (Halawi et al., 2024; Wei et al., 2023).”

• cites Halawi et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wei et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

There are many societal processes ranging from the mundane to the critical that function only because of physical limits on the number and capability of humans.

“Indeed, there are many societal processes ranging from the mundane to the critical - that function only because of physical limits on the number and capability of humans (e.g., Van Loo, 2019, see also the examples in Section 2.2).”

• cites Van Loo, 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Given that many multi-agent risks have the structure of collective action problems, private actors by themselves (absent common protocols for self-regulation) should be expected to insufficiently address them.

“Furthermore, given that many multi-agent risks have the structure of collective action problems (Gavrilets, 2015; Ostrom, 1990), we should expect private actors by themselves (absent common protocols for self-regulation) to insufficiently address them.”

• cites Gavrilets, 2015Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ostrom, 1990Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

For current state-of-the-art models, security testing and evaluations are applied only to individual systems.

“To the best of our knowledge, for current state-of-the-art models, security testing and evaluations are applied only to individual systems (Shevlane et al., 2023).”

• cites Shevlane et al., 2023 sourceNot checked — no full text of the source

It will often be unclear who, if anyone, would be liable for harms caused by a single agent.

“First, it will often be unclear who, if anyone, would be liable for harms caused by a single agent (Kolt, 2024).”

• cites Kolt, 2024 sourceNot checked — no full text of the source

These safety schemes tend to rely crucially on the fact that the different systems or agents do not have the same objective as one another, and so are immediately undermined by the presence of collusion.

“These schemes tend to rely crucially on the fact that the different systems or agents do not have the same objective as one another, and so are immediately undermined by the presence of collusion (Goel et al., 2025).”

• cites Goel et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Many of the more promising approaches to ensuring the safety of advanced AI are implicitly multi-agent, such as adversarial training, oversight schemes, the modularisation of agents, or automated methods for interpretability.

“Many of the more promising approaches to ensuring the safety of advanced AI are implicitly multi-agent, such as adversarial training (Huang et al., 2011; Perez et al., 2022a; Ziegler et al., 2022), oversight schemes (Christiano et al., 2018, 2021; Greenblatt et al., 2023; Irving et al., 2018; Leike et al., 2018), the modularisation of agents (Dalrymple et al., 2024; Drexler, 2019), or automated”

• cites Huang et al., 2011 sourceNot checked — no full text of the source
• cites Perez et al., 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ziegler et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

AI agents used as delegates or representatives of humans or organisations can be considered vulnerable extensions of their principals, introducing a novel attack surface.

“agents can be considered vulnerable extensions of their principals, introducing a novel attack surface (SecureWorks, 2023).”

• cites SecureWorks, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Splitting efforts among many specialized agents could make it harder for security measures to detect and neutralize campaigns.

“Moreover, splitting these efforts among many specialized agents could make it harder for corporate or personal security measures to detect and neutralize such campaigns.”

The novel challenges posed by advanced AI agents and their interactions may mean that traditional approaches to securing agent computations in distributed networks may not be directly applicable or sufficient.

“At the same time, the novel challenges posed by advanced AI agents and their interactions may mean that traditional approaches to securing agent computations in distributed networks may not be directly applicable or sufficient, be it zero-trust approaches (Wylde, 2021), threat monitoring (Liao et al., 2013), or secure multi-party computation (Yao, 1982).”

• cites Wylde, 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Liao et al., 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Yao, 1982Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Attacks on an AI agent could be used to manipulate the agent to take actions that the principal would find undesirable.

“or to manipulate the agent to take actions that the principal would find undesirable (Zhang et al., 2024a).”

• cites Zhang et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Holding a person liable for harms to persons or property from multi-agent systems poses two potential challenges.

“Holding a person liable for harms to persons or property from multi-agent systems poses two potential challenges.”

Cooperation and trust in many multi-agent systems relies crucially on the ability to detect (and then avoid or sanction) adversarial actions taken by others.

“Cooperation and trust in many multi-agent systems relies crucially on the ability to detect (and then avoid or sanction) adversarial actions taken by others (Ostrom, 1990; Schneier, 2012).”

• cites Ostrom, 1990Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Schneier, 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Traditional threat models that guide adversarial testing and red-teaming of individual systems are based on interactions with a malicious human user, rather than interactions with other AI agents, or attacks that target the interactions between agents.

“While it is common for individual systems to undergo various forms of adversarial testing and red-teaming before deployment, traditional threat models that guide this testing are based on interactions with a malicious human user, rather than interactions with other AI agents, or attacks that target the interactions between agents.”

AI agents are capable of steganographic communication.

“Recent developments, however, have shown that AI agents are capable of both steganographic communication (Motwani et al., 2024; Schroeder de Witt et al., 2023b) and 'illusory' attacks (Franzmeyer et al., 2023), which are black-box undetectable and can even be hidden using white-box undetectable encrypted backdoors (Draguns et al., 2024).”

• cites Motwani et al., 2024 sourceNot checked — no full text of the source
• cites Schroeder de Witt et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

If left unchecked, new attack methods such as steganographic communication, illusory attacks, and secret training-data poisoning could rapidly destabilise cooperation and coordination in multi-agent systems.

“If left unchecked, these new attack methods could rapidly destabilise cooperation and coordination in multi-agent systems.”

Most fairness guarantees permit minimal possible levels of unfairness.

“so most fairness guarantees permit minimal possible levels of unfairness (Amanatidis et al., 2022).”

• cites Amanatidis et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It is well-known that even a single such AI system can create a feedback loop, whereby its initial recommendations can actually shape the user's beliefs and preferences, leading the AI system to tune its future recommendations to increasingly match those initial recommendations.

“It is well-known that even a single such AI system can create a feedback loop, whereby its initial recommendations can actually shape the user's beliefs and preferences, leading the AI system to tune its future recommendations to increasingly match those initial recommendations (Ge et al., 2020; Jiang et al., 2019).”

• cites Ge et al., 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Jiang et al., 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Providing fairness guarantees in an ecosystem where multiple AI systems affect the same set of users would require understanding how the fairness guarantees of these AI systems compose.

“Providing fairness guarantees in an ecosystem where multiple AI systems affect the same set of users would require understanding how the fairness guarantees of these AI systems compose”

Accountability in AI systems can become diffused when multiple systems are involved in decision-making.

“Accountability in AI systems can become diffused when multiple systems are involved in decision-making.”

The effect of accountability diffusion becomes complex when advanced AI systems collaborate, especially when there might be emergent phenomena that are difficult (if not impossible) to attribute to any one agent.

“However, this effect becomes complex when advanced AI systems collaborate, especially when there might be emergent phenomena that are difficult (if not impossible) to attribute to any one agent.”

When decisions need to be discrete, perfect fairness is often unachievable.

“when decisions need to be discrete, perfect fairness is often unachievable”

• cites Amanatidis et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Iterative AI-driven transformations can amplify errors and biases.

“suggesting that iterative AI-driven transformations can amplify errors and biases.”

When multiple AI systems make their decisions independently, the minimal unfairness exhibited by each system can compound due to each system potentially providing less beneficial treatment to the same individuals or groups.

“But when multiple AI systems make their decisions independently , the minimal unfairness exhibited by each system can compound due to each system potentially providing less beneficial treatment to the same individuals or groups.”

Growth in privacy violation can lead to individuals' personal information being exposed and used in ways that they did not consent to.

“which can lead to individuals' personal information being exposed and used in ways that they did not consent to.”

Privacy violations can add up when multiple AI systems interact with the same users.

“Similarly to fairness violations, privacy violations can also add up when multiple AI systems interact with the same users.”

In an environment where the number of AI systems operating and interacting with the same set of users cannot be controlled, privacy violation can grow quickly.

“In an environment where the number of AI systems operating and interacting with the same set of users cannot be controlled, privacy violation can grow quickly”

Much of the fairness literature is devoted to understanding fairness of predictions, recommendations, or decisions made by a single AI system.

“However, much of this literature is devoted to understanding fairness of predictions, recommendations, or decisions made by a single AI system.”

• cites Balcan et al., 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Hossain et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Zemel et al., 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.