Normative Disagreement as a Challenge for Cooperative AI
Open accessAbstract
Cooperation in settings where agents have both common and conflicting interests (mixed-motive environments) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied have a single cooperative outcome on which all agents can agree. Many real-world multi-agent environments are instead bargaining problems (BPs): they have several Pareto-optimal payoff profiles over which agents have conflicting preferences. We argue that typical cooperation-inducing learning algorithms fail to cooperate in BPs when there is room for normative disagreement resulting in the existence of multiple competing cooperative equilibria, and illustrate this problem empirically. To remedy the issue, we introduce the notion of norm-adaptive policies. Norm-adaptive policies are capable of behaving according to different norms in different circumstances, creating opportunities for resolving normative disagreement. We develop a class of norm-adaptive policies and show in experiments that these significantly increase cooperation. However, norm-adaptiveness cannot address residual bargaining failure arising from a fundamental tradeoff between exploitability and cooperative robustness.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Training a negotiating agent on a distribution of counterparts with too little diversity in their negotiating tactics can lead to catastrophic overconfidence in high-stakes settings.
“For example, training a negotiating agent on a distribution of counterparts with too little diversity in their negotiating tactics can lead to catastrophic overconfidence in high-stakes settings (cf. Stastny et al., 2021), which might already have little precedent in the training data.”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Stastny et al., 2021
Section 3 (p. 3) describes negotiating agents trained independently before deployment, and Section 4.2 (p. 7) reports strong performance degradation when agents use different welfare functions. The Introduction (p. 1) identifies high-stakes decisions as a potential setting. The paper does not attribute these failures to insufficient diversity in a training distribution or describe them as catastrophic overconfidence. (LLM, conf=0.83)
Experimental — accuracy not yet measuredOne possible cause of conflict is a multiplicity of potential solutions (or equilibria).
“One possible cause of conflict is a multiplicity of potential solutions (or equilibria , see also Section 2.1).”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Stastny et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Nash, J. F. (1950) · Econometrica · doi:10.2307/1907266 · Cited by 7,973 · see citing works
Nash, J. F. (1989) · Cambridge University Press eBooks · doi:10.1017/cbo9780511528231.007 · Cited by 5,661 · see citing works
In Search of Homo Economicus: Behavioral Experiments in 15 Small-Scale Societies
Henrich, J., Boyd, R. T., Bowles, S., Camerer, C., Fehr, E., Gintis, H. M., & McElreath, R. L. (2001) · American Economic Review · doi:10.1257/aer.91.2.73 · Cited by 2,619 · see citing works
Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility
Harsanyi, J. C. (1980) · Springer eBooks · doi:10.1007/978-94-010-9327-9_2 · Cited by 2,159 · see citing works
Other Solutions to Nash's Bargaining Problem
Kalai, E., & Smorodinsky, M. (1975) · Econometrica · doi:10.2307/1914280 · Cited by 1,831 · see citing works
Subjectivity and Correlation in Randomized Strategies
Aumann, R. J. (1974) · Journal of Mathematical Economics · doi:10.1016/0304-4068(74)90037-8 · Cited by 1,630 · see citing works
Culture and the evolution of human cooperation
Boyd, R. T., & Richerson, P. J. (2009) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2009.0134 · Cited by 1,028 · see citing works
Repeated Games and Reputations
Mailath, G. J., & Samuelson, L. (2006) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195300796.001.0001 · Cited by 816 · see citing works
Proportional Solutions to Bargaining Situations: Interpersonal Utility Comparisons
Kalai, E. (1977) · Econometrica · doi:10.2307/1913954 · Cited by 759 · see citing works
Repeated Games and Reputations: Long-Run Relationships
Mailath, G. J., & Samuelson, L. (2006) · OUP Catalogue · Cited by 702 · see citing works
Young, H. P. (1996) · The Journal of Economic Perspectives · doi:10.1257/jep.10.2.105 · Cited by 540 · see citing works
Strategic Negotiation in Multiagent Environments
Kraus, S. (2001) · The MIT Press eBooks · doi:10.7551/mitpress/6479.001.0001 · Cited by 500 · see citing works
The Common Prior Assumption in Economic Theory
Morris, S. E. (1995) · Economics and Philosophy · doi:10.1017/s0266267100003382 · Cited by 483 · see citing works
Bendor, J., & Świstak, P. (2001) · American Journal of Sociology · doi:10.1086/321298 · Cited by 339 · see citing works
Burden Sharing and Fairness Principles in International Climate Policy
Ringius, L., Torvanger, A., & Underdal, A. (2002) · International Environmental Agreements Politics Law and Economics · doi:10.1023/a:1015041613785 · Cited by 334 · see citing works
Norms: The Problem of Definition and Classification
Gibbs, J. P. (1965) · American Journal of Sociology · doi:10.1086/223933 · Cited by 306 · see citing works
Sequential Optimality and Coordination in Multiagent Systems
Boutilier, C. E. (1999) · Cited by 294 · see citing works
On the emergence of social conventions: modeling, analysis, and simulations
Shoham, Y., & Tennenholtz, M. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(97)00028-3 · Cited by 290 · see citing works
Emergence of norms through social learning
Sen, S., & Airiau, S. (2007) · International Joint Conference on Artificial Intelligence · Cited by 289 · see citing works
Rosenschein, J. S., & Genesereth, M. (1988) · Elsevier eBooks · doi:10.1016/b978-0-934613-63-7.50027-9 · Cited by 273 · see citing works
COOPERATIVE MODELS OF BARGAININGPreprint
Thomson, W. M. (1994) · RePEc: Research Papers in Economics · Cited by 252 · see citing works
Binmore, K., & Dasgupta, P. S. (1990) · Cited by 224 · see citing works
The strategy of conflict . Prospectus for a reorientation of game theory
Schelling, T. C. (1958) · Journal of Conflict Resolution · doi:10.1177/002200275800200301 · Cited by 220 · see citing works
Learning with Opponent-Learning Awareness
Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., & Mordatch, I. (2018) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/hgwa8807 · Cited by 215 · see citing works
RLlib: Abstractions for Distributed Reinforcement Learning
Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K. Y., Gonzalez, J. E., Jordan, M. I., & Stoica, I. G. (2018) · International Conference on Machine Learning · Cited by 175 · see citing works
Evaluating practical negotiating agents: Results and analysis of the 2011 international competition
Baarslag, T., Fujita, K., Gerding, E., Hindriks, K. V., Itō, T., Jennings, N. R., Jonker, C. M., Kraus, S., Lin, R., Robu, V., et al. (2012) · Artificial Intelligence · doi:10.1016/j.artint.2012.09.004 · Cited by 172 · see citing works
Understanding the Emergence of Conventions in Multi-Agent Systems
Walker, A. K., & Wooldridge, M. (1995) · Cited by 171 · see citing works
An Economist's Perspective On The Evolution Of NormsPreprint
Binmore, K., & Samuelson, L. (1993) · RePEc: Research Papers in Economics · doi:10.22004/ag.econ.292698 · Cited by 159 · see citing works
Multi-agent Reinforcement Learning in Sequential Social DilemmasPreprint
Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., & Graepel, T. (2017) · arXiv (Cornell University) · doi:10.5555/3091125.3091194 · Cited by 151 · see citing works
Justice and Fairness in International Negotiation
Albin, C. (2001) · Cited by 129 · see citing works
Maintaining cooperation in complex social dilemmas using deep reinforcement learning
Lerer, A., & Peysakhovich, A. (2017) · arXiv · doi:10.48550/arxiv.1707.01068 · Cited by 101 · see citing works
Emergent Communication through NegotiationPreprint
Cao, K., Lazaridou, A., Lanctot, M., Leibo, J. Z., Tuyls, K., & Clark, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.03980 · Cited by 94 · see citing works
Open Problems in Cooperative AI
Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., & Graepel, T. (2020) · arXiv · doi:10.48550/arxiv.2012.08630 · Cited by 93 · see citing works
Self-citationAutomated peer-to-peer negotiation for energy contract settlements in residential cooperatives
Chakraborty, S., Baarslag, T., & Kaisers, M. (2019) · Applied Energy · doi:10.1016/j.apenergy.2019.114173 · Cited by 75 · see citing works
How Might Artificial Intelligence Affect the Risk of Nuclear War?
Geist, E., & Lohn, A. J. (2018) · RAND Corporation eBooks · doi:10.7249/pe296 · Cited by 63 · see citing works
Learning to compete, coordinate, and cooperate in repeated games using reinforcement learning
Crandall, J. W., & Goodrich, M. A. (2010) · Machine Learning · doi:10.1007/s10994-010-5192-9 · Cited by 60 · see citing works
Multi-agent Reinforcement Learning in Sequential Social Dilemmas
Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., & Graepel, T. (2017) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/qmsu8421 · Cited by 35
Consequentialist conditional cooperation in social dilemmas with imperfect informationPreprint
Peysakhovich, A., & Lerer, A. K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.06975 · Cited by 27 · see citing works
No Press Diplomacy: Modeling Multi-Agent GameplayPreprint
Paquette, P., Lu, Y., Bocco, S., Smith, M., Ortiz-Gagné, S., Kummerfeld, J. K., Singh, S. P., Pineau, J., & Courville, A. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02128 · Cited by 26 · see citing works
Learning Existing Social Conventions via Observationally Augmented Self-Play
Lerer, A. K., & Peysakhovich, A. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314268 · Cited by 25 · see citing works
Learning Reciprocity in Complex Sequential Social Dilemmas
Eccles, T., Hughes, E., Kramár, J., Wheelwright, S., & Leibo, J. Z. (2019) · arXiv · doi:10.48550/arxiv.1903.08082 · Cited by 19 · see citing works
Equilibrium selection via replicator dynamics in $$2 \times 2$$ 2 × 2 coordination games
Zhang, B., & Hofbauer, J. (2014) · International Journal of Game Theory · doi:10.1007/s00182-014-0437-7 · Cited by 19 · see citing works
Human-Level Performance in No-Press Diplomacy via Equilibrium SearchPreprint
Gray, J., Lerer, A. K., Bakhtin, A., & Brown, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02923 · Cited by 18 · see citing works
Anonymity conditions in social choice theory
Campbell, D. E., & Fishburn, P. C. (1980) · Theory and Decision · doi:10.1007/bf00154656 · Cited by 13 · see citing works
Model-free conventions in multi-agent reinforcement learning with heterogeneous preferencesPreprint
Koster, R., McKee, K. R., Everett, R., Weidinger, L., Isaac, W. S., Hughes, E., Duéñez‐Guzmán, E. A., Graepel, T., Botvinick, M., & Leibo, J. Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09054 · Cited by 11 · see citing works
Legible Normativity for AI Alignment
Hadfield-Menell, D., Andrus, M., & Hadfield, G. K. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314258 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Thomas Anthony, Tom Eccles, Andrea Tacchetti, János Kramár, Ian Gemp, Thomas Hudson, Nicolas Porcel, Marc Lanctot, Julien Perolat, Richard Everett, Satinder Singh, Thore Graepel, and Yoram Bachrach. Learning to play no-press diplomacy with best response policy iteration. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 17987-18003. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ d1419302db9c022ab1d48681b13d5f8b-Paper.pdf .
- Thomas C Schelling. An essay on bargaining. The American Economic Review , 46(3):281-306, 1956.
- Lin Zhou. The nash bargaining theory with non-convex problems. Econometrica: Journal of the Econometric Society , pages 681-685, 1997.
Cited by
Sorted by citations
Multi-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 10 · see citing works
Self-citationUnilateral incentive alignment in two-agent stochastic games
McAvoy, A., Madhushani, U., Hilbe, C., Chatterjee, K., Barfuß, W., Su, Q., Leonard, N. E., & Plotkin, J. B. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2319927121 · Cited by 7 · see citing works
Commitment Games with Conditional Information Disclosure
DiGiovanni, A., & Clifton, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25697 · Cited by 4 · see citing works
Self-citationLearning to Self-Reconfigure for Freeform Modular Robots via Altruism Proximal Policy Optimization
Wu, L., Guo, B., Zhang, Q., Sun, Z., Zhang, J., & Yu, Z. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/610 · Cited by 3 · see citing works
Ethical Guidelines for Designing, Developing, and Deploying AI Negotiation Agents
Smolinski, J., Smolinski, R., Kesting, P., & Kröcher, F. (2026) · Group Decision and Negotiation · doi:10.1007/s10726-026-10011-2 · Cited by 2 · see citing works
Beyond the matrix: Experimental approaches to studying cognitive agents in social-ecological systems
Hertz, U., Koster, R., Janssen, M. A., & Leibo, J. Z. (2024) · Cognition · doi:10.1016/j.cognition.2024.105993 · Cited by 2 · see citing works
- Authors
- Stastny, J., Riché, M., Lyzhov, A., Treutlein, J., Dafoe, A., & Clifton, J.
- Affiliations
- University of Tübingen (DE)
- University of Toronto (CA)
- New York University (US)
- ORCID
- Dafoe, Allan
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2111.13872
- OpenAlex
- W3215800145
- Cited by
- 3
- References
- 44
- Citations by year
- 2024: 1 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Keywords
- normative disagreement, multi-agent learning, bargaining problem, cooperative equilibria, exploitability, bargaining failure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
