Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; none of the 13 sources its claims cite is checked for support yet.

Formal contracts mitigate social dilemmas in multi-agent reinforcement learning

Open access
Haupt, Andreas, Christoffersen, Phillip, Damani, Mehul, Hadfield-Menell, Dylan · 2024 · Autonomous Agents and Multi-Agent Systems

Abstract

Multi-agent Reinforcement Learning (MARL) is a powerful tool for training autonomous agents acting independently in a common environment. However, it can lead to sub-optimal behavior when individual incentives and group incentives diverge. Humans are remarkably capable at solving these social dilemmas. It is an open problem in MARL to replicate such cooperative behaviors in selfish agents. In this work, we draw upon the idea of formal contracting from economics to overcome diverging incentives between agents in MARL. We propose an augmentation to a Markov game where agents voluntarily agree to binding transfers of reward, under pre-specified conditions. Our contributions are theoretical and empirical. First, we show that this augmentation makes all subgame-perfect equilibria of all Fully Observable Markov Games exhibit socially optimal behavior, given a sufficiently rich space of contracts. Next, we show that for general contract spaces, and even under partial observability, richer contract spaces lead to higher welfare. Hence, contract space design solves an exploration-exploitation tradeoff, sidestepping incentive issues. We complement our theoretical analysis with experiments. Issues of exploration in the contracting augmentation are mitigated using a training methodology inspired by multi-objective reinforcement learning: Multi-Objective Contract Augmentation Learning. We test our methodology in static, single-move games, as well as dynamic domains that simulate traffic, pollution management, and common pool resource management.

Overview

CoChat Summary

TL;DR

The paper formalizes formal contracting as a voluntary, binding zero-sum reward-transfer augmentation to Markov games and shows that sufficiently rich contracts can make socially optimal behavior the only subgame-perfect equilibrium under full observability or detectable deviations. It also introduces MOCA, a multi-objective training method that improves welfare over separate training and gifting and often matches or exceeds joint training in Prisoner’s Dilemma, Public Goods, Harvest, Cleanup, and Emergency Merge.

Key findings

  • With full contractible observations \(O_0(s,a)=(s,a)\) and the full contract space \(\Theta_{\text{full}}\), all subgame-perfect equilibria of the augmented fully observable Markov game are jointly optimal; the same holds for partially observable Markov games if deviations from a socially optimal profile are detectable and the contract space is sufficiently rich. If contractible observations are perfectly uninformative (\(\Omega_0=\{0\}\)), the augmented game has the same subgame-perfect equilibria as the original game.
  • More expected-transfer-expressive contract spaces improve worst-case and best-case subgame-perfect welfare: if \(C_1 \preceq_{\text{ETE}} C_2\), then \(W(C_1)\le W(C_2)\) and \(B(C_1)\le B(C_2)\). Feature monotonicity follows when one feature map factors through another, and random-feature monotonicity holds under a Markov-chain condition. Arbitrary unconditional transfers are necessary for the welfare lower bound; without them, a “Bribing the Contract Proposer” example shows more contracts can lower equilibrium welfare.
  • In experiments, a contract fining a Prisoner’s Dilemma defector 1.5 units makes cooperation dominant and \((C,C)\) the only Nash equilibrium. MOCA reaches joint optimality in Prisoner’s Dilemma for 2, 4, and 8 agents; in complex domains it at least matches joint training in almost all cases, except 2-agent Harvest and 2-agent Cleanup, and in Emergency Merge both vanilla contracting and MOCA significantly outperform separate training. In a 4-agent Cleanup monotonicity study, welfare decreases as the probability \(\alpha\) that cleaning is unobserved increases from 0 to 1, with a plateau for middle values and a sharp drop from \(\alpha=0.75\) to \(\alpha=1.0\).

Methods

  • Theoretical design: defines fully and partially observable Markov games, contracts as zero-sum reward transfers conditioned on contractible observations, and a \(C\)-augmented game where agents propose contracts and unanimous acceptance is required. Analysis uses subgame-perfect equilibrium, welfare, expected-transfer expressiveness, and forcing contracts.
  • Experimental design: trains with PPO via Ray RLlib and compares joint training, separate training, gifting, vanilla contracting, and MOCA. Domains are Prisoner’s Dilemma, Public Goods, Harvest, Cleanup, and Emergency Merge, with 2, 4, and 8 agents, 5 independent runs, 1M environment steps for static games, 10M for dynamic games, and 15M for the monotonicity study. MOCA has two phases: random contract sampling to estimate values, then frozen proposal/acceptance training with \(\eta=2\) sampled non-proposing agents.
  • Data and code: no datasets were generated or analyzed; replication code is linked at https://github.com/Algorithmic-Alignment-Lab/contracts.

Limitations

  • The paper states that a single contract proposer is crucial: with multiple proposers, subgame-perfect equilibria may be socially suboptimal. It also states fairness concerns—non-proposers are indifferent between accepting and rejecting, while the proposer captures surplus—and notes that limiting transfers may be needed for fairness at possible welfare cost.
  • The paper states that contract spaces were hand-engineered and that scaling requires managing a complexity tradeoff, domain-independent contract design, and better sample efficiency. It also states that arbitrary unconditional transfers are necessary for its welfare-bound result, that contracting in POMGs does not generally reduce to DEC-POMDPs, and that collusion is not considered.
  • Evident from the text: experiments rely on low-dimensional, hand-designed contract spaces and engineered features, so generalization to unengineered or real-world domains is untested; the paper itself notes the MOCA phase split is arbitrary, and the observed monotonicity decay is not uniform, with no strong pattern in which contract parameter is selected.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10458-024-09682-5

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
From the PDF
Fig. 2 The Contracting Augmentation. Top: Agents can propose contracts : state-dependent, zero-sum, additive augmentations to their reward functions. Other agents can accept or decline proposed contracts. Left: In case of declination, the interaction between agents happens as before. Right: In case of acceptance of the contract, the reward of the agents is altered according to the rules of the contract
Fig. 2 The Contracting Augmentation. Top: Agents can propose contracts : state-dependent, zero-sum, additive augmentations to their reward functions. Other agents can accept or decline proposed contracts. Left: In case of declination, the interaction between agents happens as before. Right: In case of acceptance of the contract, the reward of the agents is altered according to the rules of the contractFrom the PDF
Fig. 3 A table summarizing the theoretical results proven in Sects. 3 and 4. The columns along the top list theorem types, which are (1) optimality when you have a contracting space with sufficient richness, (2) monotonicity in the size of the contract space (3) monotonicity in the space of contractible features. The rows list the various problems setups considered in these sections
Fig. 3 A table summarizing the theoretical results proven in Sects. 3 and 4. The columns along the top list theorem types, which are (1) optimality when you have a contracting space with sufficient richness, (2) monotonicity in the size of the contract space (3) monotonicity in the space of contractible features. The rows list the various problems setups considered in these sectionsFrom the PDF
Image 4
From the PDF
Fig. 5 Under general partial observability, possible welfare levels at equilibrium improve, without necessarily reaching optimal social welfare, assuming arbitrary unconditional transfers
Fig. 5 Under general partial observability, possible welfare levels at equilibrium improve, without necessarily reaching optimal social welfare, assuming arbitrary unconditional transfersFrom the PDF
Image 6
From the PDF
Image 7
From the PDF
Image 8
From the PDF
Image 9
From the PDF
Image 10
From the PDF
Image 11
From the PDF

Verification

Claims the paper makes

0 Valid · 1 Not valid · 13 Not checked · 74 Own results and interpretations

49 more claims

Create a free account to see all 88 claims and to verify the 3 references still open.

See all claims — free account
  • It may state two separate claims as one. 24
  • Probably not a substantial claim. 13

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 35 references

Cited by

See all 5

Metadata

Authors
Haupt, A., Christoffersen, P., Damani, M., & Hadfield-Menell, D.
Affiliations
MIT Computer Science and Artificial Intelligence Laboratory (US)
Corresponding author
  • Haupt, Andreas
  • Christoffersen, Phillip
Venue
Autonomous Agents and Multi-Agent Systems
Published
2024-10-18
Volume
38
Issue
2
ISSN
  • 1387-2532 (ISSN-L)
  • 1573-7454
OpenAlex
W4403503820
Cited by
5
References
35
Citations by year
2026: 4 · 2025: 1
FWCI
0.74
Type
journal article
Language
en
Open access
Licence
Locations
  • Autonomous Agents and Multi-Agent Systems (publishedVersion, cc-by)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
Funders
  • Effective Giving
  • Massachusetts Institute of Technology
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
Keywords
multi-agent reinforcement learning, social dilemmas, formal contracts, incentive alignment, subgame perfect equilibrium, exploration-exploitation balance, multi-objective reinforcement learning, stochastic games, partial observability, traffic management, pollution management, common-pool resources

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Formal contracts mitigate social dilemmas in multi-agent reinforcement learning. https://app.cochat.ai/papers/10.1007/s10458-024-09682-5
The paper
Haupt, A., Christoffersen, P., Damani, M., & Hadfield-Menell, D. (2024). Formal contracts mitigate social dilemmas in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems. https://doi.org/10.1007/s10458-024-09682-5

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.