The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; none of the 13 sources its claims cite is checked for support yet.
Formal contracts mitigate social dilemmas in multi-agent reinforcement learning
Open accessAbstract
Multi-agent Reinforcement Learning (MARL) is a powerful tool for training autonomous agents acting independently in a common environment. However, it can lead to sub-optimal behavior when individual incentives and group incentives diverge. Humans are remarkably capable at solving these social dilemmas. It is an open problem in MARL to replicate such cooperative behaviors in selfish agents. In this work, we draw upon the idea of formal contracting from economics to overcome diverging incentives between agents in MARL. We propose an augmentation to a Markov game where agents voluntarily agree to binding transfers of reward, under pre-specified conditions. Our contributions are theoretical and empirical. First, we show that this augmentation makes all subgame-perfect equilibria of all Fully Observable Markov Games exhibit socially optimal behavior, given a sufficiently rich space of contracts. Next, we show that for general contract spaces, and even under partial observability, richer contract spaces lead to higher welfare. Hence, contract space design solves an exploration-exploitation tradeoff, sidestepping incentive issues. We complement our theoretical analysis with experiments. Issues of exploration in the contracting augmentation are mitigated using a training methodology inspired by multi-objective reinforcement learning: Multi-Objective Contract Augmentation Learning. We test our methodology in static, single-move games, as well as dynamic domains that simulate traffic, pollution management, and common pool resource management.
Overview
CoChat Summary
TL;DR
The paper formalizes formal contracting as a voluntary, binding zero-sum reward-transfer augmentation to Markov games and shows that sufficiently rich contracts can make socially optimal behavior the only subgame-perfect equilibrium under full observability or detectable deviations. It also introduces MOCA, a multi-objective training method that improves welfare over separate training and gifting and often matches or exceeds joint training in Prisoner’s Dilemma, Public Goods, Harvest, Cleanup, and Emergency Merge.
Key findings
- With full contractible observations \(O_0(s,a)=(s,a)\) and the full contract space \(\Theta_{\text{full}}\), all subgame-perfect equilibria of the augmented fully observable Markov game are jointly optimal; the same holds for partially observable Markov games if deviations from a socially optimal profile are detectable and the contract space is sufficiently rich. If contractible observations are perfectly uninformative (\(\Omega_0=\{0\}\)), the augmented game has the same subgame-perfect equilibria as the original game.
- More expected-transfer-expressive contract spaces improve worst-case and best-case subgame-perfect welfare: if \(C_1 \preceq_{\text{ETE}} C_2\), then \(W(C_1)\le W(C_2)\) and \(B(C_1)\le B(C_2)\). Feature monotonicity follows when one feature map factors through another, and random-feature monotonicity holds under a Markov-chain condition. Arbitrary unconditional transfers are necessary for the welfare lower bound; without them, a “Bribing the Contract Proposer” example shows more contracts can lower equilibrium welfare.
- In experiments, a contract fining a Prisoner’s Dilemma defector 1.5 units makes cooperation dominant and \((C,C)\) the only Nash equilibrium. MOCA reaches joint optimality in Prisoner’s Dilemma for 2, 4, and 8 agents; in complex domains it at least matches joint training in almost all cases, except 2-agent Harvest and 2-agent Cleanup, and in Emergency Merge both vanilla contracting and MOCA significantly outperform separate training. In a 4-agent Cleanup monotonicity study, welfare decreases as the probability \(\alpha\) that cleaning is unobserved increases from 0 to 1, with a plateau for middle values and a sharp drop from \(\alpha=0.75\) to \(\alpha=1.0\).
Methods
- Theoretical design: defines fully and partially observable Markov games, contracts as zero-sum reward transfers conditioned on contractible observations, and a \(C\)-augmented game where agents propose contracts and unanimous acceptance is required. Analysis uses subgame-perfect equilibrium, welfare, expected-transfer expressiveness, and forcing contracts.
- Experimental design: trains with PPO via Ray RLlib and compares joint training, separate training, gifting, vanilla contracting, and MOCA. Domains are Prisoner’s Dilemma, Public Goods, Harvest, Cleanup, and Emergency Merge, with 2, 4, and 8 agents, 5 independent runs, 1M environment steps for static games, 10M for dynamic games, and 15M for the monotonicity study. MOCA has two phases: random contract sampling to estimate values, then frozen proposal/acceptance training with \(\eta=2\) sampled non-proposing agents.
- Data and code: no datasets were generated or analyzed; replication code is linked at https://github.com/Algorithmic-Alignment-Lab/contracts.
Limitations
- The paper states that a single contract proposer is crucial: with multiple proposers, subgame-perfect equilibria may be socially suboptimal. It also states fairness concerns—non-proposers are indifferent between accepting and rejecting, while the proposer captures surplus—and notes that limiting transfers may be needed for fairness at possible welfare cost.
- The paper states that contract spaces were hand-engineered and that scaling requires managing a complexity tradeoff, domain-independent contract design, and better sample efficiency. It also states that arbitrary unconditional transfers are necessary for its welfare-bound result, that contracting in POMGs does not generally reduce to DEC-POMDPs, and that collusion is not considered.
- Evident from the text: experiments rely on low-dimensional, hand-designed contract spaces and engineered features, so generalization to unengineered or real-world domains is untested; the paper itself notes the MOCA phase split is arbitrary, and the observed monotonicity decay is not uniform, with no strong pattern in which contract parameter is selected.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
0 Valid · 1 Not valid · 13 Not checked · 74 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Contracting improves system performance in several complex game environments.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show, both game-theoretically and empirically, that providing agents with the option to contract allows for optimal system performance, in the sense of maximizing sum of agents' rewards, regardless of the initial reward signal.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Arbitrary unconditional transfers are necessary for Proposition 3 to hold. We provide a counterexample to Proposition 3 without arbitrary unconditional transfers in Sect. 4.5.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Hence, the statement continues to hold for a repeatedly proposing agent.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Prisoner's Dilemma, contracting reaches joint optimality for 2, 4, and 8 agents.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
49 more claims
Create a free account to see all 88 claims and to verify the 3 references still open.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 13
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4 More possible contracts improve welfare · p. 12
“Definition 7 We say a contracting space u1D6AF admits arbitrary unconditional transfers if and only if, for any u1D73D ∈ u1D6AF , and any v ∈ ℝ N , we have that u1D73D ′ ∈ u1D6AF , where u1D73D ′ is identical to u1D73D , except for an additional transfer of v at contract acceptance.”
§ 2.2 The contracting augmentation · p. 6
“Definition 3 (Contract) A contract is a function u1D73D ∶ Ω 0 → ℝ N whose range consists of zero-sum vectors”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 9 Conclusion and future work · p. 29
“We propose a specialized algorithm for contracting, MultiAgent Contract Augmentation Learning, and matches or exceeds the welfare of training all agents with one controller.”
§ 5.4 Training · p. 22
“However, it fails to match joint training in Public Goods”
Citations
Cites
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Advances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
Moral Hazard and Observability
Holmstrom, B. (1979) · The Bell Journal of Economics · doi:10.2307/3003320 · Cited by 8,311 · cited 1×
“In our proof of Theorem 1, we use a forcing contract , compare [13].”
(2012) · American Economic Review · doi:10.1257/aer.102.1.i · Cited by 8,043
COUNTERSPECULATION, AUCTIONS, AND COMPETITIVE SEALED TENDERS
Vickrey, W. (1961) · The Journal of Finance · doi:10.1111/j.1540-6261.1961.tb02789.x · Cited by 7,722 · cited 1×
“The setting of an agent proposing state-dependent reward transfers has received considerable attention in contract economics, compare the literature following [27], and mechanism design, compare [16, 28, 40].”
Myerson, R. B. (1981) · Mathematics of Operations Research · doi:10.1287/moor.6.1.58 · Cited by 5,920 · cited 1×
“The setting of an agent proposing state-dependent reward transfers has received considerable attention in contract economics, compare the literature following [27], and mechanism design, compare [16, 28, 40].”
(1994) · Cited by 5,513
The Contract Net Protocol: High-Level Communication and Control in a Distributed Problem Solver
Smith (1980) · IEEE Transactions on Computers · doi:10.1109/tc.1980.1675516 · Cited by 3,716 · cited 1×
“Smith [36] proposes the contract net protocol, which allocates tasks among agents and commits them to complete a particular task.”
Mussa, M., & Rosen, S. (1978) · Journal of Economic Theory · doi:10.1016/0022-0531(78)90085-6 · Cited by 3,306 · cited 1×
“The setting of an agent proposing state-dependent reward transfers has received considerable attention in contract economics, compare the literature following [27], and mechanism design, compare [16, 28, 40].”
The Folk Theorem in Repeated Games with Discounting or with Incomplete Information
Fudenberg, D., & Maskin, E. (1986) · Econometrica · doi:10.2307/1911307 · Cited by 2,395 · cited 1×
“Note that the system performance definition is complicated by the existence of potentially multiple equilibria, compare [7].”
Cited by
Peer Rewarding for Multi-agent Coordination: Why and How
Li, Y., Hernandez, M. P., & Aydın, M. E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39395-1_15
Pavliuchenkov, D., & Gubanov, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39395-1_35
Alqithami, S. (2026) · Mathematical and Computational Applications · doi:10.3390/mca31030069
The Cooperation and Competition Mechanisms In Multi-agent Reinforcement Learning
Li, P., Li, S., & Wang, J. (2026) · ITM Web of Conferences · doi:10.1051/itmconf/20268404004
“In the smart grid system, the decentralized energy system must coordinate to optimize energy production and consumption, but it also faces competition in the energy market [18].”
Xu, X. (2025) · Journal of Computational Methods in Sciences and Engineering · doi:10.1177/14727978251361392
Metadata
- Authors
- Haupt, A., Christoffersen, P., Damani, M., & Hadfield-Menell, D.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Corresponding author
- Haupt, Andreas
- Christoffersen, Phillip
- Venue
- Autonomous Agents and Multi-Agent Systems
- Published
- 2024-10-18
- Volume
- 38
- Issue
- 2
- ISSN
- 1387-2532 (ISSN-L)
- 1573-7454
- OpenAlex
- W4403503820
- Cited by
- 5
- References
- 35
- Citations by year
- 2026: 4 · 2025: 1
- FWCI
- 0.74
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Autonomous Agents and Multi-Agent Systems (publishedVersion, cc-by)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Funders
- Effective Giving
- Massachusetts Institute of Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Keywords
- multi-agent reinforcement learning, social dilemmas, formal contracts, incentive alignment, subgame perfect equilibrium, exploration-exploitation balance, multi-objective reinforcement learning, stochastic games, partial observability, traffic management, pollution management, common-pool resources
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Formal contracts mitigate social dilemmas in multi-agent reinforcement learning. https://app.cochat.ai/papers/10.1007/s10458-024-09682-5
- The paper
- Haupt, A., Christoffersen, P., Damani, M., & Hadfield-Menell, D. (2024). Formal contracts mitigate social dilemmas in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems. https://doi.org/10.1007/s10458-024-09682-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.