CoChat
Sign Up
Cited by 19Game Theory and ApplicationsEvolutionary Game Theory and CooperationMulti-Agent Systems and Negotiation

Foundations of Cooperative AI

Open access
Conitzer, Vincent, Oesterheld, Caspar·2023·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v37i13.26791
Open this paper in CoChat — free accountOpen PDF

Abstract

AI systems can interact in unexpected ways, sometimes with disastrous consequences. As AI gets to control more of our world, these interactions will become more common and have higher stakes. As AI becomes more advanced, these interactions will become more sophisticated, and game theory will provide the tools for analyzing these interactions. However, AI agents are in some ways unlike the agents traditionally studied in game theory, introducing new challenges as well as opportunities. We propose a research agenda to develop the game theory of highly advanced AI agents, with a focus on achieving cooperation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v37i13.26791

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Safety evaluation datasets and studies do not address the complexities inherent in multi-agent interactions and broader real-world scenarios.

    “These methods do not address the complexities inherent in multi-agent interactions and broader real-world scenarios”

    From Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents (2024) · cites this paper as [13]

    The Introduction says that aligning individual agents is not sufficient to ensure that a multi-agent system is safe, and gives real-world examples of harmful interactions. It does not assess safety evaluation datasets or studies, so it does not establish that those particular methods fail to address these complexities. The indexed copy provides no page numbers for these passages. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 69

  1. Ahmed, A. 2014. Evidence, Decision and Causality. Cam- bridge University Press.
  2. Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P.; Schul- man, J.; and Mané, D. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. · arXiv
  3. Andersen, G.; and Conitzer, V. 2013. Fast Equilibrium Com- putation for Infinitely Repeated Games. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelli- gence, 53-59.
  4. Asker, J.; Fershtman, C.; Pakes, A.; et al. 2022. Artificial Intelligence, Algorithm Design and Pricing. In AEA Papers and Proceedings, volume 112, 452-56. American Economic Association.
  5. Barasz, M.; Christiano, P.; Fallenstein, B.; Herreshoff, M.; LaVictoire, P.; and Yudkowsky, E. 2014. Robust Cooper- ation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic. arXiv preprint arXiv:1401.5577. · arXiv
  6. Bastianello, L.; and Ismail, M. S. 2021. Conditional strategy equilibrium. arXiv preprint arXiv:2103.06928. · arXiv
  7. Basu, K. 1994. The Traveler's Dilemma: Paradoxes of Ra- tionality in Game Theory. American Economic Review, 84(2): 391-395.
  8. Bell, J.; Linsefors, L.; Oesterheld, C.; and Skalse, J. 2021. Reinforcement Learning in Newcomblike Environments. In Ranzato, M.; Beygelzimer, A.; Dauphin, Y.; Liang, P.; and Vaughan, J. W., eds., Advances in Neural Information Pro- cessing Systems, volume 34, 22146-22157.
  9. Beneke, F.; and Mackenrodt, M.-O. 2021. Remedies for al- gorithmic tacit collusion. Journal of Antitrust Enforcement, 9(1): 152-176.
  10. Borgs, C.; Chayes, J.; Immorlica, N.; Kalai, A. T.; Mirrokni, V.; and Papadimitriou, C. 2010. The myth of the Folk The- orem. Games and Economic Behavior, 70(1): 34-43.
  11. Bostrom, N. 2003. Are We Living in a Computer Simula- tion? The Philosophical Quarterly, 53(211): 243-255.
  12. Briggs, R. 2010. Putting a value on Beauty. In Tamar Szabó Gendler and John Hawthorne, ed., Oxford Studies in Episte- mology: Volume 3, 3-34. Oxford University Press.
  13. Brülhart, M.; and Usunier, J.-C. 2012. Does the trust game measure trust? Economics Letters, 115(1): 20-23.
  14. Camerer, C. F. 2003. Behavioral Game Theory: Experiments in Strategic Interaction. Princeton University Press.
  15. Cavalcanti, E. G. 2010. Causation, Decision Theory, and Bell's Theorem: A Quantum Analogue of the Newcomb Problem. The British Journal for the Philosophy of Science, 61(3): 569-597.
  16. Chen, X.; Deng, X.; and Teng, S.-H. 2009. Settling the com- plexity of computing two-player Nash equilibria. Journal of the ACM, 56(3).
  17. Conitzer, V. 2015. A Dutch book against sleeping beau- ties who are evidential decision theorists. Synthese, 192(9): 2887-2899.
  18. Conitzer, V. 2019a. A Puzzle about Further Facts. Erkennt- nis, 84(3): 727-739.
  19. Conitzer, V. 2019b. Designing Preferences, Beliefs, and Identities for Artificial Intelligence. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, 9755-9759.
  20. Conitzer, V. 2020. The Personalized A-Theory of Time and Perspective. Dialectica, 74(1): 1-29.
  21. Conitzer, V.; and Sandholm, T. 2008. New Complexity Re- sults about Nash Equilibria. Games and Economic Behavior, 63(2): 621-641.
  22. Conitzer, V.; and Sandholm, T. 2011. Expressive Markets for Donating to Charities. Artificial Intelligence, 175(7-8): 1251-1271.
  23. Critch, A. 2019. A Parametric, Resource-Bounded Gener- alization of Löb's Theorem, and a Robust Cooperation Cri- terion for Open-Source Game Theory. Journal of Symbolic Logic, 84(4): 1368-1381.
  24. Critch, A.; Dennis, M.; and Russell, S. 2022. Cooper- ative and uncooperative institution designs: Surprises and problems in open-source game theory. arXiv preprint arXiv:2208.07006. · arXiv
  25. Critch, A.; and Krueger, D. 2020. AI Research Consid- erations for Human Existential Safety (ARCHES). arXiv preprint arXiv:2006.04948. · arXiv
  26. Dafoe, A.; Bachrach, Y.; Hadfield, G.; Horvitz, E.; Larson, K.; and Graepel, T. 2021. Cooperative AI: machines must learn to find common ground. Nature, 593(7857): 33-36.
  27. Daskalakis, C.; Goldberg, P.; and Papadimitriou, C. H. 2009. The Complexity of Computing a Nash Equilibrium. SIAM Journal on Computing, 39(1): 195-259.
  28. Demski, A.; and Garrabrant, S. 2019. Embedded agency. arXiv preprint arXiv:1902.09469. · arXiv
  29. Deng, Y.; and Conitzer, V. 2017. Disarmament Games. In Proceedings of the Thirty-First AAAI Conference on Artifi- cial Intelligence, 473-479.
  30. Deng, Y.; and Conitzer, V. 2018. Disarmament Games with Resources. In Proceedings of the Thirty-Second AAAI Con- ference on Artificial Intelligence. New Orleans, LA, USA.
  31. Dorner, F. E. 2021. Algorithmic collusion: A critical review. arXiv preprint arXiv:2110.04740. · arXiv
  32. Draper, K.; and Pust, J. 2008. Diachronic Dutch Books and Sleeping Beauty. Synthese, 164(2): 281-287.
  33. Elga, A. 2000. Self-locating belief and the Sleeping Beauty problem. Analysis, 60(2): 143-147.
  34. Ezrachi, A.; and Stucke, M. E. 2017. Artificial intelligence & collusion: When computers inhibit competition. Univer- sity of Illinois Law Review, (5): 1775-1809.
  35. Ghosh, A.; and Mahdian, M. 2008. Charity auctions on so- cial networks. In Proceedings of the Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 1019-1028.
  36. Hansen, K. T.; Misra, K.; and Pai, M. M. 2021. Frontiers: Algorithmic collusion: Supra-competitive prices via inde- pendent algorithms. Marketing Science, 40(1): 1-12.
  37. Hare, C. 2007. Self-Bias, Time-Bias, and the Metaphysics of Self and Time. The Journal of Philosophy, 104(7): 350-373.
  38. Hare, C. 2009. On Myself, And Other, Less Important Sub- jects. Princeton University Press.
  39. Hare, C. 2010. Realism About Tense and Perspective. Phi- losophy Compass, 5(9): 760-769.
  40. Hellie, B. 2013. Against Egalitarianism. Analysis, 73(2): 304-320.
  41. Hitchcock, C. 2004. Beauty and the bets. Synthese, 139(3): 405-420.
  42. Hutter, A. 2020. Learning in two-player games between transparent opponents. arXiv preprint arXiv:2012.02671. · arXiv
  43. Kalai, A. T.; Kalai, E.; Lehrer, E.; and Samet, D. 2010. A commitment folk theorem. Games and Economic Behavior, 69(1): 127-137.
  44. Kontogiannis, S. C.; and Spirakis, P. G. 2008. Equilibrium Points in Fear of Correlated Threats. In Proceedings of the Fourth Workshop on Internet and Network Economics (WINE), 210-221. Shanghai, China.
  45. Letcher, A.; Foerster, J.; Balduzzi, D.; Rocktäschel, T.; and Whiteson, S. 2019. Stable Opponent Shaping in Differen- tiable Games. In ICLR 2019.
  46. Letchford, J.; Conitzer, V.; and Jain, K. 2008. An Ethical Game-Theoretic Solution Concept for Two-Player Perfect- Information Games. In Proceedings of the Fourth Work- shop on Internet and Network Economics (WINE), 696-707.
  47. Levinstein, B. A.; and Soares, N. 2020. Cheating Death in Damascus. The Journal of Philosophy, 117(5).
  48. Mayer, D.; Feldmaier, J.; and Shen, H. 2016. Reinforcement Learning in Conflicting Environments for Autonomous Ve- hicles. In International Workshop on Robotics in the 21st Century: Challenges and Promises.
  49. Merlo, G. 2016. Subjectivism and the Mental. Dialectica, 70(3): 311-342.
  50. Merlo, G. 2021. The Metaphysical Problem of Other Minds. Pacific Philosophical Quarterly, 102(4): 633-664.
  51. Monderer, D.; and Tennenholtz, M. 2009. Strong mediated equilibrium. Artificial Intelligence, 173(1): 180-195.
  52. Moon, C.; and Conitzer, V. 2015. Maximal Cooperation in Repeated Games on Social Networks. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI), 216-223. Buenos Aires, Argentina.
  53. Moon, C.; and Conitzer, V. 2016. Role Assignment for Game-Theoretic Cooperation. In Proceedings of the Twenty- Fifth International Joint Conference on Artificial Intelli- gence (IJCAI), 416-423. New York, NY, USA.
  54. Musolff, L. 2022. Algorithmic Pricing Facilitates Tacit Col- lusion: Evidence from E-Commerce. In Proceedings of the 23rd ACM Conference on Economics and Computation, 32- 33.
  55. Nozick, R. 1969. Newcomb's Problem and Two Principles of Choice. In Nicholas Rescher et al., ed., Essays in Honor of Carl G. Hempel, 114-146. Springer.
  56. Oesterheld, C. 2019. Robust Program Equilibrium. Theory and Decision, 86(1): 143-159.
  57. Oesterheld, C.; and Conitzer, V. 2021. Extracting Money from Causal Decision Theorists. Philosophical Quarterly, 71(4). DOI 10.1093/pq/pqaa086. · DOI
  58. Oesterheld, C.; Treutlein, J.; Grosse, R.; Conitzer, V.; and Foerster, J. 2022. Similarity-based Cooperation. arXiv preprint arXiv:2211.14468. · arXiv
  59. Peterson, M. 2009. An Introduction to Decision Theory. Cambridge University Press.
  60. Rubinstein, A. 1998. Modeling Bounded Rationality. Zeuthen Lecture Book Series. The MIT Press.
  61. Sandholm, T.; Gilpin, A.; and Conitzer, V. 2005. Mixed- Integer Programming Methods for Finding Nash Equilibria. In Proceedings of the National Conference on Artificial In- telligence (AAAI), 495-501.
  62. Sandholm, T. W.; and Crites, R. H. 1996. Multiagent reinforcement learning in the iterated prisoner's dilemma. Biosystems, 37(1-2): 147-166.
  63. Schwalbe, U. 2018. Algorithms, machine learning, and col- lusion. Journal of Competition Law & Economics, 14(4): 568-607.
  64. Addison Wesley.
  65. Tennenholtz, M. 2004. Program equilibrium. Games and Economic Behavior, 49(2): 363-373.
  66. Titelbaum, M. G. 2013. Ten reasons to care about the Sleep- ing Beauty problem. Philosophy Compass, 8(11): 1003- 1017.
  67. Valberg, J. J. 2007. Dream, Death, and the Self. Princeton University Press.
  68. Weirich, P. 2016. Causal Decision Theory. In The Stanford Encyclopedia of Philosophy. Spring 2016 edition.
  69. Zhang, B.; Farina, G.; Celli, A.; and Sandholm, T. 2022. Op- timal Correlated Equilibria in General-Sum Extensive-Form Games: Fixed-Parameter Algorithms, Hardness, and Two- Sided Column-Generation. Available as arXiv:2203.07181. · arXiv

Cited by

Sorted by citations

Authors
Conitzer, V., & Oesterheld, C.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2023-06-26
Volume
37
Issue
13
Pages
15359-15367
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v37i13.26791
OpenAlex
W4382318237
Cited by
19
References
3
Citations by year
2026: 5 · 2025: 8 · 2024: 5 · 2023: 1
FWCI
9.26
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
  • Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
Keywords
game theory, AI agents, cooperation, AI safety, multi-agent interaction, research agenda

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review