CoChat
Sign Up
PreprintCited by 168Reinforcement Learning in RoboticsArtificial Intelligence in GamesAI-based Problem Solving and Planning

On the Utility of Learning about Humans for Human-AI Coordination

Open access
Carroll, Micah, Shah, Rohin, Ho, Mark K., Griffiths, Thomas L., Seshia, Sanjit A., Abbeel, Pieter, Dragan, Anca·2019·arXiv
DOI 10.48550/arxiv.1910.05789arXiv 1910.05789
Open this paper in CoChat — free accountOpen PDF

Abstract

While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves. Agents that assume their partner to be optimal or similar to them can converge to coordination protocols that fail to understand and be understood by humans. To demonstrate this, we introduce a simple environment that requires challenging coordination, based on the popular game Overcooked, and learn a simple model that mimics human play. We evaluate the performance of agents trained via self-play and population-based training. These agents perform very well when paired with themselves, but when paired with our human model, they are significantly worse than agents designed to play with the human model. An experiment with a planning algorithm yields the same conclusion, though only when the human-aware planner is given the exact human model that it is playing with. A user study with real humans shows this pattern as well, though less strongly. Qualitatively, we find that the gains come from having the agent adapt to the human's gameplay. Given this result, we suggest several approaches for designing agents that learn about humans in order to better coordinate with them. Code is available at https://github.com/HumanCompatibleAI/overcooked_ai.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1910.05789

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 168

Research works printed but not linked: 25

  1. In Proceedings of the twenty-first international conference on Machine learning, page 1. ACM, 2004.
  2. [2]Rachid Alami, Aurélie Clodic, Vincent Montreuil, Emrah Akin Sisbot, and Raja Chatila. Toward human-aware robot task planning. In AAAI spring symposium: to boldly go where no human-robot team has gone before, pages 39–46, 2006.
  3. [3]Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177, 2018. · arXiv
  4. [4]Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, Intelligent Agents [St. Catherine's College, Oxford, July 1995], pages 103–129, Oxford, UK, UK, 1999. Oxford University. ISBN 0-19-853867-7. URL http: //dl.acm.org/citation.cfm?id=647636.733043.
  5. [5]Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In Proceedings of the 6th conference on Theoretical aspects of rationality and knowledge, pages 195–210. Morgan Kaufmann Publishers Inc., 1996.
  6. [6]Shan Carter and Michael Nielsen. Using artificial intelligence to augment human intelligence. Distill, 2017. doi: 10.23915/distill.00009. https://distill.pub/2017/aia. · DOI
  7. [9]Anca D Dragan, Kenton CT Lee, and Siddhartha S Srinivasa. Legibility and predictability of robot motion. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction, pages 301–308. IEEE Press, 2013.
  8. [11]Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1126–1135. JMLR. org, 2017.
  9. [12]Jakob N Foerster, Francis Song, Edward Hughes, Neil Burch, Iain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent reinforcement learning. arXiv preprint arXiv:1811.01458, 2018. · arXiv
  10. [14]Michael J Gielniak and Andrea L Thomaz. Generating anticipation in robot motion. In 2011 RO-MAN, pages 449–454. IEEE, 2011.
  11. [15]Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66–83. Springer, 2017.
  12. [16]Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 4565–4573, 2016.
  13. [17]M. K. Ho, J. MacGlashan, A. Greenwald, M. L. Littman, E. M. Hilliard, C. Trimbach, S. Brawner, J. B. Tenenbaum, M. Kleiman-Weiner, and J. L. Austerweil. Feature-based joint planning and norm learning in collaborative games. In A. Papafragou, D. Grodner, D. Mirman, and J.C. Trueswell, editors, Proceedings of the 38th Annual Conference of the Cognitive Science Society, pages 1158–1163, Austin, TX, 2016. Cognitive Science Society. http://dl.acm.org/citation.cfm?id=647636.733043 http://dl.acm.org/citation.cfm?id=647636.733043 https://distill.pub/2017/aia https://github.com/openai/baselines https://store.steampowered.com/app/448510/Overcooked/ https://store.steampowered.com/app/448510/Overcooked/
  14. [18]Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez-Guzman, Anto- nio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al. Inequity aversion improves cooperation in intertemporal social dilemmas. In Advances in Neural Infor- mation Processing Systems, pages 3326–3336, 2018.
  15. [19]Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017. · arXiv
  16. [20]Max Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443):859–865, 2019. ISSN 0036-8075. doi: 10.1126/science.aau6249. URL https://science.sciencemag.org/content/364/ 6443/859. · DOI · Paper page
  17. [21]Shervin Javdani, Siddhartha S Srinivasa, and J Andrew Bagnell. Shared autonomy via hindsight optimization. Robotics science and systems: online proceedings, 2015.
  18. [22]Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 464–473. International Foundation for Autonomous Agents and Multiagent Systems, 2017.
  19. [23]Adam Lerer and Alexander Peysakhovich. Learning existing social conventions via observa- tionally augmented self-play. AAAI / ACM conference on Artificial Intelligence, Ethics, and Society, 2019.
  20. [24]Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems, pages 6379–6390, 2017.
  21. [25]Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559–7566. IEEE, 2018.
  22. [26]Stefanos Nikolaidis and Julie Shah. Human-robot cross-training: computational formulation, modeling and evaluation of a human team training strategy. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction, pages 33–40. IEEE Press, 2013.
  23. [27]Eshed Ohn-Bar, Kris M. Kitani, and Chieko Asakawa. Personalized dynamics models for adaptive assistive navigation interfaces. CoRR, abs/1804.04118, 2018. · arXiv
  24. [30]Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88–97, 1991.
  25. [33]David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017. https://science.sciencemag.org/content/364/6443/859 https://science.sciencemag.org/content/364/6443/859 https://openai.com/blog/how-to-train-your-openai-five/ https://openai.com/blog/how-to-train-your-openai-five/ https://openai.com/blog/openai-five-finals/ · arXiv

Authors
Carroll, M., Shah, R., Ho, M. K., Griffiths, T. L., Seshia, S. A., Abbeel, P., & Dragan, A.
Affiliations
  • University of California, Berkeley (US)
  • Princeton University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
32
Pages
5174-5185
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1910.05789
arXiv
1910.05789
OpenAlex
W2970894611
Cited by
168
References
0
Citations by year
2025: 20 · 2024: 45 · 2023: 31 · 2022: 29 · 2021: 25 · 2020: 15 · 2019: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
human-AI coordination, human modeling, Overcooked, self-play, population-based training, coordination protocols, user study, multi-agent games

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review