Superhuman AI for multiplayer poker
The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
AI now masters six-player poker: Computer programs have shown superiority over humans in two-player games such as chess, Go, and heads-up, no-limit Texas hold'em poker. However, poker games usually include six players—a much trickier challenge for artificial intelligence than the two-player variant. Brown and Sandholm developed a program, dubbed Pluribus, that learned how to play six-player no-limit Texas hold'em by playing against five copies of itself (see the Perspective by Blair and Saffidine). When pitted against five elite professional poker players, or with five copies of Pluribus playing against one professional, the computer performed significantly better than humans over the course of 10,000 hands of poker. Science , this issue p. 885 ; see also p. 864
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Yet, although AI outperforms humans in an ever-broadening range of tasks, from playing poker (20) and Go (21) to translating languages (22) and diagnosing skin cancer (23), it trails far behind when it comes to ToM.”
From Evaluating large language models in theory of mind tasks (2024) · cites this paper as [20]
In the citing paper
“This ability to strategically misrepresent information helped Pluribus become the first AI system to achieve superhuman performance in heads-up, no-limit Texas hold ’em poker.”
From AI deception: A survey of examples, risks, and potential solutions (2024) · cites this paper as [7]
In the citing paper
“Pluribus demonstrated a clear ability to bluff in a video of its game against five professional human poker players.”
From AI deception: A survey of examples, risks, and potential solutions (2024) · cites this paper as [7]
Citations
Cites
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084 · cited 2×
“Real-time search has been necessary for achieving super-human performance in many perfect-information games, including backgammon ( 18 ), chess ( 8 ), and Go ( 9 , 19 ).”
BackgroundMethodNash, J. (1951) · The Annals of Mathematics · doi:10.2307/1969529 · Cited by 7,002
Nash, J. (1989) · Cournot Oligopoly · doi:10.1017/cbo9780511528231.007 · Cited by 5,595
Zur Theorie der Gesellschaftsspiele
v. Neumann, J. (1928) · Mathematische Annalen · doi:10.1007/bf01448847 · Cited by 2,572
Temporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618 · cited 2×
“Real-time search has been necessary for achieving super-human performance in many perfect-information games, including backgammon ( 18 ), chess ( 8 ), and Go ( 9 , 19 ).”
BackgroundMethodCampbell, M., Hoane, A. J., & Hsu, F.-H. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(01)00129-1 · Cited by 1,226
The Complexity of Computing a Nash Equilibrium
Daskalakis, C., Goldberg, P. W., & Papadimitriou, C. H. (2009) · SIAM Journal on Computing · doi:10.1137/070699652 · Cited by 921
DeepStack: Expert-level artificial intelligence in heads-up no-limit poker
Moravčík, M., Schmid, M., Burch, N., Lisý, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., & Bowling, M. (2017) · Science · doi:10.1126/science.aam6960 · Cited by 786
Temporal Difference Learning and TD-Gammon
Tesauro, G. (1995) · ICGA Journal · doi:10.3233/icg-1995-18207 · Cited by 764
Settling the complexity of computing two-player Nash equilibria
Chen, X., Deng, X., & Teng, S.-H. (2009) · Journal of the ACM · doi:10.1145/1516512.1516516 · Cited by 560 · cited 1×
“No polynomial-time algorithm is known for finding a Nash equilibrium in two-player non-zero-sum games, and the existence of one would have sweeping surprising implications in computational complexity theory ( 13 , 14 ).”
BackgroundSuperhuman AI for heads-up no-limit poker: Libratus beats top professionals
Brown, N., & Sandholm, T. (2018) · Science · doi:10.1126/science.aao1733 · Cited by 540 · cited 4×
“The two-player poker AI Libratus sidestepped this issue by only doing real-time search when the remaining game was short enough that the depth limit would extend to the end of the game ( 6 ).”
BackgroundMethodInfluentialSelf-citationHeads-up limit hold’em poker is solved
Bowling, M., Burch, N., Johanson, M., & Tammelin, O. (2015) · Science · doi:10.1126/science.1259433 · Cited by 338
Billings, D., Davidson, A., Schaeffer, J., & Szafron, D. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(01)00130-8 · Cited by 266
One Jump Ahead: Challenging Human Supremacy in Checkers
Schaeffer, J. (1997) · ICGA Journal · doi:10.3233/icg-1997-20207 · Cited by 160
Lossless abstraction of imperfect information games
Gilpin, A., & Sandholm, T. (2007) · Journal of the ACM · doi:10.1145/1284320.1284324 · Cited by 102 · cited 1×
“Lossless abstraction only buckets together strategically identical information situations (46,47).”
BackgroundSelf-citationInapproximability of Nash Equilibrium
Rubinstein, A. (2018) · SIAM Journal on Computing · doi:10.1137/15m1039274 · Cited by 71
Solving imperfect-information games
Sandholm, T. (2015) · Science · doi:10.1126/science.aaa4614 · Cited by 34
Self-citationGibson, R. G. (2014) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r39p2wf2v · Cited by 23 · cited 2×
“CFR is also proven to eliminate iteratively strictly dominated actions in all finite games (23).”
BackgroundGanzfried, S., & Sandholm, T. (2015) · ACM Transactions on Economics and Computation · doi:10.1145/2716322 · Cited by 23
Self-citationThe lemonade stand game competition: solving unsolvable games
Zinkevich, M., Bowling, M., & Wunder, M. (2011) · ACM SIGecom Exchanges · doi:10.1145/1978721.1978730 · Cited by 20
Robust Strategies and Counter-Strategies: From Superhuman to Optimal Play
Johanson, M. B. (2016) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r37w67h0w · Cited by 13
Insecure by Design: Protocols for Encrypted Phone Calls
Murdoch, S. J. (2016) · Computer · doi:10.1109/mc.2016.70 · Cited by 7
Rebooting Computers as Learning Machines
DeBenedictis, E. P. (2016) · Computer · doi:10.1109/mc.2016.156 · Cited by 3
Research works printed but not linked: 26
- S. Ganzfried T. Sandholm in International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) (2011) pp. 533–540.
- K. Berg T. Sandholm AAAI Conference on Artificial Intelligence (AAAI) (2017).
- L. S. Shapley Advances in Game Theory M. Drescher L. S. Shapley A. W. Tucker Eds. (Princeton Univ. Press 1964).
- T. Sandholm in AAAI Conference on Artificial Intelligence (AAAI) (2015) pp. 4127–4131. Senior Member Track.
- M. Johanson N. Burch R. Valenzano M. Bowling in International Conference on Autonomous Agents and Multiagent Systems (AAMAS) (2013) pp. 271–278.
- S. Ganzfried T. Sandholm in AAAI Conference on Artificial Intelligence (AAAI) (2014) pp. 682–690.
- N. Brown S. Ganzfried T. Sandholm in International Conference on Autonomous Agents and Multiagent Systems (AAMAS) (2015) pp. 7–15.
- M. Zinkevich M. Johanson M. H. Bowling C. Piccione in Neural Information Processing Systems (NeurIPS) (2007) pp. 1729–1736.
- E. G. Jackson AAAI Workshop on Computer Poker and Imperfect Information (2013).
- E. G. Jackson AAAI Workshop on Computer Poker and Imperfect Information (2016).
- N. Brown T. Sandholm in International Joint Conference on Artificial Intelligence (IJCAI) (2016) pp. 4238–4239.
- E. G. Jackson AAAI Workshop on Computer Poker and Imperfect Information Games (2017).
- M. Lanctot K. Waugh M. Zinkevich in M. Bowling Neural Information Processing Systems (NeurIPS) (2009) pp. 1078–1086.
- M. Johanson N. Bard M. Lanctot R. Gibson M. Bowling in International Conference on Autonomous Agents and Multiagent Systems (AAMAS) (2012) pp. 837–846.
- R. Gibson M. Lanctot N. Burch D. Szafron M. Bowling in AAAI Conference on Artificial Intelligence (AAAI) (2012) pp. 1355–1361.
- N. Brown T. Sandholm AAAI Conference on Artificial Intelligence (AAAI) (2019).
- S. Ganzfried T. Sandholm in International Joint Conference on Artificial Intelligence (IJCAI) (2013) pp. 120–128.
- N. Brown T. Sandholm B. Amos in Neural Information Processing Systems (NeurIPS) (2018) pp. 7663–7674.
- M. Johanson K. Waugh M. Bowling M. Zinkevich in International Joint Conference on Artificial Intelligence (IJCAI) (2011) pp. 258–265.
- N. Burch M. Schmid M. Moravcik D. Morill M. Bowling in AAAI Conference on Artificial Intelligence (AAAI) (2018) pp. 949–956.
- K. Waugh AAAI Workshop on Computer Poker and Imperfect Information (2013).
- A. Gilpin T. Sandholm T. B. Sørensen in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) (2007) pp. 50–57.
- S. Ganzfried T. Sandholm in International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) (2015) pp. 37–45.
- N. Burch M. Johanson M. Bowling in AAAI Conference on Artificial Intelligence (AAAI) (2014) pp. 602–608.
- M. Moravcik M. Schmid K. Ha M. Hladik S. Gaukrodger in AAAI Conference on Artificial Intelligence (AAAI) (2016) pp. 572–578.
- N. Brown T. Sandholm in Neural Information Processing Systems (NeurIPS) (2017) pp. 689-699.
Cited by
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
“Since then, deep reinforcement learning (RL)- based programs and applications have been developed to determine the degree to which they can compete with human champion players in various games like Texas hold’em poker [11] and Dota 2 [12].”
BackgroundMulti-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
Zhang, K., Yang, Z., & Başar, T. (2021) · Studies in Systems, Decision and Control · doi:10.1007/978-3-030-60990-0_12 · Cited by 1,264
Accelerating scientific discovery with Co-Scientist
Gottweis, J., Weng, W., Daryin, A., Tu, T., Sirkovic, P., Myaskovsky, A., Glowaty, G., Weissenberger, F., Orlandi, A., Popovici, D., et al. (2026) · Nature · doi:10.1038/s41586-026-10644-y · Cited by 1,162
Improving the accuracy of medical diagnosis with causal machine learning
Richens, J. G., Lee, C. M., & Johri, S. (2020) · Nature Communications · doi:10.1038/s41467-020-17419-7 · Cited by 598
Alpaydın, E. (2021) · The MIT Press eBooks · doi:10.7551/mitpress/13811.001.0001 · Cited by 374
Metadata
- Authors
- Brown, N., & Sandholm, T.
- Affiliations
- Meta (United States) (US)
- Computer Science Department, Carnegie Mellon University, Pittsburgh, PA 15213, USA.
- Corresponding author
- Brown, Noam
- Venue
- Science
- Published
- 2019-08-30
- Volume
- 365
- Issue
- 6456
- Pages
- 885-890
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2960876848
- Cited by
- 592
- References
- 24
- Citations by year
- 2026: 34 · 2025: 77 · 2024: 81 · 2023: 83 · 2022: 88 · 2021: 125 · 2020: 77 · 2019: 27
- FWCI
- 36.15
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (1718457, 1617590, 1733556)
- Army Research Office (W911NF-17-1-0082)
- NSF XSEDE (CCR090023)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- no-limit Texas Hold'Em, game AI, human-machine comparison
- MeSH terms
- Games, Recreational, Artificial Intelligence, Game Theory, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.