HRLB⌃2: A Reinforcement Learning Based Framework for Believable Bots
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
The creation of believable behaviors for Non-Player Characters (NPCs) is key to improve the players’ experience while playing a game. To achieve this objective, we need to design NPCs that appear to be controlled by a human player. In this paper, we propose a hierarchical reinforcement learning framework for believable bots (HRLB⌃2). This novel approach has been designed so it can overcome two main challenges currently faced in the creation of human-like NPCs. The first difficulty is exploring domains with high-dimensional state–action spaces, while satisfying constraints imposed by traits that characterize human-like behavior. The second problem is generating behavior diversity, by also adapting to the opponent’s playing style. We evaluated the effectiveness of our framework in the domain of the 2D fighting game named Street Fighter IV. The results of our tests demonstrate that our bot behaves in a human-like manner.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Nonetheless, they lack specific methods for directing the learned policy to demonstrate a particular play style.”
From UBCL: A Reinforcement Learning Framework for Controllable and Diverse Player Behaviors (2026) · cites this paper as [32]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Graph-Based Algorithms for Boolean Function Manipulation
Bryant (1986) · IEEE Transactions on Computers · doi:10.1109/tc.1986.1676819 · Cited by 8,891
(1999) · Cited by 5,147
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,435
(2015) · Journal of Machine Learning Research · Cited by 1,119
Pretending and believing: Issues in the theory of ToMM
Leslie, A. M. (1994) · Cognition · doi:10.1016/0010-0277(94)90029-9 · Cited by 740 · cited 1×
“Therefore, from a machine-learning standpoint, SFIV provides an excellent testbed for fast planning under uncertainty and cognitive skills based on the Theory of Mind [40].”
MethodSPUDD: Stochastic Planning using Decision Diagrams
Hoey, J., St‐Aubin, R., Hu, A. J., & Boutilier, C. E. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.6704 · Cited by 380
A Storytelling Robot: Modeling and Evaluation of Human-like Gaze Behavior
Mutlu, B., Forlizzi, J., & Hodgins, J. (2006) · 2006 6th IEEE-RAS International Conference on Humanoid Robots · doi:10.1109/ichr.2006.321322 · Cited by 348
Towards automatic personalised content creation for racing games
Togelius, J., De Nardi, R., & Lucas, S. M. (2007) · 2007 IEEE Symposium on Computational Intelligence and Games · doi:10.1109/cig.2007.368106 · Cited by 283
BootES: An R package for bootstrap confidence intervals on effect sizes
Kirby, K. N., & Gerlanc, D. (2013) · Behavior Research Methods · doi:10.3758/s13428-013-0330-5 · Cited by 269
Algebraic Decision Diagrams and Their Applications
Bahar, R. I., Frohm, E. A., Gaona, C. M., Hachtel, G. D., Macii, E., Pardo, A., & Somenzi, F. (1997) · Formal Methods in System Design · doi:10.1023/a:1008699807402 · Cited by 267
Activity Recognition Using Hierarchical Hidden Markov Models on a Smartphone with 3D Accelerometer
Lee, Y.-S., & Cho, S.-B. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-21219-2_58 · Cited by 228
Percentile Optimization for Markov Decision Processes with Parameter Uncertainty
Delage, E., & Mannor, S. (2010) · Operations Research · doi:10.1287/opre.1080.0685 · Cited by 221
A video game description language for model-based or interactive learning
Schaul, T. (2013) · 2013 IEEE Conference on Computational Inteligence in Games (CIG) · doi:10.1109/cig.2013.6633610 · Cited by 214
Li, T. S., Chang, S.-J., & Chen, Y.-X. (2003) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2003.817490 · Cited by 196
The 2014 General Video Game Playing Competition
Perez-Liebana, D., Samothrakis, S., Togelius, J., Schaul, T., Lucas, S. M., Couetoux, A., Lee, J., Lim, C.-U., & Thompson, T. (2016) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2015.2402393 · Cited by 191
Learning to Play the Game of Chess
Thrun, S. (1994) · neural information processing systems · Cited by 176
Principled methods for advising reinforcement learning agents
Wiewiora, E., Cottrell, G. W., & Elkan, C. (2003) · Cited by 147
Imitating human playing styles in Super Mario Bros
Ortega, J. G., Shaker, N., Togelius, J., & Yannakakis, G. N. (2012) · Entertainment Computing · doi:10.1016/j.entcom.2012.10.001 · Cited by 104
Usunier, N., Synnaeve, G., Lin, Z., & Chintala, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.02993 · Cited by 91
Probabilistic Graphical Models
Sucar, L. E. (2015) · Advances in computer vision and pattern recognition · doi:10.1007/978-1-4471-6699-3 · Cited by 88
Strategic Attentive Writer for Learning Macro-Actions
-, A., Vezhnevets, Mnih, V., Agapiou, J., Osindero, S., Graves, A., Vinyals, O., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04695 · Cited by 78
Believable Bots: Can Computers Play Like People?
Hingston, P. (2012) · Springer eBooks · doi:10.1007/978-3-642-32323-2 · Cited by 57
Inverse Reinforcement Learning
Abbeel, P., & Ng, A. Y. (2010) · Springer eBooks · doi:10.1007/978-0-387-30164-8_417 · Cited by 56
Cruz, C. A., & Uresti, J. A. R. (2017) · Entertainment Computing · doi:10.1016/j.entcom.2017.02.003 · Cited by 53
Reinforcement Learning in First Person Shooter Games
McPartland, M., & Gallagher, M. R. (2010) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2010.2100395 · Cited by 50
Modifying MCTS for human-like general video game playing
Khalifa, A., Isaksen, A., Togelius, J., & Nealen, A. (2016) · OAR@UM (University of Malta) · Cited by 48
UT⁁2: Human-like behavior via neuroevolution of combat behavior and replay of human traces
Schrum, J., Karpov, I. V., & Miikkulainen, R. P. (2011) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2011.6032024 · Cited by 38
Online Planning for Large Markov Decision Processes with Hierarchical Decomposition
Bai, A., Wu, F., & Chen, X. (2015) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/2717316 · Cited by 35
Potkonjak, V., Tzafestaş, S. G., Kostić, D., & Djordjevic, G. (2001) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/s0736-5845(01)00005-9 · Cited by 31
Hierarchical model-based reinforcement learning
Jong, N. K., & Stone, P. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390211 · Cited by 31
An approach to level design using procedural content generation and difficulty curves
Adrian, D.-F. H., & Luisa, S.-G. C. A. (2013) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2013.6633640 · Cited by 29
Characterizing and optimizing game level difficulty
Berseth, G., Haworth, B., Kapadia, M., & Faloutsos, P. (2014) · Motion in Games · doi:10.1145/2668064.2668100 · Cited by 25
Real-time performance-driven finger motion synthesis
Mousas, C., & Anagnostopoulos, C. (2017) · Computers & Graphics · doi:10.1016/j.cag.2017.03.001 · Cited by 24
Human-Like Combat Behaviour via Multiobjective Neuroevolution
Schrum, J., Karpov, I. V., & Miikkulainen, R. P. (2012) · Springer eBooks · doi:10.1007/978-3-642-32323-2_5 · Cited by 23
Modeling player-like behavior for game AI design
Conroy, D., Wyeth, P. A., & Johnson, D. M. (2011) · International Conference on Advances in Computer Entertainment Technology (ACE) · doi:10.1145/2071423.2071434 · Cited by 20
Miyashita, S., Lian, X., Zeng, X., Matsubara, T., & Uehara, K. (2017) · ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD) · doi:10.1109/snpd.2017.8022767 · Cited by 16
Hingston, P. (2009) · IEEE Symposium on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2009.5286505 · Cited by 15
A Machine Consciousness Approach to the Design of Human-Like Bots
Arrabales, R., Muñoz, J., Ledezma, A., Gutiérrez, G., & Sanchis, A. (2012) · Springer eBooks · doi:10.1007/978-3-642-32323-2_7 · Cited by 14
To Create Intelligent Adaptive Game Opponent by Using Monte-Carlo for the Game of Pac-Man
Liu, X., Li, Y., He, S., Fu, Y., Yang, J., Ji, D., & Chen, Y. (2009) · International Conference on Natural Computation (ICNC) · doi:10.1109/icnc.2009.633 · Cited by 11
To Create Intelligent Adaptive Game Opponent by Using Monte-Carlo for Tree Search
Yang, J., Gao, Y., He, S., Liu, X., Fu, Y., Chen, Y., & Ji, D. (2009) · International Conference on Natural Computation (ICNC) · doi:10.1109/icnc.2009.710 · Cited by 7
Research works printed but not linked: 6
- Laird, J.E., and Duchi, J.C. (2000). Creating Human-Like Synthetic Characters With Multiple Skill-Levels: A Case Study Using the Soar Quakebot, AAAI.
- Ng, A.Y., Harada, D., and Russell, S. (1999). Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping, ICML.
- Hingston, P. (2012). Assessing Believability. Believable Bots, Springer. · DOI
- Usunier, N., Synnaeve, G., Lin, Z., and Chintala, S. (arXiv, 2016). Episodic Exploration for Deep Deterministic Policies for StarCraft Micromanagement, arXiv.
- Puterman, M.L. (2014). Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons.
- Coe, R. (2002, January 12–14). It’s the effect size, stupid: What effect size is and why it is important. Presented at the Annual Conference of the British Educational Research Association, University of Exeter, Exeter, UK.
Cited by
A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges
Cruz, C. A., & Igarashi, T. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395525 · Cited by 131
The Students’ Flow Experience With the Continuous Intention of Using Online English Platforms
Hong, Z., & Khan, A. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.807084 · Cited by 94
Recent Research on AI in Games
Xia, B., Ye, X., & Abuassba, A. O. (2020) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc48107.2020.9148327 · Cited by 43
Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games
Milani, S., Juliani, A., Momennejad, I., Georgescu, R., Rzepecki, J., Shaw, A., Costello, G., Fang, F., Devlin, S., & Hofmann, K. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581348 · Cited by 36
A Survey on Interactive Reinforcement Learning: Design Principles and Open ChallengesPreprint
Cruz, C. A., & Igarashi, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12949 · Cited by 23
Metadata
- Authors
- Arzate Cruz, C., & Ramirez Uresti, J. A.
- Affiliations
- School of Engineering and Science, Tecnologico de Monterrey, CP 64849 Monterrey, Mexico
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2018-12-01
- Volume
- 8
- Issue
- 12
- Pages
- 2453
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W2902782170
- Cited by
- 21
- References
- 39
- Citations by year
- 2026: 1 · 2025: 2 · 2023: 7 · 2022: 4 · 2021: 2 · 2020: 3 · 2019: 2
- FWCI
- 0.99
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Sociology and Political Science › Digital Games and Media
- Keywords
- non-player characters, hierarchical reinforcement learning, human-like behavior, behavioral diversity, game AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.