HRLB⌃2: A Reinforcement Learning Based Framework for Believable Bots
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
The creation of believable behaviors for Non-Player Characters (NPCs) is key to improve the players’ experience while playing a game. To achieve this objective, we need to design NPCs that appear to be controlled by a human player. In this paper, we propose a hierarchical reinforcement learning framework for believable bots (HRLB⌃2). This novel approach has been designed so it can overcome two main challenges currently faced in the creation of human-like NPCs. The first difficulty is exploring domains with high-dimensional state–action spaces, while satisfying constraints imposed by traits that characterize human-like behavior. The second problem is generating behavior diversity, by also adapting to the opponent’s playing style. We evaluated the effectiveness of our framework in the domain of the 2D fighting game named Street Fighter IV. The results of our tests demonstrate that our bot behaves in a human-like manner.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Nonetheless, they lack specific methods for directing the learned policy to demonstrate a particular play style.”
From UBCL: A Reinforcement Learning Framework for Controllable and Diverse Player Behaviors (2026) · cites this paper as [32]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Graph-Based Algorithms for Boolean Function Manipulation
Bryant (1986) · IEEE Transactions on Computers · doi:10.1109/tc.1986.1676819 · Cited by 8,891
(1999) · Cited by 5,147
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,435
(2015) · Journal of Machine Learning Research · Cited by 1,119
Pretending and believing: Issues in the theory of ToMM
Leslie, A. M. (1994) · Cognition · doi:10.1016/0010-0277(94)90029-9 · Cited by 740 · cited 1×
“Therefore, from a machine-learning standpoint, SFIV provides an excellent testbed for fast planning under uncertainty and cognitive skills based on the Theory of Mind [40].”
MethodSPUDD: Stochastic Planning using Decision Diagrams
Hoey, J., St‐Aubin, R., Hu, A. J., & Boutilier, C. E. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.6704 · Cited by 380
A Storytelling Robot: Modeling and Evaluation of Human-like Gaze Behavior
Mutlu, B., Forlizzi, J., & Hodgins, J. (2006) · 2006 6th IEEE-RAS International Conference on Humanoid Robots · doi:10.1109/ichr.2006.321322 · Cited by 348
Towards automatic personalised content creation for racing games
Togelius, J., De Nardi, R., & Lucas, S. M. (2007) · 2007 IEEE Symposium on Computational Intelligence and Games · doi:10.1109/cig.2007.368106 · Cited by 283
Research works printed but not linked: 6
- Laird, J.E., and Duchi, J.C. (2000). Creating Human-Like Synthetic Characters With Multiple Skill-Levels: A Case Study Using the Soar Quakebot, AAAI.
- Ng, A.Y., Harada, D., and Russell, S. (1999). Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping, ICML.
- Hingston, P. (2012). Assessing Believability. Believable Bots, Springer. · DOI
- Usunier, N., Synnaeve, G., Lin, Z., and Chintala, S. (arXiv, 2016). Episodic Exploration for Deep Deterministic Policies for StarCraft Micromanagement, arXiv.
- Puterman, M.L. (2014). Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons.
- Coe, R. (2002, January 12–14). It’s the effect size, stupid: What effect size is and why it is important. Presented at the Annual Conference of the British Educational Research Association, University of Exeter, Exeter, UK.
Cited by
A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges
Cruz, C. A., & Igarashi, T. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395525 · Cited by 131
The Students’ Flow Experience With the Continuous Intention of Using Online English Platforms
Hong, Z., & Khan, A. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.807084 · Cited by 94
Recent Research on AI in Games
Xia, B., Ye, X., & Abuassba, A. O. (2020) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc48107.2020.9148327 · Cited by 43
Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games
Milani, S., Juliani, A., Momennejad, I., Georgescu, R., Rzepecki, J., Shaw, A., Costello, G., Fang, F., Devlin, S., & Hofmann, K. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581348 · Cited by 36
A Survey on Interactive Reinforcement Learning: Design Principles and Open ChallengesPreprint
Cruz, C. A., & Igarashi, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12949 · Cited by 23
Metadata
- Authors
- Arzate Cruz, C., & Ramirez Uresti, J. A.
- Affiliations
- School of Engineering and Science, Tecnologico de Monterrey, CP 64849 Monterrey, Mexico
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2018-12-01
- Volume
- 8
- Issue
- 12
- Pages
- 2453
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W2902782170
- Cited by
- 21
- References
- 39
- Citations by year
- 2026: 1 · 2025: 2 · 2023: 7 · 2022: 4 · 2021: 2 · 2020: 3 · 2019: 2
- FWCI
- 0.99
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Sociology and Political Science › Digital Games and Media
- Keywords
- non-player characters, hierarchical reinforcement learning, human-like behavior, behavioral diversity, game AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: HRLB⌃2: A Reinforcement Learning Based Framework for Believable Bots. https://app.cochat.ai/papers/10.3390/app8122453
- The paper
- Arzate Cruz, C., & Ramirez Uresti, J. A. (2018). HRLB⌃2: A Reinforcement Learning Based Framework for Believable Bots. Applied Sciences. https://doi.org/10.3390/app8122453
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.