Generating Personas for Games with Multimodal Adversarial Imitation Learning
Open accessThe DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
Reinforcement learning has been widely successful in producing agents capable of playing games at a human level. However, this requires complex reward engineering, and the agent’s resulting policy is often unpredictable. Going beyond reinforcement learning is necessary to model a wide range of human playstyles, which can be difficult to represent with a reward function. This paper presents a novel imitation learning approach to generate multiple persona policies for playtesting. Multimodal Generative Adversarial Imitation Learning (Multi-GAIL) uses an auxiliary input parameter to learn distinct personas using a single-agent model. MultiGAIL is based on generative adversarial imitation learning and uses multiple dis-criminators as reward models, inferring the environment reward by comparing the agent and distinct expert policies. The reward from each discriminator is weighted according to the auxiliary input. Our experimental analysis demonstrates the effectiveness of our technique in two environments with continuous and discrete action spaces.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies, such as [11], [20], [37], [43], address this issue by utilizing human gameplay data to define feasible behavior targets.”
From UBCL: A Reinforcement Learning Framework for Controllable and Diverse Player Behaviors (2026) · cites this paper as [20]
Citations
Cites
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 1×
“Policy learning is done through Proximal Policy Optimization (PPO) [29] with the reward signal given by Equation (2).”
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
“Least squares generative adversarial networks,”
Mao, X., Li, Q., Xie, H., Lau, R. Y. K., Wang, Z., & Smolley, S. P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.304 · Cited by 5,437
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018) · Science · doi:10.1126/science.aar6404 · Cited by 3,562
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 1×
“at a human, or even beyond human, level of performance [1, 2, 3].”
BackgroundGenerative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314 · cited 2×
“However, leveraging standard RL by defining and shaping a complex reward function like the work by Le Pelletier de Woillemont et al. [18], can be impractical due to the difficulty of a human to define a reward function that captures qualitative behaviors [19].”
BackgroundMethodMastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.01815 · Cited by 1,102 · cited 1×
“Monte-Carlo Tree Search (MCTS) has been shown to be able to master games such as Chess, Shogi and Go [12], but has also been applied to mimic human playstyles in Match-3 games to analyze the performance range of human players [13].”
BackgroundDota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065 · cited 1×
“Self-learning Reinforcement Learning (RL) agents have been successful in creating agents able to play complex games at a human, or even beyond human, level of performance [1, 2, 3].”
BackgroundChallenges of real-world reinforcement learning: definitions, benchmarks and analysis
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021) · Machine Learning · doi:10.1007/s10994-021-05961-4 · Cited by 645
AMP: adversarial motion priors for stylized physics-based character control
Peng, X. B., Ma, Z., Abbeel, P., Levine, S., & Kanazawa, A. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459670 · Cited by 396
A framework for behavioural cloning
Bain, M., & Sammut, C. (2000) · Machine Intelligence 15 · doi:10.1093/oso/9780198538677.003.0006 · Cited by 360
Finn, C., Christiano, P. F., Abbeel, P., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03852 · Cited by 185
Wuji: Automatic Online Combat Game Testing Using Evolutionary Deep Reinforcement Learning
Zheng, Y., Xie, X., Su, T., Ma, L., Hao, J., Meng, Z., Liu, Y., Shen, R., Chen, Y., & Fan, C. (2019) · 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00077 · Cited by 174
Automated Playtesting With Procedural Personas Through MCTS With Evolved Heuristics
Holmgard, C., Green, M. C., Liapis, A., & Togelius, J. (2019) · IEEE Transactions on Games · doi:10.1109/tg.2018.2808198 · Cited by 125 · cited 1×
“These methods were implemented in 2D discrete environments by engineering their fitness or reward function to condition the policies [10, 17].”
MethodAugmenting Automated Game Testing with Deep Reinforcement Learning
Bergdahl, J., Gordillo, C., Tollmar, K., & Gisslen, L. (2020) · 2020 IEEE Conference on Games (CoG) · doi:10.1109/cog47356.2020.9231552 · Cited by 91
Self-citationHuman-Like Playtesting with Deep Learning
Gudmundsson, S. F., Eisen, P., Poromaa, E., Nodet, A., Purmonen, S., Kozakowski, B., Meurling, R., & Cao, L. (2018) · 2018 IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2018.8490442 · Cited by 90 · cited 1×
“Supervised deep learning has been used to simulate humanlike playtesting to predict level difficulty [14]; however, the”
MethodAn empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2020) · arXiv · doi:10.48550/arxiv.2003.11881 · Cited by 78
Towards automated video game testing: still a long way to go
Politowski, C., Guéhéneuc, Y., & Petrillo, F. D. S. (2022) · doi:10.1145/3524494.3527627 · Cited by 64
Improving Playtesting Coverage via Curiosity Driven Reinforcement Learning Agents
Gordillo, C., Bergdahl, J., Tollmar, K., & Gisslen, L. (2021) · 2021 IEEE Conference on Games (CoG) · doi:10.1109/cog52621.2021.9619048 · Cited by 50
Self-citationAutomated Gameplay Testing and Validation With Curiosity-Conditioned Proximal Trajectories
Sestini, A., Gisslén, L., Bergdahl, J., Tollmar, K., & Bagdanov, A. D. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3226910 · Cited by 47
Self-citationA GAN-Like Approach for Physics-Based Imitation Learning and Interactive Character Control
Xu, P., & Karamouzas, I. (2021) · Proceedings of the ACM on Computer Graphics and Interactive Techniques · doi:10.1145/3480148 · Cited by 46
MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators
Choi, J. O., & Han, B. (2022) · neural information processing systems · doi:10.52202/068431-2146 · Cited by 29
Stahlke, S., Nova, A., & Mirza-Babaei, P. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3313039 · Cited by 28
Automated Playtesting of Matching Tile Games
Mugrai, L., Silva, F., Holmgard, C., & Togelius, J. (2019) · 2019 IEEE Conference on Games (CoG) · doi:10.1109/cig.2019.8848057 · Cited by 28
Online game testing using scenario-based control of massive virtual users
Cho, C.-S., Sohn, K.-M., Park, C.-J., & Kang, J.-H. (2010) · International Conference on Advanced Communication Technology · Cited by 26
Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learning
Sestini, A., Bergdahl, J., Tollmar, K., Bagdanov, A. D., & Gisslén, L. (2023) · IEEE Conference on Games (CoG) · doi:10.1109/cog57401.2023.10333234 · Cited by 21
Self-citationConfigurable Agent With Reward As Input: A Play-Style Continuum Generation
Le Pelletier de Woillemont, P., Labory, R., & Corruble, V. (2021) · 2021 IEEE Conference on Games (CoG) · doi:10.1109/cog52621.2021.9619127 · Cited by 12
Policy Fusion for Adaptive and Customizable Reinforcement Learning Agents
Sestini, A., Kuhnle, A., & Bagdanov, A. D. (2021) · IEEE Conference on Games (CoG) · doi:10.1109/cog52621.2021.9618983 · Cited by 10
Self-citationMCL-GAN: Generative Adversarial Networks with Multiple Specialized DiscriminatorsPreprint
Choi, J., & Han, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07260 · Cited by 8
Acting with Style: Towards Designer-centred Reinforcement Learning for the Video Games Industry
Aytemiz, B., Jacob, M., & Devlin, S. (2021) · Cited by 8
Imitating Playstyle with Dynamic Time Warping Imitation
Ferguson, M., Devlin, S., Kudenko⋆, D., & Walker, J. A. (2022) · International Conference on the Foundations of Digital Games (FDG) · doi:10.1145/3555858.3555878 · Cited by 5
Cited by
Humanlike Behavior in a Third-Person Shooter with Imitation Learning
Farhang, A., Mulcahy, B., Holden, D., Matthews, I., & Yue, Y. (2024) · IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645651 · Cited by 10
Zhang, J. M., Li, Z., Lu, M., & Gan, C. (2024) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.3915 · Cited by 5
Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensembles
Zhang, R., Luo, Z., Sjölund, J., Mattsson, P., Gisslén, L., & Sestini, A. (2025) · IEEE Conference on Games (CoG) · doi:10.1109/cog64752.2025.11114163 · Cited by 4
Self-citationAugmenting Game AI with Deep Reinforcement Learning
Sestini, A., Bergdahl, J., baghi, A., Barrette-LaPierre, J.-P., Fuchs, F., & Gisslén, L. (2026) · IEEE Conference on Games (CoG) · doi:10.48550/arxiv.2606.20210 · Cited by 1
Self-citationUBCL: A Reinforcement Learning Framework for Controllable and Diverse Player Behaviors
Cilan, A., & Ozgovde, A. (2026) · IEEE Transactions on Games · doi:10.1109/tg.2026.3703355
“Collectively, these approaches demonstrate high fidelity to observed human trajectories but share a key limitation: they depend heavily on large-scale demonstration datasets and struggle to generalize to unseen or intentionally designed behaviors [11], [19], [20], [21].”
Influential
Metadata
- Authors
- Ahlberg, W., Sestini, A., Tollmar, K., & Gisslén, L.
- Affiliations
- Electronic Arts (United States) (US)
- Venue
- 2023 IEEE Conference on Games (CoG)
- Publisher
- IEEE
- Published
- 2023-08-21
- Pages
- 1-8
- OpenAlex
- W4389315142
- Cited by
- 5
- References
- 33
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 2
- FWCI
- 0.89
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Conference on Games (CoG) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Social Sciences › Sociology and Political Science › Digital Games and Media
- Keywords
- generative adversarial imitation learning, reward modeling, auxiliary input, game-playing agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.