CoChat

UBCL: A Reinforcement Learning Framework for Controllable and Diverse Player Behaviors

Open access
Cilan, Atahan, Ozgovde, Atay · 2026 · IEEE Transactions on Games

Published version of a preprint

The DOI matches its Crossref record; 46 of the 47 references it cites were found in a registry or an index; 19 of the 33 sources its claims cite were checked for support.

Abstract

This paper introduces a reinforcement learning framework that enables controllable and diverse player behaviors without relying on human gameplay data. Existing approaches often require large-scale player trajectories, train separate models for different player types, or provide no direct mapping between interpretable behavioral parameters and the learned policy, limiting their scalability and controllability. We define player behavior in an N-dimensional continuous space and uniformly sample target behavior vectors from a region that encompasses the subset representing real human styles. During training, each agent receives both its current and target behavior vectors as input, and the reward is based on the normalized reduction in distance between them. This allows the policy to learn how actions influence behavioral statistics, enabling smooth control over attributes such as aggressiveness, mobility, and cooperativeness. A single PPO-based multi-agent policy can reproduce new or unseen play styles without retraining. Experiments conducted in a custom multi-player Unity game show that the proposed framework produces significantly greater behavioral diversity than a win-only baseline and matches specified behavior vectors across diverse targets, with highest accuracy on score-based dimensions and moderate degradation on cooperation and mobility. The method offers a scalable solution for automated playtesting, game balancing, human-like behavior simulation, and replacing disconnected players in online games.

Overview

CoChat Summary

TL;DR

UBCL (Uniform Behavior Conditioned Learning) is a reinforcement learning framework that trains a single PPO policy to reproduce controllable, diverse player behaviors by conditioning agents on target behavior vectors sampled uniformly from an N-dimensional behavior space, without any human gameplay data. In a custom 2v2 Unity game, the policy matched specified behavior vectors across diverse targets and produced greater behavioral diversity than a win-only baseline, with highest accuracy on score-based dimensions and moderate degradation on cooperation and mobility.

Key findings

  • The UBCL policy generally achieved its target behavior vectors across 50-session radar-chart evaluations of representative player types (offensive, collector, exploiter, explorer and variations), maintaining overall tendencies even when specific objectives failed due to limited in-game resources and conflicting player goals.
  • Diversity: in a 2D PCA projection of 1,000 evaluation games, UBCL covered a broad region of the behavior space (interior and edges), CARI produced "directed but edge-dominated" behaviors, and the win-only policy clustered in a narrow region resembling collector-type behavior.
  • Error analysis across 1,250 episodes (5,000 behavior vectors) showed low variance on score-related parameters, higher error on T-Distance (teammate coordination), and a positive, "considerably high" mean error on Mobility, attributed to infeasible targets in the Mobility sampling interval.
  • Behavior-space correlations: strongest negative coupling D/S–K/S (−0.60) from the sum-to-one constraint; K/S–Dominance correlation negligible (0.10); error couplings K/S–Dominance (+0.47) and C/S–D/S (−0.48). Policies with 4, 5, and 6 behavioral dimensions showed comparable convergence.

Methods

  • Design: a single shared PPO actor-critic policy conditioned on a six-dimensional behavior vector (Mobility, T-Distance, C/S ratio, D/S ratio, K/S ratio, Dominance); each agent samples a target vector per episode and receives reward equal to the normalized reduction in Euclidean distance between current and target vectors (Eq. 1, λ scaling; max return = λ = 10).
  • Environment and data: a custom 2v2 grid-based multiplayer game built in Unity with ML-Agents, where players collect coins/diamonds or eliminate opponents; agents observe an (8, 80, 80) grid tensor plus vector game states, current behavior vector, and target behavior vector. No human gameplay data used in training.
  • Training/analysis: ~200 million time steps (≈5,500 hours of gameplay) on an RTX 3080 (12 GB), 32 GB RAM, Intel i5-13600KF, with 16 parallel environment instances; baselines were a win-only policy and a reimplemented CARI policy trained for 200M steps under identical conditions; evaluation used radar charts, PCA projection, per-dimension error box plots, and in-match error trajectories.

Limitations

  • Stated: reliance on manually designed behavioral parameters — six dimensions inevitably constrain the representable play-style space, so generated styles are "discretized approximations of highly detailed human behaviors"; cooperation and long-term strategy remain challenging due to the short-horizon reward formulation; T-Distance error is consistently higher because per-agent distance-reduction reward averages out interaction patterns, giving insufficient signal for joint spatiotemporal control; the Mobility sampling interval includes infeasible targets.
  • Stated as future work (i.e., not yet addressed): interaction-aware coordination objectives, learned latent behavior spaces via autoencoders, validation in continuous-action environments with alternative distance formulations, and a dedicated human study to validate reproduction of individual player styles.
  • Evident from the text: all results come from one custom 2v2 grid game with discrete actions and six hand-chosen metrics, so generalization to other genres, continuous action spaces, or larger behavior-vector dimensionalities is untested; the paper also notes that growing dimensionality expands the target space exponentially and may require longer training or curriculum-based sampling.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tg.2026.3703355

Cites

46 of 47 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Sample scenario that involves representative models of each player in an online multiplayer game. If a player is disconnected, the AI model corresponding to that player can replace it until the player reconnects.
Fig. 1 Sample scenario that involves representative models of each player in an online multiplayer game. If a player is disconnected, the AI model corresponding to that player can replace it until the player reconnects.From the PDF
Image 2
From the PDF
Image 3
From the PDF
Fig. 4 Overview of the agent network architecture. The model processes three observation types: game map, game states, and behavior vectors. The game map is converted into a grid representation and encoded separately using convolutional layers with Leaky ReLU activation. The resulting features are concatenated with the game states and behavior vectors to form a shared representation, which is then passed through fully connected layers and divided into actor and critic branches for PPO training.
Fig. 4 Overview of the agent network architecture. The model processes three observation types: game map, game states, and behavior vectors. The game map is converted into a grid representation and encoded separately using convolutional layers with Leaky ReLU activation. The resulting features are concatenated with the game states and behavior vectors to form a shared representation, which is then passed through fully connected layers and divided into actor and critic branches for PPO training.From the PDF
Fig. 5 Reward convergence curves for different MLP configurations trained for 100M time steps under identical hyperparameters. The theoretical maximum cumulative reward per episode is λ = 10 (Eq. 1).
Fig. 5 Reward convergence curves for different MLP configurations trained for 100M time steps under identical hyperparameters. The theoretical maximum cumulative reward per episode is λ = 10 (Eq. 1).From the PDF
                           Fig. 6. Learned behavior examples. Radar charts illustrate the alignment between target behavior vectors (dashed lines) and actual agent behaviors (solid lines) for four distinct profiles. Each chart is derived from 50 game episodes with a fixed target behavior vector. Shaded areas indicate the 1 σ range. (a) Offensive agent with a high proportion of kill score. (b) Low cooperation offensive agent. (c) High mobility offensive agent. (d) Collector agent primarily scoring from high-risk, high-value resources. (e) High mobility collector agent. (f) Low cooperation collector agent. (g) Exploiter agent focusing on low-risk, widely available resources. (h) Low cooperation exploiter agent. (i) High dominance exploiter agent. ∆ b i highlights parameters with relatively high error. (j) Explorer agent characterized by high mobility and balanced score preferences. (k) High dominance explorer agent. (l) Mean behavior generated by the Win Only policy, showing only solid lines and 1 σ ranges, as target behaviors are not defined for this policy.
Fig. 6. Learned behavior examples. Radar charts illustrate the alignment between target behavior vectors (dashed lines) and actual agent behaviors (solid lines) for four distinct profiles. Each chart is derived from 50 game episodes with a fixed target behavior vector. Shaded areas indicate the 1 σ range. (a) Offensive agent with a high proportion of kill score. (b) Low cooperation offensive agent. (c) High mobility offensive agent. (d) Collector agent primarily scoring from high-risk, high-value resources. (e) High mobility collector agent. (f) Low cooperation collector agent. (g) Exploiter agent focusing on low-risk, widely available resources. (h) Low cooperation exploiter agent. (i) High dominance exploiter agent. ∆ b i highlights parameters with relatively high error. (j) Explorer agent characterized by high mobility and balanced score preferences. (k) High dominance explorer agent. (l) Mean behavior generated by the Win Only policy, showing only solid lines and 1 σ ranges, as target behaviors are not defined for this policy.From the PDF
Fig. 7 Two-dimensional PCA projection of realized behavior vectors. In 1000 evaluation games, one agent was randomly assigned to either the winonly or CARI policy, while the remaining agents followed the UBCL policy with uniformly sampled target behavior vectors to simulate various match scenarios. Blue dots: UBCL, green dots: CARI, red triangles: win-only.
Fig. 7 Two-dimensional PCA projection of realized behavior vectors. In 1000 evaluation games, one agent was randomly assigned to either the winonly or CARI policy, while the remaining agents followed the UBCL policy with uniformly sampled target behavior vectors to simulate various match scenarios. Blue dots: UBCL, green dots: CARI, red triangles: win-only.From the PDF
Fig. 8 Box-and-whisker plot illustrating the distribution of error values for each behavioral parameter across 1,250 game episodes, corresponding to 5,000 behavior vectors generated during gameplay. The error for each parameter is computed as b result -b target .
Fig. 8 Box-and-whisker plot illustrating the distribution of error values for each behavioral parameter across 1,250 game episodes, corresponding to 5,000 behavior vectors generated during gameplay. The error for each parameter is computed as b result -b target .From the PDF
Image 9
From the PDF
Image 10
From the PDF
Fig. 11 Reward convergence curves for 4, 5, and 6-dimensional behavior vector configurations trained under identical conditions.
Fig. 11 Reward convergence curves for 4, 5, and 6-dimensional behavior vector configurations trained under identical conditions.From the PDF

Verification

Claims the paper makes

11 Valid · 4 Not valid · 7 Not checked · 53 Own results and interpretations

31 more claims · 1 with their cited sources checked

Create a free account to see all 75 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 25
  • Probably not a substantial claim. 18

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 47 references

Research works printed but not linked: 2

  1. Rttgers, 2024, Automatic matchmaking in two-versus-two sports, 458
  2. Glavin, 2011, Incorporating reinforcement learning into the creation of human-like autonomous agents in first person shooter games, 16

Cited by

See all 1

Metadata

Authors
Cilan, A., & Ozgovde, A.
Affiliations
  • Turkish Aerospace Industries (Turkey) (TR)
  • Boğaziçi University
Venue
IEEE Transactions on Games
Published
2026-09
Volume
18
Issue
3
Pages
588-601
ISSN
  • 2475-1502 (ISSN-L)
  • 2475-1510
OpenAlex
W7164478481
Cited by
0
References
30
FWCI
0.00
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Games (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
Keywords
behavioral diversity, reinforcement learning, multi-agent reinforcement learning, PPO, game balance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: UBCL: A Reinforcement Learning Framework for Controllable and Diverse Player Behaviors. https://app.cochat.ai/papers/10.1109/tg.2026.3703355
The paper
Cilan, A., & Ozgovde, A. (2026). UBCL: A Reinforcement Learning Framework for Controllable and Diverse Player Behaviors. IEEE Transactions on Games. https://doi.org/10.1109/tg.2026.3703355

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.