CoChat

Generalization through Diversity: Improving Unsupervised Environment Design

Open access
Li, Wenjun, Varakantham, Pradeep, Li, Dexun · 2023 · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence

The DOI matches its Crossref record; 30 of the 37 references it cites were found in a registry or an index; 3 of the 20 sources its claims cite were checked for support.

Abstract

Agent decision making using Reinforcement Learning (RL) heavily relies on either a model or simulator of the environment (e.g., moving in an 8x8 maze with three rooms, playing Chess on an 8x8 board). Due to this dependence, small changes in the environment (e.g., positions of obstacles in the maze, size of the board) can severely affect the effectiveness of the policy learned by the agent. To that end, existing work has proposed training RL agents on an adaptive curriculum of environments (generated automatically) to improve performance on out-of-distribution (OOD) test scenarios. Specifically, existing research has employed the potential for the agent to learn in an environment (captured using Generalized Advantage Estimation, GAE) as the key factor to select the next environment(s) to train the agent. However, such a mechanism can select similar environments (with a high potential to learn) thereby making agent training redundant on all but one of those environments. To that end, we provide a principled approach to adaptively identify diverse environments based on a novel distance measure relevant to environment design. We empirically demonstrate the versatility and effectiveness of our method in comparison to multiple leading approaches for unsupervised environment design on three distinct benchmark problems used in literature.

Overview

CoChat Summary

TL;DR

DIPLR improves unsupervised environment design by selecting training levels that are diverse according to Wasserstein distance between policy-induced occupancy distributions, rather than relying only on regret/GAE. Across Minigrid, BipedalWalker, and Car-Racing, it outperforms leading UED baselines on zero-shot OOD generalization, with diversity alone often surpassing regret.

Key findings

  • In Minigrid, DIPLR surpasses all benchmark approaches and exceeds the leading baseline PLR⊥ by more than 20% in both IQM and optimality gap; the diversity-only ablation DIPLR− performs on par or better than PLR⊥.
  • In BipedalWalker, DIPLR dominates all benchmarks in both IQM and optimality gap, and DIPLR− surpasses PLR⊥ in five test environments, indicating diversity alone contributes more than regret in continuous domains.
  • In Car-Racing, DIPLR outperforms PLR⊥ and Domain Randomization on both IQM and optimality gap.
  • The paper reports that diversity solely works better than regret in the model, and the two complement each other to achieve the best performance when combined.

Methods

  • DIPLR builds on PLR⊥ and uses Wasserstein distance between occupancy distributions induced by the current student policy’s trajectories to measure level diversity; it combines diversity and regret/GAE via replay probability \(P_{\text{replay}} = \rho \cdot P_D + (1-\rho) \cdot P_R\).
  • Experiments compare DIPLR and DIPLR− against Domain Randomization, Minimax, PAIRED, and PLR⊥ on three UPOMDP domains: Minigrid, BipedalWalker, and Car-Racing, using PPO for teacher and student training.
  • Training budgets: Minigrid 30k PPO updates (~250M steps), BipedalWalker 30k PPO updates (~1B steps), Car-Racing ~3k PPO updates (~5.5M steps); evaluation uses zero-shot OOD test environments and reports IQM and optimality gap, with Minigrid and BipedalWalker aggregate results over 10 runs and Car-Racing over 10 independent runs.

Limitations

  • The paper does not state a limitations section.
  • Evident from the text: evaluation is limited to three simulated benchmark domains (Minigrid, BipedalWalker, Car-Racing) and zero-shot OOD test sets, with no real-world or broader-domain validation.
  • Evident from the text: DIPLR relies on empirical Wasserstein distance over collected trajectories and tunable parameters such as \(\rho\) and \(\beta\), but the paper reports no sensitivity or computational-cost analysis for these choices.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.24963/ijcai.2023/601

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 1 Not valid · 8 Not checked · 39 Own results and interpretations

21 more claims

Create a free account to see all 50 claims and to verify the 3 references still open.

See all claims — free account
  • It may state two separate claims as one. 16
  • Probably not a substantial claim. 11
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Research works printed but not linked: 2

  1. [Flamary et al. , 2021] Remi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z Alaya, Aurelie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, et al. Pot: Python optimal transport. J. Mach. Learn. Res. , 22(78):18, 2021.
  2. [Monge, 1781] Gaspard Monge. M´ emoire sur la th´ eorie des d´ eblais et des remblais. Mem. Math. Phys. Acad. Royale Sci. , pages 666-704, 1781.

Cited by

See all 3

Metadata

Authors
Li, W., Varakantham, P., & Li, D.
Affiliations
Singapore Management University (SG)
Corresponding author
Varakantham, Pradeep
Venue
Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence
Published
2023-08
Pages
5411-5419
OpenAlex
W4385768192
Cited by
3
References
37
Citations by year
2025: 1 · 2024: 1 · 2023: 1
FWCI
0.53
Type
conference paper
Language
en
Open access
Locations
  • International Joint Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Research Foundation
  • National Research Foundation Singapore (AISG2- RP-2020-017)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
out-of-distribution generalization, adaptive curriculum, generalized advantage estimation, reinforcement learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.