CoChat
Sign Up
Cited by 133Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningArtificial Intelligence in Games

Automatic Curriculum Learning For Deep RL: A Short Survey

Open access
Portelas, Rémy, Colas, Cédric, Weng, Lilian, Hofmann, Katja, Oudeyer, Pierre-Yves·2020·Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
DOI 10.24963/ijcai.2020/671
Open this paper in CoChat — free accountOpen PDF

Abstract

Automatic Curriculum Learning (ACL) has become a cornerstone of recent successes in Deep Reinforcement Learning (DRL). These methods shape the learning trajectories of agents by challenging them with tasks adapted to their capacities. In recent years, they have been used to improve sample efficiency and asymptotic performance, to organize exploration, to encourage generalization or to solve sparse reward problems, among others. To do so, ACL mechanisms can act on many aspects of learning problems. They can optimize domain randomization for Sim2Real transfer, organize task presentations in multi-task robotic settings, order sequences of opponents in multi-agent scenarios, etc. The ambition of this work is dual: 1) to present a compact and accessible introduction to the Automatic Curriculum Learning literature and 2) to draw a bigger picture of the current state of the art in ACL to encourage the cross-breeding of existing concepts and the emergence of new ideas.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.24963/ijcai.2020/671

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • ALP-GMM is a particularly relevant method in ACL, introduced by Portelas et al. [40].

    “A particular relevant method in this space is ALP-GMM , introduced by Portelas et al. [40].”

    From Improving Environment Novelty Quantification for Effective Unsupervised Environment Design (2024) · cites this paper as [40]

    The survey’s classification table lists ALP-GMM as an ACL method, and its “Intermediate difficulty” section explains how the method uses absolute learning progress. But the survey credits ALP-GMM to Portelas et al. (2019); the cited 2020 survey does not introduce it. Page numbers are unavailable in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Both environment generation and DR can often be seen as a form of automatic curriculum learning, especially if the context distribution is changing during training and adapting to the agent's performance.

    “Both can often also be seen as a form of automatic curriculum learning (Portelas et al., 2020), especially if the context distribution is changing during training and adapting to the agent's performance.”

    From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Portelas et al., 2020

  • UED is related to Automated Curriculum Learning (ACL), which encompasses a family of mechanisms that automatically adapt the distribution of training data by selecting learning situations tailored to the capabilities of DRL agents.

    “UED is related to Automated Curriculum Learning (ACL;[42]), which emcompasses a family of mechanisms that automatically adapt the distribution of training data by selecting learning situations tailored to the capabilities of DRL agents.”

    From Improving Environment Novelty Quantification for Effective Unsupervised Environment Design (2024) · cites this paper as [42]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 14 references couldn’t be matched to a known paper.

Research works printed but not linked: 45

  1. Marcin Andrychowicz et al. Hind- sight experience replay. In NeurIPS.
  2. Baker et al., 2019] Bowen Baker et al. Emergent tool use from multi-agent autocurricula. arXiv.
  3. Trapit Bansal et al. Emergent complexity via multi-agent competition. arXiv.
  4. Bellemare et al., 2016] Marc Bellemare et al. Unifying count- based exploration and intrinsic motivation. In NeurIPS. [Bengio et al., 2009]
  5. Yoshua Bengio et al. Curriculum learning. In ICML.
  6. Yuri Burda et al. Exploration by random network distillation. ICLR.
  7. Cideron et al., 2019] Geoffrey Cideron et al. Self-educated language agent with hindsight experience replay for instruc- tion following. arXiv.
  8. Cobbe et al., 2018] Karl Cobbe et al. Quantifying generaliza- tion in reinforcement learning. arXiv. [Colas et al., 2019] Cédric Colas et al. CURIOUS: Intrinsi- cally motivated modular multi-goal reinforcement learning. In ICML.
  9. Colas et al., 2020] Cédric Colas et al. Language as a cognitive tool to imagine goals in curiosity-driven exploration. arXiv.
  10. Wojciech Czarnecki et al. Mix & match agent curricula for reinforcement learning. In ICML.
  11. Wojciech Marian Czarnecki et al. Dis- tilling policy distillation. arXiv.
  12. Benjamin Eysenbach et al. Diversity is all you need: Learning skills without a reward function. arXiv.
  13. Meng Fang et al. Curriculum-guided hind- sight experience replay. In NeurIPS.
  14. Carlos Florensa et al. Reverse curricu- lum generation for reinforcement learning. CoRL.
  15. Florensa et al., 2018] Carlos Florensa et al. Automatic goal generation for reinforcement learning agents. In ICML.
  16. Forestier et al., 2017] Sébastien Forestier et al. Intrinsically motivated goal exploration processes with automatic curricu- lum learning. arXiv.
  17. Fournier et al., 2018] Pierre Fournier et al. Accuracy-based curriculum learning in deep reinforcement learning. arXiv.
  18. Franc ¸ois-Lavet et al., 2015] Vincent Franc ¸ois-Lavet et al. How to discount deep reinforcement learning: Towards new dy- namic strategies. arXiv.
  19. Dan Horgan et al. Distributed prioritized experience replay. arXiv.
  20. Ivanovic et al., 2018] Boris Ivanovic et al. Barc: Backward reachability curriculum for robotic reinforcement learning. ICRA.
  21. Allan Jabri et al. Unsupervised curricula for visual meta-reinforcement learning. In NeurIPS.
  22. Jaderberg et al., 2016] Max Jaderberg et al. Reinforcement learning with unsupervised auxiliary tasks. arXiv.
  23. Lair et al., 2019] Nicolas Lair et al. Language grounding through social interactions and curiosity-driven multi-goal learning. arXiv.
  24. Laversanne-Finot et al., 2018] Adrien Laversanne-Finot et al. Curiosity driven exploration of learned disentangled goal spaces. [Lesort et al., 2019] Timothée Lesort et al. Continual learning for robotics. arXiv.
  25. Matiisen et al., 2017] Tambet Matiisen et al. Teacher-student curriculum learning. IEEE TNNLS.
  26. Mehta et al., 2019] Bhairav Mehta et al. Active domain ran- domization. CoRL.
  27. S. Mysore et al. Reward-guided curricu- lum for robust reinforcement learning. preprint.
  28. Karthik Narasimhan et al. Language understanding for text-based games using deep reinforcement learning. arXiv.
  29. OpenAI et al., 2019] OpenAI et al. Solving rubik's cube with a robot hand. arXiv.
  30. Pierre-Yves Oudeyer et al. Intrinsic mo- tivation systems for autonomous mental development. IEEE trans. on evolutionary comp. [Pathak et al., 2017] Deepak Pathak et al. Curiosity-driven ex- ploration by self-supervised prediction. In CVPR.
  31. Deepak Pathak et al. Self-supervised ex- ploration via disagreement. arXiv.
  32. Pinto et al., 2017] Lerrel Pinto et al. Robust adversarial rein- forcement learning. arXiv.
  33. Vitchyr H. Pong et al. Skew-fit: State- covering self-supervised reinforcement learning. arXiv.
  34. Portelas et al., 2019] Rémy Portelas et al. Teacher algorithms for curriculum learning of deep rl in continuously parameter- ized environments. CoRL.
  35. Sébastien Racanière et al. Automated curricula through setter-solver interactions. arXiv.
  36. Risi and Togelius, 2019] Sebastian Risi and Julian Togelius. Procedural content generation: From automatically generat- ing game levels to increasing generality in machine learning. arXiv.
  37. Schaul et al., 2015a] Tom Schaul et al. Universal value func- tion approximators. In ICML.
  38. Schaul et al., 2015b] Tom Schaul et al. Prioritized experience replay. arXiv.
  39. Oliver G Selfridge et al. Training and tracking in robotics. In IJCAI.
  40. Pranav Shyam et al. Model-based active exploration. arXiv.
  41. David Silver et al. Mastering the game of go without human knowledge. Nature. [Sukhbaatar et al., 2017] Sainbayar Sukhbaatar et al. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv.
  42. Rui Wang et al. Paired open-ended trail- blazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv.
  43. Zha et al., 2019] Daochen Zha et al. Experience replay opti- mization. arXiv.
  44. Rui Zhao and Volker Tresp. Energy- based hindsight experience prioritization. arXiv.
  45. Rui Zhao and Volker Tresp. Curiosity- driven experience prioritization via density estimation. arXiv.

Cited by

Sorted by citations

Related

Authors
Portelas, R., Colas, C., Weng, L., Hofmann, K., & Oudeyer, P.-Y.
Affiliations
  • Institut national de recherche en sciences et technologies du numérique (FR)
  • OpenAI (United States) (US)
  • Microsoft (United States) (US)
ORCID
Venue
Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
Publisher
International Joint Conferences on Artificial Intelligence Organization
Published
2020-07
Pages
4819-4825
DOI
10.24963/ijcai.2020/671
OpenAlex
W3011429776
Cited by
127
References
14
Citations by year
2026: 10 · 2025: 19 · 2024: 20 · 2023: 20 · 2022: 23 · 2021: 24 · 2020: 11
FWCI
7.45
Type
conference paper
Language
en
Open access
Locations
  • International Joint Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
Keywords
deep reinforcement learning, sample efficiency, exploration, generalization, sparse rewards, domain randomization, sim-to-real transfer, multi-task learning, multi-agent learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review