The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Automatic Curriculum Learning For Deep RL: A Short Survey
Open accessAbstract
Automatic Curriculum Learning (ACL) has become a cornerstone of recent successes in Deep Reinforcement Learning (DRL). These methods shape the learning trajectories of agents by challenging them with tasks adapted to their capacities. In recent years, they have been used to improve sample efficiency and asymptotic performance, to organize exploration, to encourage generalization or to solve sparse reward problems, among others. To do so, ACL mechanisms can act on many aspects of learning problems. They can optimize domain randomization for Sim2Real transfer, organize task presentations in multi-task robotic settings, order sequences of opponents in multi-agent scenarios, etc. The ambition of this work is dual: 1) to present a compact and accessible introduction to the Automatic Curriculum Learning literature and 2) to draw a bigger picture of the current state of the art in ACL to encourage the cross-breeding of existing concepts and the emergence of new ideas.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A particular relevant method in this space is ALP-GMM , introduced by Portelas et al. [40].”
From Improving Environment Novelty Quantification for Effective Unsupervised Environment Design (2024) · cites this paper as [40]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Both can often also be seen as a form of automatic curriculum learning (Portelas et al., 2020), especially if the context distribution is changing during training and adapting to the agent's performance.”
From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Portelas et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“UED is related to Automated Curriculum Learning (ACL;[42]), which emcompasses a family of mechanisms that automatically adapt the distribution of training data by selecting learning situations tailored to the capabilities of DRL agents.”
From Improving Environment Novelty Quantification for Effective Unsupervised Environment Design (2024) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,825
Learning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 2×
“In the following years, curriculum learning was applied to organize the presentation of training examples or the growth in model capacity in various supervised learning settings [Elman, 1993; Krueger and Dayan, 2009; Bengio et al. , 2009].”
BackgroundMethodInfluentialTransfer Learning for Reinforcement Learning Domains: A Survey
Taylor, M. E., & Stone, P. (2009) · doi:10.5555/1577069.1755839 · Cited by 1,524
Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time
Moore, A., & Atkeson, C. G. (1993) · Machine Learning · doi:10.1023/a:1022635613229 · Cited by 842
Curious model-building control systems
Schmidhuber, J. (1991) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1991.170605 · Cited by 601
Active learning of inverse models with intrinsically motivated goal exploration in robots
Baranès, A., & Oudeyer, P. (2012) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2012.05.008 · Cited by 498
Self-citationOn The Power of Curriculum Learning in Training Deep NetworksPreprint
Hacohen, G., & Weinshall, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03626 · Cited by 325
Flexible shaping: How learning in small steps helps
Krueger, K. A., & Dayan, P. (2009) · Cognition · doi:10.1016/j.cognition.2008.11.014 · Cited by 233
Learning Montezuma's Revenge from a Single DemonstrationPreprint
Salimans, T., & Chen, R. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.03381 · Cited by 152
Reinforcement Learning for Improving Agent Design
Ha, D. (2019) · Artificial Life · doi:10.1162/artl_a_00301 · Cited by 136
Research works printed but not linked: 45
- Marcin Andrychowicz et al. Hind- sight experience replay. In NeurIPS.
- Baker et al., 2019] Bowen Baker et al. Emergent tool use from multi-agent autocurricula. arXiv.
- Trapit Bansal et al. Emergent complexity via multi-agent competition. arXiv.
- Bellemare et al., 2016] Marc Bellemare et al. Unifying count- based exploration and intrinsic motivation. In NeurIPS. [Bengio et al., 2009]
- Yoshua Bengio et al. Curriculum learning. In ICML.
- Yuri Burda et al. Exploration by random network distillation. ICLR.
- Cideron et al., 2019] Geoffrey Cideron et al. Self-educated language agent with hindsight experience replay for instruc- tion following. arXiv.
- Cobbe et al., 2018] Karl Cobbe et al. Quantifying generaliza- tion in reinforcement learning. arXiv. [Colas et al., 2019] Cédric Colas et al. CURIOUS: Intrinsi- cally motivated modular multi-goal reinforcement learning. In ICML.
- Colas et al., 2020] Cédric Colas et al. Language as a cognitive tool to imagine goals in curiosity-driven exploration. arXiv.
- Wojciech Czarnecki et al. Mix & match agent curricula for reinforcement learning. In ICML.
- Wojciech Marian Czarnecki et al. Dis- tilling policy distillation. arXiv.
- Benjamin Eysenbach et al. Diversity is all you need: Learning skills without a reward function. arXiv.
- Meng Fang et al. Curriculum-guided hind- sight experience replay. In NeurIPS.
- Carlos Florensa et al. Reverse curricu- lum generation for reinforcement learning. CoRL.
- Florensa et al., 2018] Carlos Florensa et al. Automatic goal generation for reinforcement learning agents. In ICML.
- Forestier et al., 2017] Sébastien Forestier et al. Intrinsically motivated goal exploration processes with automatic curricu- lum learning. arXiv.
- Fournier et al., 2018] Pierre Fournier et al. Accuracy-based curriculum learning in deep reinforcement learning. arXiv.
- Franc ¸ois-Lavet et al., 2015] Vincent Franc ¸ois-Lavet et al. How to discount deep reinforcement learning: Towards new dy- namic strategies. arXiv.
- Dan Horgan et al. Distributed prioritized experience replay. arXiv.
- Ivanovic et al., 2018] Boris Ivanovic et al. Barc: Backward reachability curriculum for robotic reinforcement learning. ICRA.
- Allan Jabri et al. Unsupervised curricula for visual meta-reinforcement learning. In NeurIPS.
- Jaderberg et al., 2016] Max Jaderberg et al. Reinforcement learning with unsupervised auxiliary tasks. arXiv.
- Lair et al., 2019] Nicolas Lair et al. Language grounding through social interactions and curiosity-driven multi-goal learning. arXiv.
- Laversanne-Finot et al., 2018] Adrien Laversanne-Finot et al. Curiosity driven exploration of learned disentangled goal spaces. [Lesort et al., 2019] Timothée Lesort et al. Continual learning for robotics. arXiv.
- Matiisen et al., 2017] Tambet Matiisen et al. Teacher-student curriculum learning. IEEE TNNLS.
- Mehta et al., 2019] Bhairav Mehta et al. Active domain ran- domization. CoRL.
- S. Mysore et al. Reward-guided curricu- lum for robust reinforcement learning. preprint.
- Karthik Narasimhan et al. Language understanding for text-based games using deep reinforcement learning. arXiv.
- OpenAI et al., 2019] OpenAI et al. Solving rubik's cube with a robot hand. arXiv.
- Pierre-Yves Oudeyer et al. Intrinsic mo- tivation systems for autonomous mental development. IEEE trans. on evolutionary comp. [Pathak et al., 2017] Deepak Pathak et al. Curiosity-driven ex- ploration by self-supervised prediction. In CVPR.
- Deepak Pathak et al. Self-supervised ex- ploration via disagreement. arXiv.
- Pinto et al., 2017] Lerrel Pinto et al. Robust adversarial rein- forcement learning. arXiv.
- Vitchyr H. Pong et al. Skew-fit: State- covering self-supervised reinforcement learning. arXiv.
- Portelas et al., 2019] Rémy Portelas et al. Teacher algorithms for curriculum learning of deep rl in continuously parameter- ized environments. CoRL.
- Sébastien Racanière et al. Automated curricula through setter-solver interactions. arXiv.
- Risi and Togelius, 2019] Sebastian Risi and Julian Togelius. Procedural content generation: From automatically generat- ing game levels to increasing generality in machine learning. arXiv.
- Schaul et al., 2015a] Tom Schaul et al. Universal value func- tion approximators. In ICML.
- Schaul et al., 2015b] Tom Schaul et al. Prioritized experience replay. arXiv.
- Oliver G Selfridge et al. Training and tracking in robotics. In IJCAI.
- Pranav Shyam et al. Model-based active exploration. arXiv.
- David Silver et al. Mastering the game of go without human knowledge. Nature. [Sukhbaatar et al., 2017] Sainbayar Sukhbaatar et al. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv.
- Rui Wang et al. Paired open-ended trail- blazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv.
- Zha et al., 2019] Daochen Zha et al. Experience replay opti- mization. arXiv.
- Rui Zhao and Volker Tresp. Energy- based hindsight experience prioritization. arXiv.
- Rui Zhao and Volker Tresp. Curiosity- driven experience prioritization via density estimation. arXiv.
Cited by
Sorted by citations
A Survey on Curriculum Learning
Wang, X., Chen, Y., & Zhu, W. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3069908 · Cited by 692
Explainability in deep reinforcement learning
Heuillet, A., Couthouis, F., & Díaz-Rodríguez, N. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106685 · Cited by 374
A survey on data‐efficient algorithms in big data era
Adadi, A. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00419-9 · Cited by 331
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224
Voyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222
“[43] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer.”
Background
Metadata
- Authors
- Portelas, R., Colas, C., Weng, L., Hofmann, K., & Oudeyer, P.-Y.
- Affiliations
- Institut national de recherche en sciences et technologies du numérique (FR)
- OpenAI (United States) (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
- Published
- 2020-07
- Pages
- 4819-4825
- OpenAlex
- W3011429776
- Cited by
- 127
- References
- 14
- Citations by year
- 2026: 10 · 2025: 19 · 2024: 20 · 2023: 20 · 2022: 23 · 2021: 24 · 2020: 11
- FWCI
- 7.45
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- deep reinforcement learning, sample efficiency, exploration, generalization, sparse rewards, domain randomization, sim-to-real transfer, multi-task learning, multi-agent learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
