Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
Open accessAbstract
Deep Reinforcement Learning has been successfully applied to learn robotic control. However, the corresponding algorithms struggle when applied to problems where the agent is only rewarded after achieving a complex task. In this context, using demonstrations can significantly speed up the learning process, but demonstrations can be costly to acquire. In this paper, we propose to leverage a sequential bias to learn control policies for complex robotic tasks using a single demonstration. To do so, our method learns a goal-conditioned policy to control a system between successive low-dimensional goals. This sequential goal-reaching approach raises a problem of compatibility between successive goals: we need to ensure that the state resulting from reaching a goal is compatible with the achievement of the following goals. To tackle this problem, we present a new algorithm called DCIL-II. We show that DCIL-II can solve with unprecedented sample efficiency some challenging simulated tasks such as humanoid locomotion and stand-up as well as fast running with a simulated Cassie robot. Our method leveraging sequentiality is a step towards the resolution of complex robotic tasks under minimal specification effort, a key feature for the next generation of autonomous robots.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several recent works learn policies using only a handful of demonstrations by replaying demonstrations in new scenes.
“Several recent works learn policies using only a handful of demonstrations by replaying the demonstrations in new scenes [8-11, 46-48].”
From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [47]
The Introduction describes learning a control policy from a single demonstration by converting it into sequential goals, supporting the few-demonstrations part of the claim. Neither that passage nor the paper’s description of replay in the Relabelling section says demonstrations are replayed in new scenes; the latter concerns relabelling the agent’s own trajectories. (LLM, conf=0.84)
Experimental — accuracy not yet measuredReplay-based imitation learning methods are promising for learning manipulation tasks using a handful of demonstrations.
“Replay-based imitation learning methods are promising for learning manipulation tasks using a handful of demonstrations [8-11, 46-48]”
From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 2× · see citing works
“A discount factor γ describes the importance of long-term rewards in this sequence of interactions [11].”
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · see citing works
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 2× · see citing works
“We evaluate DCIL-II in five environments: the Dubins Maze environment [23], the Fetch environment [18], two variants of the Humanoid environment from the OpenAI Gym Mujoco suite [45] [46] and the Cassie Run environment based on [47].”
Rapidly-exploring random trees : a new tool for path planning
LaValle, S. M. (1998) · Cited by 3,565 · see citing works
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · cited 5× · see citing works
“In Option-based HRL [38, 39], an option contains: 1) a set of initial states where the option can be activated, 2) a termination function deciding if the option should be terminated given the current state, 3) the intra-option policy which controls the agent at a single time step scale and, 4) the policy over options…”
InfluentialDubins, L. E. (1957) · American Journal of Mathematics · doi:10.2307/2372560 · Cited by 3,074 · cited 3× · see citing works
“1) Dubins Maze: The Dubins Maze is a navigation task where the agent controls a Dubins car [2] in a 2D maze.”
InfluentialGenerative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942 · cited 1× · see citing works
“Expert demonstrations can accelerate learning in these cases [19, 24].”
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355 · cited 2× · see citing works
“To simplify exploration, GCRL agents often use the Hindsight Experience Replay (HER) relabeling technique [2].”
Peng, X. B., Abbeel, P., Levine, S., & van de Panne, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201311 · Cited by 946 · see citing works
(2016) · neural information processing systems · Cited by 735 · see citing works
Efficient Training of Artificial Neural Networks for Autonomous Navigation
Pomerleau, D. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.88 · Cited by 693 · cited 3× · see citing works
“3).”
Universal Value Function Approximators
Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015) · Cited by 677 · see citing works
Visual Reinforcement Learning with Imagined GoalsPreprint
Nair, A., Pong, V. H., Dalal, M., Bahl, S., Lin, S., & Levine, S. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1807.04742 · Cited by 623 · cited 1× · see citing works
“Other methods plan sequences of simpler goals using a goal-conditioned value function [30, 11, 31, 16].”
Solving Rubik's Cube with a Robot Hand
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.07113 · Cited by 602 · cited 1× · see citing works
“Deep reinforcement learning (DRL) offers an al-1 Sorbonne Université, CNRS, Institut des Systèmes Intelligents et de Robotique, ISIR F-75005 Paris, France chenu.isir@gmail.com ternative, learning control policies from reward signals alone [1, 3, 42].”
A Comparison of Selection Schemes Used in Evolutionary Algorithms
Blickle, T., & Thiele, L. (1996) · Evolutionary Computation · doi:10.1162/evco.1996.4.4.361 · Cited by 601 · cited 1× · see citing works
“Such distribution is obtained using the roulette wheel selection operator commonly used in Evolutionary Algorithms [25].”
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582 · cited 2× · see citing works
“We evaluate DCIL-II in five environments: the Dubins Maze environment [23], the Fetch environment [18], two variants of the Humanoid environment from the OpenAI Gym Mujoco suite [45] [46] and the Cassie Run environment based on [47].”
Dayan, P., & Hinton, G. E. (1992) · Cited by 488 · see citing works
Efficient Reductions for Imitation Learning
Ross, S., & Bagnell, D. (2018) · doi:10.1184/r1/6554699 · Cited by 366 · see citing works
Temporal abstraction in reinforcement learning
Precup, D., & Sutton, R. S. (2000) · Cited by 236 · see citing works
The Option-Critic Architecture
Bacon, P., Harb, J., & Precup, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10916 · Cited by 219 · cited 3× · see citing works
“A state s is STIL DCIL [12] R - DSC [6] Option-Critic [4] PWIL [13] BC [34] Learning from demo.”
Kaelbling, L. P. (1993) · Cited by 206 · see citing works
Go-Explore: a New Approach for Hard-Exploration Problems
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1901.10995 · Cited by 205 · cited 1× · see citing works
“We then extend it to GC-SeqMDPs by incorporating GCRL, and conclude with an overview of STIL .”
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2021) · RePEc: Research Papers in Economics · Cited by 201 · see citing works
Kostrikov, I., Agrawal, K. K., Dwibedi, D., Levine, S., & Tompson, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.02925 · Cited by 155 · cited 1× · see citing works
“Expert demonstrations can accelerate learning in these cases [19, 24].”
Learning Multi-Level Hierarchies with HindsightPreprint
Levy, A., Konidaris, G., Platt, R., & Saenko, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00948 · Cited by 144 · see citing works
Hierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129 · cited 2× · see citing works
“S seq : At step t , an extended state is described by a ( s t , i t ) pair where i t is the index of the current goal.”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
Eysenbach, B., Salakhutdinov, R., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1906.05253 · Cited by 123 · cited 1× · see citing works
“Transitions - p seq : After a successful transition, the goal index is automatically incremented or not: Rewards -”
Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
Johns, E. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9560942 · Cited by 119 · cited 1× · see citing works
“Thus, when considering a sequence of goals corresponding to low-dimensional projections of high-dimensional states, it is essential to ensure that the agent can achieve each goal by reaching states compatible with the achievement of the next goal.”
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile CriticsPreprint
Kuznetsov, A., Shvechikov, P., Grishin, A., & Vetrov, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04269 · Cited by 110 · cited 1× · see citing works
“Demonstrations are obtained after 18M training steps using the TQC algorithm [49] on a hand-designed dense reward function [50].”
Planning with Goal-Conditioned PoliciesPreprint
Nasiriany, S., Pong, V. H., Lin, S., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08453 · Cited by 108 · cited 1× · see citing works
“Other methods plan sequences of simpler goals using a goal-conditioned value function [30, 11, 31, 16].”
Learning Montezuma's Revenge from a Single DemonstrationPreprint
Salimans, T., & Chen, R. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.03381 · Cited by 102 · cited 1× · see citing works
“Few approaches address single-demonstration learning from scratch, including Backplay [36, 39], PBCS [27], and PWIL [13].”
Hierarchical visuomotor control of humanoidsPreprint
Merel, J., Ahuja, A., Pham, V., Tunyasuvunakool, S., Liu, S., Tirumala, D., Heess, N., & Wayne, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.09656 · Cited by 82 · cited 1× · see citing works
“For instance, a simple 2D Dubins car reaching an ( x, y ) position near a wall may be unable to move to the next goal if it faces the wrong direction (see Fig.”
Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural NetworksPreprint
Houthooft, R., Chen, X., Duan, Y., Schulman, J. D., de Turck, F., & Abbeel, P. (2016) · arXiv.org · doi:10.48550/arxiv.1605.09674 · Cited by 81 · see citing works
Gupta, A., Yu, J., Zhao, T. Z., Kumar, V., Rovinsky, A., Xu, K., Devlin, T. J., & Levine, S. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561384 · Cited by 75 · cited 1× · see citing works
“In future work, we will explore ways to improve performance under the single-reset constraint, possibly by making the robot come back to previously reached states using its own control law, as in [15, 17].”
Learning From Demonstration in the Wild
Behbahani, F., Shiarlis, K., Chen, X., Kurin, V., Kasewa, S., Stirbu, C., Gomes, J., Paul, S., Oliehoek, F. A., Messias, J. V., et al. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794412 · Cited by 57 · cited 1× · see citing works
“IL methods based on Behavioral Cloning (BC) [5] or Inverse Reinforcement Learning (IRL) [6] fail to imitate complex behaviors when a single demonstration is available [26, 27, 28].”
Option Discovery using Deep Skill Chaining
Bagaria, A., & Konidaris, G. (2020) · International Conference on Learning Representations · Cited by 56 · see citing works
Primal Wasserstein Imitation LearningPreprint
Dadashi, R., Hussenot, L., Geist, M., & Pietquin, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04678 · Cited by 47 · cited 2× · see citing works
“Few approaches address single-demonstration learning from scratch, including Backplay [36, 39], PBCS [27], and PWIL [13].”
Learnings Options End-to-End for Continuous Action TasksPreprint
Klissarov, M., Bacon, P., Harb, J., & Precup, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00004 · Cited by 32 · cited 1× · see citing works
“A similar semi-MDP which also propagates value between options is defined in the Option-Critic framework [41, 43, 44].”
Multi-Value-Functions: Efficient Automatic Action Hierarchies for Multiple Goal MDPs
Moore, A., Baird, L. C., & Kaelbling, L. P. (1999) · Cited by 31 · see citing works
Backplay: "Man muss immer umkehren"Preprint
Resnick, C., Răileanu, R., Kapoor, S., Peysakhovich, A., Cho, K., & Bruna, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.06919 · Cited by 29 · cited 2× · see citing works
“IL methods based on Behavioral Cloning (BC) [5] or Inverse Reinforcement Learning (IRL) [6] fail to imitate complex behaviors when a single demonstration is available [26, 27, 28].”
Goal-Conditioned Reinforcement Learning with Imagined SubgoalsPreprint
Chane-Sane, E., Schmid, C., & Laptev, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00541 · Cited by 24 · cited 2× · see citing works
“Transitions - p seq : After a successful transition, the goal index is automatically incremented or not: Rewards -”
Unified Inter and Intra Options Learning Using Policy Gradient Methods
Levy, K. Y., & Shimkin, N. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-29946-9_17 · Cited by 22 · cited 1× · see citing works
“2c for an illustration).”
Matheron, G., Perrin, N., & Sigaud, O. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61616-8_24 · Cited by 21 · cited 1× · see citing works
“However, the method often suffers from catastrophic forgetting [32] which reduces its applicability to complex imitation tasks.”
Self-citationRobustly Learning Composable Options in Deep Reinforcement Learning
Bagaria, A., Senthil, J., Slivinski, M., & Konidaris, G. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/298 · Cited by 20 · cited 4× · see citing works
“R - DSC [6] adopts this idea, using HER to train local policies and propagate value between options for goal selection.”
InfluentialWhen Waiting Is Not an Option: Learning Options With a Deliberation Cost
Harb, J., Bacon, P., Klissarov, M., & Precup, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11831 · Cited by 19 · cited 1× · see citing works
“A similar semi-MDP which also propagates value between options is defined in the Option-Critic framework [41, 43, 44].”
Divide & Conquer Imitation Learning
Chenu, A., Perrin, N., & Sigaud, O. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9982020 · Cited by 9 · cited 4× · see citing works
“3).”
InfluentialSelf-citation
Flagged first, then by citations
2 of the 48 references couldn’t be matched to a known paper.
Research works printed but not linked: 8
- S. Russell, "Learning agents for uncertain environments," in Proceedings of the eleventh annual conference on Computational learning theory, 1998, pp. 101-103.
- X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne, "Deepmimic: Example-guided deep reinforcement learn- ing of physics-based character skills," ACM Transactions On Graphics (TOG), vol. 37, no. 4, pp. 1-14, 2018.
- Morgan Kaufmann, 1993, pp. 1094-1098.
- T. Schaul, D. Horgan, K. Gregor, and D. Silver, "Uni- versal value function approximators," in International conference on machine learning. PMLR, 2015, pp. 1312-1320.
- O. Nachum, S. S. Gu, H. Lee, and S. Levine, "Data- efficient hierarchical reinforcement learning," Advances in neural information processing systems, vol. 31, 2018.
- R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel, "Vime: Variational in- formation maximizing exploration," Advances in neural information processing systems, vol. 29, 2016.
- Y. Burda, H. Edwards, A. Storkey, and O. Klimov, "Ex- ploration by random network distillation," arXiv preprint arXiv:1810.12894, 2018. · arXiv
- A. Chenu, N. Perrin-Gilbert, and O. Sigaud, "Di- vide & conquer imitation learning," arXiv preprint arXiv:2204.07404, 2022. · arXiv
Cited by
Sorted by citations
SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and DeploymentPreprint
Garrett, C. R., Mandlekar, A., Wen, B., & Fox, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.18907
NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors
Cheng, S., Garrett, C. R., Mandlekar, A., et al. (2023) · Conference on Robot Learning · doi:10.48550/arxiv.2311.01530 · Cited by 12
“Other recent works have proposed to replay a small set of human demos in new situations to facilitate sample-efficient generalization [16, 44, 45, 46, 47, 48, 49, 50], but replay without adaptation can fail for novel object instances.”
BackgroundMimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., & Fox, D. (2023) · arXiv · doi:10.48550/arxiv.2310.17596 · Cited by 5 · see citing works
Online Self-Training for Co-Adaptation in Hierarchical Diffusion Policies
Grislain, C., Kappel, M., Sigaud, O., et al. (2026) · doi:10.48550/arxiv.2603.05291 · Cited by 1
“This can be explained by the partial observability in the subgoal representation [49]: a visual plan can correspond to various execution trajectories, some successful, some not.”
Self-citationLocally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic
Serris, O., Doncieux, S., & Sigaud, O. (2026) · doi:10.48550/arxiv.2608.30406
Self-citationSingle-Reset Divide & Conquer Imitation Learning
Chenu, A., Serris, O., Sigaud, O., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2402.09355
“Value propagation mechanisms [1, 2, 9] can be used to achieve both these objectives.”
BackgroundMethodInfluentialSelf-citationLi, W.-D., Qian, C., Luo, W., et al. (2026) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2026.103431
- Authors
- Chenu, A., Serris, O., Sigaud, O., & Perrin-Gilbert, N.
- Affiliations
- Sorbonne Université (FR)
- ORCID
- Sigaud, Olivier
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2211.04786
- OpenAlex
- W4308757807
- Semantic Scholar
- 6ea2aa7eb2e11b7c4512d8029d071d2e2a99bb80
- Cited by
- 0
- References
- 48
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion)
- arXiv (Cornell University)
- Funders
- European Commission (101070381)
- Agence Nationale de la Recherche (ANR18, ANR-18-CE33-0005)
- Grand Équipement National De Calcul Intensif
- HORIZON EUROPE Framework Programme (101070381)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- goal-conditioned policies, sample efficiency, robot control, humanoid locomotion, Cassie robot, deep reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
