Learning Multi-Stage Tasks with One Demonstration via Self-Replay

“We also compared a non learning-based keypoint-detection algorithm, SIFT [31], to automatically extract matching features from the current observation and the bottleneck pose observation, used to compute how to align the end-effector to the bottleneck pose.”

“Reinforcement Learning methods [2] can use demonstrations to bootstrap the agent’s policy [22, 23] or shape its reward function [24, 25].”

“3 - Reinforcement Learning from Demonstrations: we developed Deep Deterministic Policy Gra300 dient from Demonstrations (DDPGfD) [22, 23], that takes human demonstrations to train a policy, 301”

“Manipulation Tasks We designed a set of 8 tasks, inspired by the recent imitation learning literature 257 [3, 13, 4, 5, 7, 8], and by everyday tasks to test our method.”

“3 - Reinforcement Learning from Demonstrations: we developed Deep Deterministic Policy Gra300 dient from Demonstrations (DDPGfD) [22, 23], that takes human demonstrations to train a policy, 301”

“Reinforcement Learning methods [2] can use demonstrations to bootstrap the agent’s 76 policy [22, 23] or shape its reward function [24, 25].”

“67 Behavioural Cloning methods [13, 4, 14, 5, 7, 6, 1, 15, 16] learn a policy network directly from 68 human demonstrations.”

“Meta Learning methods 71 [17, 18, 19, 20, 21] require one or few demonstrations to learn to solve a new task.”

“This is in strong contrast with other approaches, like reinforcement learning-based methods, that require hundreds or thousands of resets [2], or classic behavioural cloning methods, in which the human operator needs to provide tens of demonstrations [5, 8], resetting the environment each time.”

“Meta Learning methods 71 [17, 18, 19, 20, 21] require one or few demonstrations to learn to solve a new task.”

“Manipulation Tasks We designed a set of 8 tasks, inspired by the recent imitation learning literature 257 [3, 13, 4, 5, 7, 8], and by everyday tasks to test our method.”

“To solve 80 multi-stage tasks, the recent literature has also proposed several methods that combine ideas from 81 different fields: auto-regressive sequence modelling [26], neural task programming [27], exploiting 82 spatial symmetries [28], hierarchical decomposition [1], explicit task planning [29], etc.”

“When dealing with multi-stage tasks, the amount of demonstrations often increases considerably [1, 14].”

“To solve 80 multi-stage tasks, the recent literature has also proposed several methods that combine ideas from 81 different fields: auto-regressive sequence modelling [26], neural task programming [27], exploiting 82 spatial symmetries [28], hierarchical decomposition [1], explicit task planning [29], etc.”

“Behavioural Cloning: we implemented two state-of-the-art methods from the recent literature to learn multi-stage tasks from demonstrations: Latent Motor Plans (LMP) [14] and Relay Imitation Learning (RIL) [1].”

“67 Behavioural Cloning methods [13, 4, 14, 5, 7, 6, 1, 15, 16] learn a policy network directly from 68 human demonstrations.”

“Following this framework, introduced in [3] as Figure 2: An example of the training steps for a two-stage task: picking a cup and placing it on a plate.”

“Manipulation Tasks We designed a set of 8 tasks, inspired by the recent imitation learning literature 257 [3, 13, 4, 5, 7, 8], and by everyday tasks to test our method.”

“We show how Behavioural Cloning, extended to tackle multi-stage tasks with the Relay Imitation Learning method, quickly starts to degrade in performance when the number of stages increases, while our method always obtains a large number of successes.”

“Meta Learning methods 71 [17, 18, 19, 20, 21] require one or few demonstrations to learn to solve a new task.”

“However, they 72 need large amounts of data collected before hand on similar tasks, another laborious process, and the 73 amount of previously collected data also increases when tackling multi-stage tasks [21].”

“To solve 80 multi-stage tasks, the recent literature has also proposed several methods that combine ideas from 81 different fields: auto-regressive sequence modelling [26], neural task programming [27], exploiting 82 spatial symmetries [28], hierarchical decomposition [1], explicit task planning [29], etc.”

“These methods are FlowControl [30], a recent state-of-the-art technique that used a learned optical-flow model to compute how to align the end-effector, from its current observation, to the bottleneck pose.”

“To solve 80 multi-stage tasks, the recent literature has also proposed several methods that combine ideas from 81 different fields: auto-regressive sequence modelling [26], neural task programming [27], exploiting 82 spatial symmetries [28], hierarchical decomposition [1], explicit task planning [29], etc.”

“Behavioural Cloning methods [13, 4, 14, 5, 7, 6, 1, 15, 16] learn a policy network directly from human demonstrations.”

“It has been successfully applied to RL [16], sim-to-real control [25], and multi-stage IL [7].”

“…with 49 everyday objects (20 train, 29 test) and 8 types of tasks: grasping , pouring , opening , inserting , sweeping , stacking , assembling , flipping and 3 multi-stage tasks following the demonstration procedure in [35]: open+pick+place , hanging cups and opening and loading a dishwasher .”

“…applied in imitation learning to obtain ground-truth action labels for image-based policies by manually resetting the robot to a goal or known configuration, perturbing the end effector by a known displacement, and using the displacement with the initial pose to compute an action label [8, 25, 45].”

“While the latter has been explored in [5], [9], [14], the combination of retrieval and decomposition is distinctly novel, and leads to a general and scalable framework that can continue to improve with further data.”

“As in Di Palo et al. [11], we assume that tasks consist of a sequence of object-centric subtasks ( S 1 ( o S 1 ) , S 2 ( o S 2 ) , ..., S M ( o S M )) , where the manipulation in each subtask S i ( o S i ) is relative to a single object’s ( o S i ∈ O ) coordinate frame.”

“As real-world tasks involve multi-stage nature consisting of a sequence of tasks and requiring long-horizon planning and control, reinforcement learning researchers have designed multi-stage environments to address these challenges [17], [18], [19], [20], [21], [22], [23], [24], [25].”

“The use of learned optical flow for servoing is similar to [10], [11], [12], and the use of demonstration sequence tracking is similar to [13], [6], [14], [15].”

“Re-thinking evaluation in the real-world, outside of simulation environments, has inspired our own recent work on imitation learning: from just a single human demonstration, with just a single environment reset, and without any prior object knowledge [20, 21].”