Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition

“• Observation Encoder : All approaches encode images using a ResNet18 [57] architecture.”

“We evaluate on the proprietary GPT3 [66] (175B text-davinci-003) and the open LLAMA2 [67] (7B and 13B).”

“Our work presents the novel formulation of bringing diffusion-based [49, 50] policies [12] into the language-conditioned [51, 52] visuomotor domain.”

“This policy takes as input a task description CLIP [56] feature, proprioception history, and visual observations, and outputs a sequence of end effector control commands.”

“Our work presents the novel formulation of bringing diffusion-based [49, 50] policies [12] into the language-conditioned [51, 52] visuomotor domain.”

“…aspects: First, how to scale up the data collection for a diverse range of manipulation skills, which involves efforts such as improving the hardware [1, 2] and software [3, 4] which support demonstration collection, utilization of non-robotics datasets [5, 6], or trial-and-error explorations [7].”

“Although the original architecture was designed with an average pooling layer, its typical for robotic policies to use a spatial softmax pooling [44] layer instead.”

“Our work presents the novel formulation of bringing diffusion-based [49, 50] policies [12] into the language-conditioned [51, 52] visuomotor domain.”

“We evaluate on the proprietary GPT3 [66] (175B text-davinci-003) and the open LLAMA2 [67] (7B and 13B).”

“Our work presents the novel formulation of bringing diffusion-based [49, 50] policies [12] into the language-conditioned [51, 52] visuomotor domain.”

“When used in conjunction with the DDIM [58] noise scheduler, we found that we could use a 10 × shorter diffusion process at inference (5 timesteps at inference, 50 timesteps at training) while retaining a comparable performance.”

“Human egocentric videos [6, 23, 24] has also been shown to be relevant to robot learning [5, 25], but is not robot-complete (lacks action labels), and requires cross-embodiment transfer.”

“Namely, actions should not require human demonstrations to acquire [9, 10, 13–16 • Language-friendly.”

“The benchmark is built on top of the Mujoco [3] simulator, using assets from the Google Scanned dataset [53, 54].”

“We then distill only successful trajectories into an observable-information [41–43] policy, allowing the distilled policy to improve upon its LLM data collection policy’s performance.”

“Success-filtering [11, 53] can be viewed as the simplest form of offline RL [54].”

“Planar actions [10, 37] aren’t flexible enough to manipulate prismatic and revolute joints.”

“Thus, we introduce the following baselines, which build on top of Code-as-Policy and each other as follows: • LLM-as-Policy (2D) : Similar to code-as-policy using planar pick-and-place, but we use ground truth object segmentation instead of their off-the-shelf object detectors [61, 62].”

“When providing action labels, humans can either provide task-specific [9, 10, 15, 18], or task-agnostic (“play”) data [13, 14, 16, 19].”

“Success-filtering [11, 53] can be viewed as the simplest form of offline RL [54].”

“…aspects: First, how to scale up the data collection for a diverse range of manipulation skills, which involves efforts such as improving the hardware [1, 2] and software [3, 4] which support demonstration collection, utilization of non-robotics datasets [5, 6], or trial-and-error explorations [7].”

“…aspects: First, how to scale up the data collection for a diverse range of manipulation skills, which involves efforts such as improving the hardware [1, 2] and software [3, 4] which support demonstration collection, utilization of non-robotics datasets [5, 6], or trial-and-error explorations [7].”

“To enable effective learning of high entropy, diverse robot trajectories, we extend the diffusion policy [12] to handle language-based conditioning for multi-task learning.”

“However, in contrast to recent works which use LLMs as the final policy [34–40], we use the LLM-based planner as a suboptimal data-collection policy.”

“• Action Space : Besides our absolute end effector pose action space, Delta-Action and velocity control spaces is another popular action space choice [4, 15, 63–65].”

“Thus, we introduce the following baselines, which build on top of Code-as-Policy and each other as follows: • LLM-as-Policy (2D) : Similar to code-as-policy using planar pick-and-place, but we use ground truth object segmentation instead of their off-the-shelf object detectors [61, 62].”

“Another related question has been how to best train multi-task policies [47, 48], including those conditioned on language [9, 10, 13, 15, 16, 18].”

“Another option for the autonomous data collection policy is to use a model-based policy, e . g . task and motion planning (TAMP) [33].”

“Towards unsupervised exploration, prior works have also investigated evolving environments [26, 27] and embodiments [28], automatic task generation [29], leveraging language guidance [30, 31] and world-model error [32], but have not been demonstrated to scale to 6 DoF robotic skill learning.”

“…aspects: First, how to scale up the data collection for a diverse range of manipulation skills, which involves efforts such as improving the hardware [1, 2] and software [3, 4] which support demonstration collection, utilization of non-robotics datasets [5, 6], or trial-and-error explorations [7].”

“However, in contrast to recent works which use LLMs as the final policy [34–40], we use the LLM-based planner as a suboptimal data-collection policy.”

“Human egocentric videos [6, 23, 24] has also been shown to be relevant to robot learning [5, 25], but is not robot-complete (lacks action labels), and requires cross-embodiment transfer.”

“Planar actions [10, 37] aren’t flexible enough to manipulate prismatic and revolute joints.”

“Within this category, humans can either provide episodic task-specific expert demonstrations [13, 14, 19, 22], or non-task-specific (“play”) data [17, 18, 20, 23].”

“…aspects: First, how to scale up the data collection for a diverse range of manipulation skills, which involves efforts such as improving the hardware [1, 2] and software [3, 4] which support demonstration collection, utilization of non-robotics datasets [5, 6], or trial-and-error explorations [7].”

“• Action Space : Besides our absolute end effector pose action space, Delta-Action and velocity control spaces is another popular action space choice [4, 15, 63–65].”

“Towards unsupervised exploration, prior works have also investigated evolving environments [26, 27] and embodiments [28], automatic task generation [29], leveraging language guidance [30, 31] and world-model error [32], but have not been demonstrated to scale to 6 DoF robotic skill learning.”

“Attempts in the field have focused on two primary aspects: First, how to scale up the data collection for a diverse range of manipulation skills, which involves efforts such as improving the hardware [1, 2] and software [3, 4] which support demonstration collection, utilization of non-robotics datasets [5, 6], or tria…”

“The second aspect of this question concerns effective learning from the collected data, which delves into exploring effective action representations [8–10] and policy formulations [11, 12] that can robustly model the training data and generalize to novel scenarios.”

“When providing action labels, humans can either provide task-specific [9, 10, 15, 18], or task-agnostic (“play”) data [13, 14, 16, 19].”

“Namely, actions should not require human demonstrations to acquire [9, 10, 13–16 • Language-friendly.”

“To scale language annotation, prior works study using visual-language models [20, 21], or procedurally post-hoc provided in simulation [19].”

“Another related question has been how to best train multi-task policies [47, 48], including those conditioned on language [9, 10, 13, 15, 16, 18].”

“When providing action labels, humans can either provide task-specific [9, 10, 15, 18], or task-agnostic (“play”) data [13, 14, 16, 19].”

“• Action Space : Besides our absolute end effector pose action space, Delta-Action and velocity control spaces is another popular action space choice [4, 15, 63–65].”

“The second aspect of this question concerns effective learning from the collected data, which delves into exploring effective action representations [8–10] and policy formulations [11, 12] that can robustly model the training data and generalize to novel scenarios.”

“Following Robomimic [4]’s findings, we use a wrist-mounted view in addition to a global (workspace) view to help with tasks requiring precise manipulation.”

“Towards unsupervised exploration, prior works have also investigated evolving environments [26, 27] and embodiments [28], automatic task generation [29], leveraging language guidance [30, 31] and world-model error [32], but have not been demonstrated to scale to 6 DoF robotic skill learning.”

“To scale action labels, methods study how to use autonomous sub-optimal policies from random [7] to learned [22] policies.”

“As another option for scalability not depending on robots, human ego-vision data [6, 26, 27] has been shown to be relevant to robot learning [5, 28], but lacks “action” (motor) labels, and requires cross-embodiment transfer.”

“The second aspect of this question concerns effective learning from the offline data [8], which delves into dataset filtering [9, 10] and exploring effective observation [11], action [12–15], and policy [9, 16] formulations that can robustly model the training data and generalize to novel scenarios.”

“Towards unsupervised exploration, prior works have also investigated evolving environments [26, 27] and embodiments [28], automatic task generation [29], leveraging language guidance [30, 31] and world-model error [32], but have not been demonstrated to scale to 6 DoF robotic skill learning.”

“Towards unsupervised exploration, prior works have also investigated evolving environments [26, 27] and embodiments [28], automatic task generation [29], leveraging language guidance [30, 31] and world-model error [32], but have not been demonstrated to scale to 6 DoF robotic skill learning.”

“Towards unsupervised exploration, prior works have also investigated evolving environments [26, 27] and embodiments [28], automatic task generation [29], leveraging language guidance [30, 31] and world-model error [32], but have not been demonstrated to scale to 6 DoF robotic skill learning.”

“To scale language annotation, prior works study using visual-language models [20, 21], or procedurally post-hoc provided in simulation [19].”

“Towards unsupervised exploration, prior works have also investigated evolving environments [26, 27] and embodiments [28], automatic task generation [29], leveraging language guidance [30, 31] and world-model error [32], but have not been demonstrated to scale to 6 DoF robotic skill learning.”

“…aspects: First, how to scale up the data collection for a diverse range of manipulation skills, which involves efforts such as improving the hardware [1, 2] and software [3, 4] which support demonstration collection, utilization of non-robotics datasets [5, 6], or trial-and-error explorations [7].”

“Image-based imitation learning tends to learn implicit spatial representations from fixed camera views [1, 4, 9, 15, 47, 57].”

“…of sequences of micro-actions represented as end-effector positions, generally encompassing translation and rotation depending on end-effector actuation (Chi et al., 2023; Ze et al., 2024; Xu et al., 2023; Li X. et al., 2024; Si et al., 2024; Scheikl et al., 2024; Ke et al., 2024; Ha et al., 2023).”

“Learning language-guided grasping skills has garnered significant interest due to its potential to enhance the adaptability and flexibility of robot systems in interactive operating scenarios [2], [3], [4], [5].”

“LLMs were used to generate task curricula and predict skills to execute in Minecraft worlds [32], [33], [34] Following the seminal work of Code as Policies, many works map language to programs over given skills [35] or hand-designed motion planners [36].”

“Approaches include leveraging large language models [24] or hardware 16363 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”

“The progress in Large Language Model (LLM) offers a promising solution for DSIL [208].”

“Lastly, linguistic guidance enriches robotic systems with semantic information [22], enhancing their learning capabilities without necessitating expert demonstrations or specific engineering [23].”

“The concurrent work [58] utilizes an LLM to generate synthetic experiences in simulation together with success conditions, and distills the successful trials into a language-conditioned policy with behavioral cloning.”

“A concurrent letter [9] also distilled robot skills from LLMs, but we simultaneously distilled knowledge from both LLMs and VLMs, focusing on more fine-grained (i.e., coordinate-level) functional rearrangement priors.”

“Text2Reward [239] Eureka [240] ASD [241] Provide reward signal ALF [242] UVD [243] ROBOFUME [244] MOKA [245] RLFP [246] Reward function estimation Non-parametric estimation MWM [247] VoxPoser [248] LIV [249] RL-VLM-F [250] CenterGrasp [251] LAMP [252] SARU [253] FuRL [254] VIPER [255] Diffusion…”

“…action distributions, several researches has demonstrated the advantages using diffusion-based policies, i.e., goal-directed navigation for mobile robots [20], the integration of human behavior in collaborative human-robot tasks [21], and language-guided robot skill learning activities [22].”

“The language-guided robot skill acquisition approach [24] initially decomposes the task description recursively into subtasks, represented as hierarchical plans resembling a task tree.”

“Some recent efforts have used Large Language Models to generate datasets in simulation [45, 12], but it can still involve carefully engineered pipelines.”

“Baselines: Our method is evaluated in comparison with the following baselines: (1) LLM, which is the language-driven robot data generation framework in ScalingUp [7] .”

“Especially, robot manipulators and arms [3-6] that can mimic human actions such as opening a cabinet [7-9], assembling objects [10,11], and picking and placing items [12] are being researched increasingly.”

“The study in [29] uses a Large Language Model (LLM) and diffusion policy [30] to generate manipulation trajectories.”