Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning
Open accessAbstract
We present DOME, a novel method for one-shot imitation learning, where a task can be learned from just a single demonstration and then be deployed immediately, without any further data collection or training. DOME does not require prior task or object knowledge, and can perform the task in novel object configurations and with distractors. At its core, DOME uses an image-conditioned object segmentation network followed by a learned visual servoing network, to move the robot's end-effector to the same relative pose to the object as during the demonstration, after which the task can be completed by replaying the demonstration's end-effector velocities. We show that DOME achieves near 100% success rate on 7 real-world everyday tasks, and we perform several studies to thoroughly understand each individual component of DOME. Videos and supplementary material are available at: https://www.robot-learning.uk/dome.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several recent works propose imitation learning methods that replay previous human demonstrations.
“In fact, several recent works build on this intuition and propose imitation learning methods that replay previous human demonstrations [8-11].”
From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [10]
Replay-based imitation learning methods are promising for learning manipulation tasks using a handful of demonstrations.
“Replay-based imitation learning methods are promising for learning manipulation tasks using a handful of demonstrations [8-11, 46-48]”
From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [10]
Several recent works learn policies using only a handful of demonstrations by replaying demonstrations in new scenes.
“Several recent works learn policies using only a handful of demonstrations by replaying the demonstrations in new scenes [8-11, 46-48].”
From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 1× · see citing works
“In order to account for the reality gap [24], we randomise various simulated environment conditions, such as (1) light parameters including the number, colour, location and strength of light sources, (2) the number of distractor objects present in the live images, and (3) object colour and texture parameters.”
MethodFilm: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490 · see citing works
Learning Multi-Stage Tasks with One Demonstration via Self-Replay
Di Palo, N., & Johns, E. (2021) · arXiv · doi:10.48550/arxiv.2111.07447 · Cited by 8 · cited 2× · see citing works
“It has been successfully applied to RL [16], sim-to-real control [25], and multi-stage IL [7].”
BackgroundMethodSelf-citation
Flagged first, then by citations
Research works printed but not linked: 23
- M. Argus et al. Flowcontrol: Optical flow based visual servoing. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
- C.-Y. Chai et al. Multi-step pick-and-place tasks using object-centric dense correspondences. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.
- A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, et al. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], 2015. · arXiv
- M. Denninger et al. Blenderproc. arXiv preprint arXiv:1911.01911, 2019. · arXiv
- M. Denninger et al. Blenderproc: Reducing the reality gap with photorealistic rendering. In International Conference on Robotics: Sciene and Systems (RSS), 2020.
- Y. o. Duan. One-shot imitation learning. Advances in Neural Information Processing Systems, 2017.
- C. Finn et al. One-shot visual imitation learning via meta-learning. In Conference on Robot Learning (CoRL), 2017.
- S. Fujimoto et al. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning (ICML), 2018.
- A. Ganapathi et al. Learning dense visual correspondences in simulation to smooth and fold real fabrics. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
- S. Hutchinson and F. Chaumette. Visual servo control, part i: Basic approaches. IEEE Robotics and Automation Magazine, 2006.
- E. Jang et al. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning (CoRL), 2021.
- T. Johannink et al. Residual reinforcement learning for robot control. In IEEE International Conference on Robotics and Automation (ICRA), 2019.
- E. Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
- M. A. Lee et al. Guided uncertainty-aware policy optimization: Combining learning and model-based strategies for sample-efficient policy learning. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
- A. Y. Ng and S. Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2000.
- D. A. Pomerleau. ALVINN: An autonomous land vehicle in a neural network. In Advances in Neural Information Processing Systems, 1988.
- G. Schoettler et al. Deep reinforcement learning for industrial insertion tasks with visual inputs and natural reward signals. In International Conference on Machine Learning (ICML), 2019.
- E. Valassakis et al. Coarse-to-fine for sim-to-real: Sub-millimetre pre- cision across wide task spaces. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
- M. Vecerik et al. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817, 2017. · arXiv
- Z. Wu et al. 3d shapenets: A deep representation for volumetric shapes. In IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2015.
- C. Yu et al. Siamese convolutional neural network for sub-millimeter- accurate camera pose estimation and visual servoing. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.
- T. Zhang et al. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. IEEE International Conference on Robotics and Automation (ICRA), 2018.
- B. Ziebart et al. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial intelligence, 2008.
Cited by
Sorted by citations
On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation
Palo, N. D., & Johns, E. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3349832 · Cited by 11 · see citing works
“We compare the overall performance of all methods, and how each scales with more data, in order to more closely study the individual contributions of the retrieval component, and the decomposition component (into alignment and replay).”
BackgroundMethodSelf-citationMimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., & Fox, D. (2023) · arXiv · doi:10.48550/arxiv.2310.17596 · Cited by 5 · see citing works
- Authors
- Valassakis, E., Papagiannis, G., Di Palo, N., & Johns, E.
- Affiliations
- Imperial College London (GB)
- Imperial College London,The Robot Learning Lab
- ORCID
- Johns, Edward
- Venue
- 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2022-10-23
- Pages
- 8614-8621
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W4312297897
- Cited by
- 32
- References
- 5
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 12 · 2023: 5 · 2022: 2
- FWCI
- 9.92
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Royal Academy of Engineering
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- visual servoing, robotic manipulation, end-effector pose estimation, distractors, everyday tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
