The DOI matches its DataCite record.
On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation
Open accessAbstract
Imitation learning with visual observations is notoriously inefficient when addressed with end-to-end behavioural cloning methods. In this paper, we explore an alternative paradigm which decomposes reasoning into three phases. First, a retrieval phase, which informs the robot what it can do with an object. Second, an alignment phase, which informs the robot where to interact with the object. And third, a replay phase, which informs the robot how to interact with the object. Through a series of real-world experiments on everyday tasks, such as grasping, pouring, and inserting objects, we show that this decomposition brings unprecedented learning efficiency, and effective inter- and intra-class generalisation. Videos are available at https://www.robot-learning.uk/retrieval-alignment-replay.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 18
- Shir Amir et al. Deep vit features as dense visual descriptors. arXiv:2112.05814, 2021. · arXiv
- Jessica Borja-Diaz et al. Affordance learning from play for sample-efficient policy learning. arXiv:2203.00352, 2022. · arXiv
- Mathilde Caron et al. Emerging properties in self- supervised vision transformers. In ICCV, 2021.
- Open X-Embodiment Collaboration. Open X- Embodiment: Robotic learning datasets and RT-X models. arXiv:2310.08864, 2023. · arXiv
- Norman Di Palo and Edward Johns. Learning multi-stage tasks with one demonstration via self-replay. In CoRL. PMLR, 2021.
- Alexey Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
- Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. Behavior retrieval: Few-shot imitation learn- ing by querying unlabeled datasets. arXiv preprint arXiv:2304.08742, 2023. · arXiv
- Kaiming He et al. Deep residual learning for image recognition. In CVPR, 2016.
- Edward Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In ICRA, 2021.
- Michelle A Lee et al. Guided uncertainty-aware pol- icy optimization: Combining learning and model-based strategies for sample-efficient policy learning. In ICRA. IEEE, 2020.
- Suraj Nair et al. R3m: A universal visual representation for robot manipulation. arXiv:2203.12601, 2022. · arXiv
- Jyothish Pari et al. The surprising effective- ness of representation learning for visual imitation. arXiv:2112.01511, 2021. · arXiv
- Alec Radford et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
- Eugene Valassakis et al. Demonstrate once, imitate immediately (dome): Learning visual servoing for one- shot imitation learning. arXiv:2204.02863, 2022. · arXiv
- Pietro Vitiello, Kamil Dreczkowski, and Edward Johns. One-shot imitation learning: A pose estimation perspec- tive. In CoRL, 2023.
- Vitalis Vosylius and Edward Johns. Few-shot in-context imitation learning via implicit graph alignment. In CoRL, 2023.
- Vitalis Vosylius and Edward Johns. Where to start? transferring simple skills to complex environments. In CoRL, pages 471-481. PMLR, 2023.
- Bowen Wen et al. You only demonstrate once: Category- level manipulation from single visual demonstration. arXiv:2201.12716, 2022. · arXiv
Metadata
- Authors
- Di Palo, N., & Johns, E.
- Affiliations
- Imperial College London (GB)
- ORCID
- Johns, Edward
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Volume
- 9
- Issue
- 3
- Pages
- 2032-2039
- ISSN
- 2377-3766 (ISSN-L)
- arXiv
- 2312.12345
- OpenAlex
- W4390578009
- Cited by
- 12
- References
- 0
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 3
- FWCI
- 1.88
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Royal Academy of Engineering
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- imitation learning, visual observation, object retrieval, alignment, replay, manipulation, learning efficiency, generalization, grasping, pouring, object insertion, human-robot interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
