On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation
Abstract
Imitation learning with visual observations is notoriously inefficient when addressed with end-to-end behavioural cloning methods. In this letter, we explore an alternative paradigm which decomposes reasoning into three phases. First, aretrievalphase, which informs the robotwhatit can do with an object. Second, analignmentphase, which informs the robotwhereto interact with the object. And third, areplayphase, which informs the robothowto interact with the object. Through a series of real-world experiments on everyday tasks, such as grasping, pouring, and inserting objects, we show that this decomposition brings unprecedented learning efficiency, and effective inter- and intra-class generalisation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2× · see citing works
“We use ResNet-50 [8], CLIP [13], R3M [11], and ViT DINO [3].”
MethodAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 3× · see citing works
“extracted via a Vision Transformer trained with the DINO framework achieve the best accuracy, confirming recent findings from the literature [1], [3] that such features encode semantic, geometric, global and local information of the observation effectively.”
MethodCoarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
Johns, E. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9560942 · Cited by 119 · cited 8× · see citing works
“As shown in [9], this enables a task to be learned from just a single demonstration.”
BackgroundMethodInfluentialSelf-citationDeep ViT Features as Dense Visual DescriptorsPreprint
Amir, S., Gandelsman, Y., Bagon, S., & Dekel, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.05814 · Cited by 114 · see citing works
Open X-Embodiment: Robotic Learning Datasets and RT-X Models
Embodiment Collaboration, O'Neill, A., Rehman, A., Gupta, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.08864 · Cited by 111 · see citing works
Self-citationThe Surprising Effectiveness of Representation Learning for Visual Imitation
Pari, J., Shafiullah, N. M. (., Arunachalam, S. P., & Pinto, L. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.010 · Cited by 95 · cited 5× · see citing works
“The use of retrieval for few-shot learning, especially in robotics, reinforcement learning and control, has also been investigated elsewhere [7], [12].”
BackgroundMethodInfluentialYou Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration
Wen, B., Lian, W., Bekris, K., & Schaal, S. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.044 · Cited by 70 · cited 3× · see citing works
“However, most have stronger constraints in their design compared to our proposed framework: [18] needs synthetic visual data, and therefore models of the objects, to train a perception system in simulation.”
BackgroundMethodLee, M. A., Florensa, C., Tremblay, J., Ratliff, N., Garg, A., Ramos, F. T., & Fox, D. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197125 · Cited by 57 · cited 5× · see citing works
“As the original paper [10] proposes the use of Reinforcement Learning for the policy, while we use Behaviour Cloning for time efficiency, we denote this method as BC-GUAPO.”
BackgroundMethodInfluentialOne-Shot Imitation Learning: A Pose Estimation Perspective
Vitiello, P., Dreczkowski, K., & Johns, E. (2023) · Conference on Robot Learning · doi:10.48550/arxiv.2310.12077 · Cited by 36
Self-citationValassakis, E., Papagiannis, G., Di Palo, N., & Johns, E. (2022) · 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981982 · Cited by 32 · cited 2× · see citing works
“We compare the overall performance of all methods, and how each scales with more data, in order to more closely study the individual contributions of the retrieval component, and the decomposition component (into alignment and replay).”
BackgroundMethodSelf-citationAffordance Learning from Play for Sample-Efficient Policy Learning
Diaz, J. L. B., Mees, O., Kalweit, G., Hermann, L., Boedecker, J., & Burgard, W. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811889 · Cited by 30 · see citing works
Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets
Du, M., Nair, S., Sadigh, D., & Finn, C. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.011 · Cited by 22 · see citing works
Few-Shot In-Context Imitation Learning via Implicit Graph Alignment
Vosylius, V., & Johns, E. (2023) · Conference on Robot Learning · doi:10.48550/arxiv.2310.12238 · Cited by 19
Self-citationLearning Multi-Stage Tasks with One Demonstration via Self-Replay
Di Palo, N., & Johns, E. (2021) · arXiv · doi:10.48550/arxiv.2111.07447 · Cited by 8 · cited 3× · see citing works
“While the latter has been explored in [5], [9], [14], the combination of retrieval and decomposition is distinctly novel, and leads to a general and scalable framework that can continue to improve with further data.”
BackgroundSelf-citationWhere To Start? Transferring Simple Skills to Complex Environments
Vosylius, V., & Johns, E. (2022) · arXiv · doi:10.48550/arxiv.2212.06111 · Cited by 2
Self-citationR3M: A Universal Visual Representation for Robot Manipulation
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., & Gupta, A. (2022) · arXiv · doi:10.48550/arxiv.2203.12601
Flagged first, then by citations
2 of the 20 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Language Models as Zero-Shot Trajectory Generators
Kwon, T., Palo, N. D., & Johns, E. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3410155 · Cited by 59 · see citing works
“…not the main focus of our work, there have been numerous methods to ground the visual information for language-conditioned robot manipulation, such as embedding CLIP features into NeRFs [17], [18], [19], and keypoint extraction for object encoding [12] and demonstration retrieval [20], [21].”
MethodSelf-citationJu, Y., Hu, K., Zhang, G., Zhang, G., Jiang, M., & Xu, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72940-9_13 · Cited by 42 · see citing works
DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models
Palo, N. D., & Johns, E. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610923 · Cited by 30 · see citing works
Self-citationLearning Precise Affordances From Egocentric Videos for Robotic Manipulation
Li, G., Tsagkas, N., Song, J., Mon-Williams, R., Vijayakumar, S., Shao, K., & Sevilla-Lara, L. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00985 · Cited by 6 · see citing works
“This strategy prevents the image feature from being excessively affected by the depth feature, making the training process more stable.”
PALM: Enhanced Generalizability for Local Visuomotor Policies via Perception Alignment
Wang, R., Zhuang, Z., Kragić, D., & Pokorny, F. T. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3664620 · Cited by 1 · see citing works
“Coarse-to-fine manipulation meth-ods [21], [24], [25] rely on trained controllers to move the end-effector near objects and perform state-based replay, but they often struggle on long-horizon tasks.”
Deep networks for few-shot manipulation learning from scratch
Chen, Y., Yao, X., & Tripp, B. (2025) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2025.105056
Sample-efficient multi-modal grasp learning from demonstrationPreprint
Przybylski, M., Klimaszewski, J., & Kuipers, B. J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5554495
Task-Critical Part Alignment Improves Imitation of Object Interactions
Hadjivelichkov, D., Aegidius, S., Modugno, V., et al. (2025) · doi:10.1109/icar65334.2025.11338707
“Norman Di Palo et al. [21] further extend this family of methods to novel object instances – this is done by first retrieving a relevant demonstration and then aligning with the demonstrated manipulandum with the observed one using a trained alignment policy.”
Weakly-Supervised Learning of Dense Functional Correspondences
Stojanov, S., Zhao, L., Zhang, Y., Yamins, D. L. K., & Wu, J. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00656
“Powerful correspondence representations have been found to emerge [1, 67] in DINO [3, 54] and Stable Diffusion [62], which have led to direct applications in low-shot affordances [29] and object manipulation [12, 29, 35, 55, 56].”
Tactile Memory With Soft Robot: Robust Object Insertion via Masked Encoding and Soft Wrist
Kamijo, T., Nishimura, M., Shibasaki, N., et al. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3692097
“While reusing prior experiences has shown substantial potential for robot manipulation [9], [10], [11], the challenge lies Fig.”
ReMoBot: Retrieval-Based Few-Shot Imitation Learning for Mobile Manipulation
Zhang, Y., Yang, W., Verdoja, F., et al. (2026) · IEEE/ASME International Conference on Advanced Intelligent Mechatronics · doi:10.1109/aim65483.2026.11658109
“Some methods directly execute retrieved behaviors via learned embeddings [8], [21], flow-based alignment [22], affordance reasoning [23], or foundation models [7], while others integrate retrieval into policy optimization [24]–[26].”
- Authors
- Palo, N. D., & Johns, E.
- Affiliations
- Imperial College London (GB)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2024-03
- Volume
- 9
- Issue
- 3
- Pages
- 2032-2039
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W4390578009
- Semantic Scholar
- 768aa61cca4bba52b8dd1b8319fccb8e556357ba
- Cited by
- 11
- References
- 20
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 3
- FWCI
- 1.80
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- Funders
- Royal Academy of Engineering
- Research Fellowship Scheme
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- imitation learning, visual observation, object retrieval, alignment, replay, manipulation, learning efficiency, generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
