Interactive Language: Talking to Robots in Real Time
Open accessAbstract
We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuo-linguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. "make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several related efforts also collect large datasets (e.g. 1000s of demonstrations) by using a large number of human operators over extended periods of time.
“Several related efforts [3-6, 27] also collect large datasets (e.g. 1000s of demonstrations) by using a large number of human operators over extended periods of time.”
From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 15
- M. Shridhar, L. Manuelli, and D. Fox, "Cliport: What and where pathways for robotic manipulation," in Conference on Robot Learning. PMLR, 2022, pp. 894-906.
- S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn et al., "Learning language- conditioned robot behavior from offline data and crowd-sourced annotation," in Conference on Robot Learning. PMLR, 2022, pp. 1303-1315.
- E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, "Bc-z: Zero-shot task generalization with robotic imitation learning," in Conference on Robot Learning. PMLR, 2022, pp. 991-1002.
- S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies," The Journal of Machine Learning Research, vol. 17, no. 1, pp. 1334-1373, 2016.
- Y. Ding, C. Florensa, P. Abbeel, and M. Phielipp, "Goal-conditioned imitation learning," Advances in neural information processing systems, vol. 32, 2019.
- C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, "Learning latent plans from play," in Conference on robot learning. PMLR, 2020, pp. 1113-1132.
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learning from human preferences," Advances in neural information processing systems, vol. 30, 2017.
- A. Broad, J. Arkin, N. Ratliff, T. Howard, B. Argall, and D. C. Graph, "Towards real-time natural language corrections for assistive robots," in RSS Workshop on Model Learning for Human-Robot Communication, 2016.
- S. Karamcheti, M. Srivastava, P. Liang, and D. Sadigh, "Lila: Language- informed latent actions," in Conference on Robot Learning. PMLR, 2022, pp. 1379-1390.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervision," in International Conference on Machine Learning. PMLR, 2021, pp. 8748-8763.
- R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu, "On layer normalization in the transformer architecture," in International Conference on Machine Learning. PMLR, 2020, pp. 10 524-10 533.
- T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, "Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation," in IEEE International Conference on Robotics and Automation (ICRA), 2018.
- B. Wu, S. Nair, F.-F. Li, and C. Finn, "Example-driven model-based reinforcement learning for solving long-horizon visuomotor tasks," in 5th Annual Conference on Robot Learning, 2021. [Online]. Available: https://openreview.net/forum?id= daq0uh6yXr
- B. Hayes and B. Scassellati, "Challenges in shared-environment human-robot collaboration," learning, vol. 8, no. 9, 2013.
Cited by
Sorted by citations
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276 · see citing works
Self-citationDeep Learning Approaches to Grasp Synthesis: A Review
Newbury, R., Gu, M., Chumbley, L., Mousavian, A., Eppner, C., Leitner, J., Bohg, J., Morales, A., Asfour, T., Kragić, D., et al. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3280597 · Cited by 273 · see citing works
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · see citing works
Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., Deane, G., Fleming, S. M., Frith, C., Ji, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08708 · Cited by 196 · see citing works
O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611477 · Cited by 210 · see citing works
Self-citationRoCo: Dialectic Multi-Robot Collaboration with Large Language Models
Mandi, Z., Jain, S., & Song, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610855 · Cited by 206 · see citing works
Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174 · see citing works
A review of visual SLAM for robotics: evolution, properties, and future applications
Al-Tawil, B., Hempel, T., Abdelrahman, A. A., & Al-Hamadi, A. K. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1347985 · Cited by 92 · see citing works
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Lu, J., Clark, C., Lee, S.-H., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., & Kembhavi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02497 · Cited by 98 · see citing works
Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
Kawaharazuka, K., Oh, J., Yamada, J., Posner, I., & Zhu, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3609980 · Cited by 95 · see citing works
PIRLNav: Pretraining with Imitation and RL Finetuning for OBJECTNAV
Ramrakhya, R., Batra, D., Wijmans, E., & Das, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01716 · Cited by 71 · see citing works
RT-H: Action Hierarchies Using Language
Belkhale, S., Ding, T., Xiao, T., Sermanet, P., Vuong, Q., Tompson, J., Chebotar, Y., Dwibedi, D., & Sadigh, D. (2024) · arXiv · doi:10.48550/arxiv.2403.01823 · Cited by 71
“Data driven approaches have also been explored, requiring large datasets of language corrections [49– 52] or shared autonomy during deployment [2].”
MethodSelf-citationChat with the Environment: Interactive Multimodal Perception Using Large Language Models
Zhao, X., Li, M., Weber, C., Hafez, M. B., & Wermter, S. (2023) · 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342363 · Cited by 62 · see citing works
Frering, L., Steinbauer-Wagner, G., & Holzinger, A. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109771 · Cited by 55 · see citing works
From screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 59 · see citing works
RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Sermanet, P., Ding, T., Zhao, J. W., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac-Arnold, G., Maddineni, S., Joshi, N. J., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610216 · Cited by 49 · see citing works
Self-citationOpenVLA: An Open-Source Vision-Language-Action Model
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09246 · Cited by 62 · see citing works
“…Play [95] 0.4% Berkeley Cable Routing [96] 0.2% Roboturk [97] 2.3% Viola [98] 0.9% Berkeley Autolab UR5 [99] 1.2% Toto [100] 2.0% Language Table [101] 4.4% Stanford Hydra Dataset [102] 4.4% Austin Buds Dataset [103] 0.2% NYU Franka Play Dataset [104] 0.8% Furniture Bench Dataset [105] 2.4% UCSD…”
MethodInfluentialRaptis, E. K., Kapoutsis, A. C., & Kosmatopoulos, E. B. (2025) · Frontiers in Robotics and AI · doi:10.3389/frobt.2025.1605405 · Cited by 47 · see citing works
Jiang, Y., Fan, L., Sun, X., Luo, Z., Wang, H., Lai, R., Wang, J., Gan, Q., Li, N., & Tian, J. (2024) · Advanced Science · doi:10.1002/advs.202402705 · Cited by 42 · see citing works
Robot learning in the era of foundation models: a survey
Xiao, X., Liu, J., Wang, Z., Zhou, Y., Qi, Y., Jiang, S., He, B., & Cheng, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129963 · Cited by 47 · see citing works
Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections
Zha, L., Cui, Y., Lin, L.-H., Kwon, M., Arenas, M. G., Zeng, A., Xia, F., & Sadigh, D. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610455 · Cited by 36 · see citing works
DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment
Guo, Y. F., Wang, Y., Zha, L., & Chen, J. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802284 · Cited by 37 · see citing works
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models
Xiao, T. G., Chan, H., Sermanet, P., Wahid, A., Brohan, A., Hausman, K., Levine, S., & Tompson, J. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.029 · Cited by 42 · see citing works
Self-citationWu, D., Zheng, P., Zhao, Q., Zhang, S., Qi, J., Hu, J., Zhu, G., & Wang, L. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103064 · Cited by 35 · see citing works
RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins
Mu, Y., Chen, T., Chen, Z., Peng, S., Lan, Z., Gao, Z., Liang, Z., Yu, Q., Zou, Y., Xu, M., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02575 · Cited by 34 · see citing works
AgentLens: Visual Analysis for Agent Behaviors in LLM-Based Autonomous Systems
Lu, J., Pan, B., Chen, J., Feng, Y., Hu, J., Peng, Y., & Chen, W. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3394053 · Cited by 31 · see citing works
StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects
Liu, W., Du, Y., Hermans, T., Chernova, S., & Paxton, C. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.031 · Cited by 36 · see citing works
Gong, R., Huang, J., Zhao, Y., Geng, H., Gao, X., Wu, Q., Ai, W., Zhou, Z., Terzopoulos, D., Zhu, S. C., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01873 · Cited by 31 · see citing works
GRID: Scene-Graph-based Instruction-driven Robotic Task Planning
Ni, Z., Deng, X., Tai, C., Zhu, X., Xie, Q., Huang, W., Wu, X., & Zeng, L. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801291 · Cited by 33 · see citing works
Exploring Embodied Multimodal Large Models: Development, datasets, and future directions
Chen, S., Wu, Z., Zhang, K., Li, C., Zhang, B., Ma, F., Yu, F. R., & Li, Q. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103198 · Cited by 31 · see citing works
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Ha, H., Florence, P., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2307.14535 · Cited by 24 · see citing works
“When providing action labels, humans can either provide task-specific [9, 10, 15, 18], or task-agnostic (“play”) data [13, 14, 16, 19].”
BackgroundSelf-citationExploring the Robustness of Decision-Level Through Adversarial Attacks on LLM-Based Embodied Models
Liu, S., Chen, J., Ruan, S., Su, H., & Yin, Z. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680616 · Cited by 26 · see citing works
RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (Early Version)
Mu, Y., Chen, T., Peng, S., Chen, Z., Gao, Z., Zou, Y., Lin, L., Xie, Z., & Luo, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91813-1_17 · Cited by 21 · see citing works
LLM-controller: Dynamic robot control adaptation using large language models
Zahedifar, R., Baghshah, M. S., & Taheri, A. (2025) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2024.104913 · Cited by 20 · see citing works
Music in the digital age: commodity, community, communion
Cross, I. R. M. (2023) · AI & Society · doi:10.1007/s00146-023-01670-9 · Cited by 28 · see citing works
DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning
Jiang, Z., Xie, Y., Lin, K., Xu, Z., Wan, W., Mandlekar, A., Fan, L. J., & Zhu, Y. (2025) · 2025 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra55743.2025.11127809 · Cited by 24 · see citing works
Take a shot! Natural language control of intelligent robotic X-ray systems in surgery
Killeen, B. D., Chaudhary, S., Osgood, G. M., & Unberath, M. (2024) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-024-03120-3 · Cited by 19 · see citing works
An Interactive Agent Foundation Model
Durante, Z., Gong, R., Sarkar, B., Wake, N., Taori, R., Tang, P. C., Lakshmikanth, S. K., Schulman, K. A., Milstein, A. S., Vo, H., et al. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00350 · Cited by 28 · see citing works
Learning to generate pointing gestures in situated embodied conversational agents
Deichler, A., Wang, S., Alexanderson, S., & Beskow, J. (2023) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1110534 · Cited by 16 · see citing works
CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation
Wang, J., Qin, Y., Kuang, K., Korkmaz, Y., Gurumoorthy, A., Su, H., & Wang, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01700 · Cited by 16 · see citing works
Unlocking Robotic Autonomy: A Survey on the Applications of Foundation Models
Jang, D.-S., Cho, D.-H., Lee, W.-C., Ryu, S.-K., Jeong, B.-M., Hong, M., Jung, M., Kim, M., Lee, M., Lee, S., et al. (2024) · International Journal of Control Automation and Systems · doi:10.1007/s12555-024-0438-7 · Cited by 16 · see citing works
SUGAR : Pre-training 3D Visual Representations for Robotics
Chen, S., García, R., Laptev, I., & Schmid, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01709 · Cited by 15 · see citing works
Lightweight Language-driven Grasp Detection using Conditional Consistency Model
Nguyen, N., Vu, M. N., Huang, B., Vuong, A., Le, N., Vo, T., & Nguyen, A. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802007 · Cited by 15 · see citing works
Learning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 12 · see citing works
“We use datasets previously collected from these simulators, i.e., Habitat object navigation with HM3D [16] and Language Table Data from [17] to train the UniSim.”
MethodSelf-citation(2024) · doi:10.1109/icra57147.2024.10611448 · Cited by 12 · see citing works
TalkWithMachines: Enhancing Human-Robot Interaction Through Large/Vision Language Models
Abbas, A. N., & Beleznai, C. (2024) · IEEE International Conference on Robotic Computing (IRC) · doi:10.1109/irc63610.2024.00039 · Cited by 13 · see citing works
IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learning
Hoque, R., Mandlekar, A., Garrett, C. R., Goldberg, K. Y., & Fox, D. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801523 · Cited by 10 · see citing works
Transferring Foundation Models for Generalizable Robotic Manipulation
Yang, J., Tan, W., Jin, C., Yao, K., Liu, B., Fu, J., Song, R., Wu, G., & Wang, L. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00201 · Cited by 17 · see citing works
Medical & healthcare robotics: a roadmap for enhanced precision, safety, and efficacy
Iakovidis, D. K., Vartholomeos, P., Gall, A. L., Cianchetti, M., Ozioko, O., Dahiya, R., Mazomenos, E. B., Vasconcelos, F., Stoyanov, D. V., Philpott, J., et al. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/ae09bf · Cited by 9 · see citing works
Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations
Heravi, N., Wahid, A., Lynch, C., Florence, P., Armstrong, T., Tompson, J., Sermanet, P., Bohg, J., & Dwibedi, D. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160888 · Cited by 9 · see citing works
Self-citationFine-Tuning Multimodal Transformer Models for Generating Actions in Virtual and Real Environments
Staroverov, A., Gorodetsky, A., Krishtopik, A. S., Izmesteva, U. A., Yudin, D. A., Kovalev, A. K., & Panov, A. I. (2023) · IEEE Access · doi:10.1109/access.2023.3334791 · Cited by 8 · see citing works
SPRINT: Scalable Policy Pre-Training via Language Instruction Relabeling
Zhang, J. M., Pertsch, K., Zhang, J., & Lim, J. J. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610606 · Cited by 8 · see citing works
HiCRISP: An LLM-Based Hierarchical Closed-Loop Robotic Intelligent Self-Correction Planner
Ming, C., Lin, J., Fong, P., Wang, H., Duan, X., & He, J. (2024) · China Automation Congress (CAC) · doi:10.1109/cac63892.2024.10865457 · Cited by 9 · see citing works
Using Social Robotics to Identify Educational Behavior: A Survey
de Vaca, A. J. R.-C., Meléndez-Armenta, R. Á., & Ponce-Espinosa, H. (2024) · Electronics · doi:10.3390/electronics13193956 · Cited by 7 · see citing works
Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning
Yang, J., Zhu, H., Wang, Y., Wu, G., He, T., & Wang, L. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00653 · Cited by 7 · see citing works
This&That: Language-Gesture Controlled Video Generation for Robot Planning
Wang, B., Sridhar, N., Feng, C., der Merwe, M. V., Fishman, A., Fazeli, N., & Park, J. J. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11128780 · Cited by 8 · see citing works
A survey of robotic manipulation: From bottom-up approaches to end-to-end paradigms with LLMs
Peng, K., Li, Q., He, Z.-J., Zhang, B.-W., Fu, X., Li, B., Wang, X.-H., Cheng, Z.-Q., Yan, Y., & Peng, X.-J. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.132921 · Cited by 7 · see citing works
ICRT: In-Context Imitation Learning via Next-Token Prediction
Fu, M., Huang, H., Datta, G., Chen, L. Y., Panitch, W., Liu, F.-C., Li, H., & Goldberg, K. (2024) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra55743.2025.11128272 · Cited by 95 · see citing works
Advancing Sustainable Construction: Discrete Modular Systems & Robotic Assembly
Liu, Y., Belousov, B., Schneider, T., Harsono, K., Cheng, T.-W., Shih, S.-G., Tessmann, O., & Peters, J. (2024) · Sustainability · doi:10.3390/su16156678 · Cited by 6 · see citing works
IRASim: A Fine-Grained World Model for Robot Manipulation
Zhu, F., Wu, H., Guo, S., Liu, Y., Cheang, C., & Kong, T. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00917 · Cited by 6 · see citing works
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
NVIDIA, :, Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L. "., Fang, Y., Fox, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14734 · Cited by 10 · see citing works
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., & Fox, D. (2023) · arXiv · doi:10.48550/arxiv.2310.17596 · Cited by 5 · see citing works
“One popular data source is human demonstrators that teleoperate robot arms [2–6, 23– 27 ], but collecting 3 Problem Setup Imitation Learning.”
BackgroundGuo, Y., Ni, Y., Jin, J., Jiang, Y., Li, D., Zhao, H., & Shen, Y. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3590433 · Cited by 5 · see citing works
PTQ4RIS: Post-Training Quantization for Referring Image Segmentation
Jiang, X., Yang, H., Zhu, K., Qiu, X., Zhao, S., & Zhou, S. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11128841 · Cited by 7 · see citing works
Instructing Hierarchical Tasks to Robots by Verbal Commands
Telkes, P., Angleraud, A., & Pieters, R. (2024) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii58957.2024.10417491 · Cited by 6 · see citing works
Intelligent control of robotic X-ray devices using a language-promptable digital twin
Killeen, B. D., Suresh, A., Gómez, C., Íñigo, B., Bailey, C., & Unberath, M. (2025) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-025-03351-y · Cited by 5 · see citing works
The linguistic feedback of tourism robots significantly influences visitors’ ecotourism behaviors
Chang, R., Yang, D., Lyu, D., Cao, J. J., Wei, Y., & Xiao, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-01045-8 · Cited by 5 · see citing works
Language-Conditioned Offline RL for Multi-Robot Navigation
Morad, S., Shankar, A., Blumenkamp, J., & Prorok, A. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11127288 · Cited by 5 · see citing works
Language as a cognitive and social tool at the time of large language models
Borghi, A. M., Livio, C. D., Gervasi, A. M., Mannella, F., Nolfi, S., & Tummolini, L. (2024) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-024-00152-8 · Cited by 6 · see citing works
Talk Through It: End User Directed Manipulation Learning
Winge, C., Imdieke, A., Aldeeb, B., Kang, D., & Desingh, K. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3433309 · Cited by 4 · see citing works
Detect2Interact: Localizing Object Key Field in Visual Question Answering with LLMs
Wang, J., Zhu, M., Li, Y., Li, H., Yang, L., & Woo, W. L. (2024) · IEEE Intelligent Systems · doi:10.1109/mis.2024.3384513 · Cited by 4 · see citing works
Fostering Interactive Language Use: Strategies and Challenges in English Language Teaching
Yuhan, S. (2024) · Lecture Notes in Education Psychology and Public Media · doi:10.54254/2753-7048/54/20241563 · Cited by 4 · see citing works
Navigating Education 5.0 Robotic Technique for Teaching Foreign Languages in Todays Classroom
Muthmainnah, M., Apriani, E., Marzuki, A. G., Yakin, A. M. A., Aeni, N., Azar, A. S., & Riyawi, M. R. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-3041-8.ch008 · Cited by 4 · see citing works
Multi-task real-robot data with gaze attention for dual-arm fine manipulation
Kim, H., Ohmura, Y., & Kuniyoshi, Y. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802034 · Cited by 4 · see citing works
PolarNet: 3D Point Clouds for Language-Guided Robotic ManipulationPreprint
Chen, S., García, R., Schmid, C., & Laptev, I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.15596 · Cited by 7 · see citing works
Intelligent Indoor Navigation for Home Robots based on Large Language Models
Chen, Z., Su, Z., Chesser, C., Bourlon, C., & Sheng, W. (2025) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case58245.2025.11163777 · Cited by 3 · see citing works
Controllable Traffic Simulation through LLM-Guided Hierarchical Reasoning and Refinement
Liu, Z., Li, L., Wang, Y., Lin, H., Cheng, H., Liu, Z., He, L., & Wang, J. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11247747 · Cited by 3 · see citing works
Song, S., Kang, D., & Park, C.-E. (2026) · IEEE Access · doi:10.1109/access.2026.3664145 · Cited by 4 · see citing works
Minimizing learning experiences in embodied agents language and action learning
Milano, N., & Nolfi, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129510 · Cited by 3 · see citing works
Chen, Z., Shi, Z., Lu, X., He, L., Qian, S., Zhou, E., Yin, Z., Ouyang, W., Shao, J., Qiao, Y., et al. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11246228 · Cited by 2 · see citing works
Policy Learning from Large Vision-Language Model Feedback Without Reward Modeling
Luu, T. M., Lee, D.-H., Lee, Y.-H., & Yoo, C. D. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11246902 · Cited by 2 · see citing works
FlowDreamer: A RGB-D World Model With Flow-Based Motion Representations for Robot Manipulation
Guo, J., Ma, X., Wang, Y., Yang, M., Liu, H., & Li, Q. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3653273 · Cited by 2 · see citing works
World Models for Robotic Manipulation: A Survey
Wang, F., Wang, Z., Pei, G., Zhang, M., 梁燦錫, Hu, J., Li, Z., Wu, J., Han, N., Zhang, Z., et al. (2026) · SmartBot · doi:10.1002/smb2.70053 · Cited by 4 · see citing works
Did You Mean...? Confidence-based Trade-offs in Semantic Parsing
Stengel-Eskin, E., & Durme, B. V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.159 · Cited by 2 · see citing works
Real-World Robot Control Based on Contrastive Deep Active Inference With Demonstrations
Fujii, K., Isomura, T., & Murata, S. (2024) · IEEE Access · doi:10.1109/access.2024.3477306 · Cited by 2 · see citing works
Toward Universal Embodied Planning in Scalable Heterogeneous Field Robots Collaboration and Control
Wan, H., Zhang, Y., Wang, J., Wu, D., Li, M., Chen, X., Deng, Y., Huang, Y., Sun, Z., Li, Z., et al. (2025) · Journal of Field Robotics · doi:10.1002/rob.22522 · Cited by 2 · see citing works
‘What Did the Robot Do in My Absence?’ Video Foundation Models to Enhance Intermittent Supervision
Katuwandeniya, K., Tian, L., & Kulić, D. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3539118 · Cited by 2 · see citing works
Robot Behavior Personalization From Sparse User Feedback
Patel, M., & Chernova, S. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3550833 · Cited by 2 · see citing works
InteLiPlan: An Interactive Lightweight LLM-Based Planner for Domestic Robot Autonomy
Ly, K. T., Lu, K., & Havoutis, I. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3662577 · Cited by 6 · see citing works
Zero-shot Object Navigation with Vision-Language Foundation Models Reasoning
Yuan, S., Shafique, M., Baghdadi, R., Khorrami, F., Tzes, A. P., & Fang, Y. (2024) · International Conference on Automation, Robotics and Applications · doi:10.1109/icara60736.2024.10553173 · Cited by 2 · see citing works
Nishimura, T., Kuyo, K., Kambara, M., & Sugiura, K. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802596 · Cited by 2 · see citing works
Working Backwards: Learning to Place by Picking
Limoyo, O., Konar, A., Ablett, T., Kelly, J. W., Hogan, F. R., & Dudek, G. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802173 · Cited by 1 · see citing works
LiLMaps: Learnable Implicit Language Maps
Kruzhkov, E., & Behnke, S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00749 · Cited by 1 · see citing works
Double-Feedback: Enhancing Large Language Models Reasoning in Robotic Tasks by Knowledge Graphs
Wang, H., Zhang, S., Wang, S., Jiang, T., & Ge, Y. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3562776 · Cited by 2 · see citing works
Interactive Agent Foundation Model: A Meta-Learning Strategy
Dwivedi, M., & Dwivedi, A. K. (2025) · International Conference on Intelligent Systems, Advanced Computing and Communication (ISACC) · doi:10.1109/isacc65211.2025.10969383 · Cited by 1 · see citing works
Visual Instruction as an Intuitive Interface for Robotic Sorting
Müller, A. T., & Stache, N. C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93429-2_12 · Cited by 1 · see citing works
Kube, D., Hadwiger, S., & Meisen, T. (2025) · Biomimetic Intelligence and Robotics · doi:10.1016/j.birob.2025.100249 · Cited by 1 · see citing works
Development of a Mobile Assistive Robot for Daily Living Support*
Bui, T. M., Kim, Y.-O., Moon, J., Cho, M.-Y., Seo, M., & Shin, D. (2025) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/ur65550.2025.11078095 · Cited by 1 · see citing works
Joint Action Language Modelling for Transparent Policy Execution
Wulff, T., Maharjan, R. S., Chi, X., & Cangelosi, A. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229330 · Cited by 1 · see citing works
- Authors
- Lynch, C., Wahid, A., Tompson, J., Ding, T., Betker, J., Baruch, R., Armstrong, T., & Florence, P.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 1-8
- ISSN
- 2377-3766 (ISSN-L)
- arXiv
- 2210.06407
- OpenAlex
- W4391759936
- Semantic Scholar
- 0e34addae55a571d7efd3a5e2543e86dd7d41a83
- Cited by
- 113
- References
- 0
- Citations by year
- 2026: 10 · 2025: 52 · 2024: 38 · 2023: 13
- FWCI
- 15.67
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- natural language interaction, human-robot interaction, real-time robot control, behavior cloning, robot datasets, long-horizon manipulation, conversational robots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
