The DOI matches its DataCite record; 46 of the 46 references it cites were found in a registry or an index.
Interactive Language: Talking to Robots in Real Time
Open accessAbstract
We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuo-linguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. "make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Several related efforts [3-6, 27] also collect large datasets (e.g. 1000s of demonstrations) by using a large number of human operators over extended periods of time.”
From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 15
- M. Shridhar, L. Manuelli, and D. Fox, "Cliport: What and where pathways for robotic manipulation," in Conference on Robot Learning. PMLR, 2022, pp. 894-906.
- S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn et al., "Learning language- conditioned robot behavior from offline data and crowd-sourced annotation," in Conference on Robot Learning. PMLR, 2022, pp. 1303-1315.
- E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, "Bc-z: Zero-shot task generalization with robotic imitation learning," in Conference on Robot Learning. PMLR, 2022, pp. 991-1002.
- S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies," The Journal of Machine Learning Research, vol. 17, no. 1, pp. 1334-1373, 2016.
- Y. Ding, C. Florensa, P. Abbeel, and M. Phielipp, "Goal-conditioned imitation learning," Advances in neural information processing systems, vol. 32, 2019.
- C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, "Learning latent plans from play," in Conference on robot learning. PMLR, 2020, pp. 1113-1132.
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learning from human preferences," Advances in neural information processing systems, vol. 30, 2017.
- A. Broad, J. Arkin, N. Ratliff, T. Howard, B. Argall, and D. C. Graph, "Towards real-time natural language corrections for assistive robots," in RSS Workshop on Model Learning for Human-Robot Communication, 2016.
- S. Karamcheti, M. Srivastava, P. Liang, and D. Sadigh, "Lila: Language- informed latent actions," in Conference on Robot Learning. PMLR, 2022, pp. 1379-1390.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervision," in International Conference on Machine Learning. PMLR, 2021, pp. 8748-8763.
- R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu, "On layer normalization in the transformer architecture," in International Conference on Machine Learning. PMLR, 2020, pp. 10 524-10 533.
- T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, "Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation," in IEEE International Conference on Robotics and Automation (ICRA), 2018.
- B. Wu, S. Nair, F.-F. Li, and C. Finn, "Example-driven model-based reinforcement learning for solving long-horizon visuomotor tasks," in 5th Annual Conference on Robot Learning, 2021. [Online]. Available: https://openreview.net/forum?id= daq0uh6yXr
- B. Hayes and B. Scassellati, "Challenges in shared-environment human-robot collaboration," learning, vol. 8, no. 9, 2013.
Cited by
Sorted by citations
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276
Self-citationDeep Learning Approaches to Grasp Synthesis: A Review
Newbury, R., Gu, M., Chumbley, L., Mousavian, A., Eppner, C., Leitner, J., Bohg, J., Morales, A., Asfour, T., Kragić, D., et al. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3280597 · Cited by 294
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., Deane, G., Fleming, S. M., Frith, C., Ji, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08708 · Cited by 196
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration 0
O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611477 · Cited by 210
Self-citationRoCo: Dialectic Multi-Robot Collaboration with Large Language Models
Mandi, Z., Jain, S., & Song, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610855 · Cited by 316
RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
Mandi, Z., Jain, S., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2307.04738 · Cited by 180
Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174
A review of visual SLAM for robotics: evolution, properties, and future applications
Al-Tawil, B., Hempel, T., Abdelrahman, A. A., & Al-Hamadi, A. K. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1347985 · Cited by 92
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Lu, J., Clark, C., Lee, S.-H., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., & Kembhavi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02497 · Cited by 195
Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
Kawaharazuka, K., Oh, J., Yamada, J., Posner, I., & Zhu, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3609980 · Cited by 135
PIRLNav: Pretraining with Imitation and RL Finetuning for OBJECTNAV
Ramrakhya, R., Batra, D., Wijmans, E., & Das, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01716 · Cited by 71
RT-H: Action Hierarchies Using Language
Belkhale, S., Ding, T., Xiao, T., Sermanet, P., Vuong, Q., Tompson, J., Chebotar, Y., Dwibedi, D., & Sadigh, D. (2024) · arXiv · doi:10.48550/arxiv.2403.01823 · Cited by 71
“Data driven approaches have also been explored, requiring large datasets of language corrections [49– 52] or shared autonomy during deployment [2].”
MethodSelf-citationChat with the Environment: Interactive Multimodal Perception Using Large Language Models
Zhao, X., Li, M., Weber, C., Hafez, M. B., & Wermter, S. (2023) · 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342363 · Cited by 62
Frering, L., Steinbauer-Wagner, G., & Holzinger, A. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109771 · Cited by 57
From screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 76
RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Sermanet, P., Ding, T., Zhao, J. W., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac-Arnold, G., Maddineni, S., Joshi, N. J., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610216 · Cited by 108
Self-citationOpenVLA: An Open-Source Vision-Language-Action Model
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09246 · Cited by 62
“…Play [95] 0.4% Berkeley Cable Routing [96] 0.2% Roboturk [97] 2.3% Viola [98] 0.9% Berkeley Autolab UR5 [99] 1.2% Toto [100] 2.0% Language Table [101] 4.4% Stanford Hydra Dataset [102] 4.4% Austin Buds Dataset [103] 0.2% NYU Franka Play Dataset [104] 0.8% Furniture Bench Dataset [105] 2.4% UCSD…”
MethodInfluentialRaptis, E. K., Kapoutsis, A., & Kosmatopoulos, E. B. (2025) · Frontiers Robotics AI · doi:10.3389/frobt.2025.1605405 · Cited by 58
Jiang, Y., Fan, L., Sun, X., Luo, Z., Wang, H., Lai, R., Wang, J., Gan, Q., Li, N., & Tian, J. (2024) · Advanced Science · doi:10.1002/advs.202402705 · Cited by 44
Robot learning in the era of foundation models: a survey
Xiao, X., Liu, J., Wang, Z., Zhou, Y., Qi, Y., Jiang, S., He, B., & Cheng, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129963 · Cited by 56
Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections
Zha, L., Cui, Y., Lin, L.-H., Kwon, M., Arenas, M. G., Zeng, A., Xia, F., & Sadigh, D. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610455 · Cited by 56
DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment
Guo, Y., Wang, Y.-J., Zha, L.-H., Jiang, Z., & Chen, J. (2023) · IEEE/RJS International Conference on Intelligent RObots and Systems · doi:10.1109/iros58592.2024.10802284 · Cited by 118
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models
Xiao, T. G., Chan, H., Sermanet, P., Wahid, A., Brohan, A., Hausman, K., Levine, S., & Tompson, J. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.029 · Cited by 84
Self-citationWu, D., Zheng, P., Zhao, Q., Zhang, S., Qi, J., Hu, J., Zhu, G., & Wang, L. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103064 · Cited by 38
RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins
Mu, Y., Chen, T., Chen, Z., Peng, S., Lan, Z., Gao, Z., Liang, Z., Yu, Q., Zou, Y., Xu, M., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02575 · Cited by 65
AgentLens: Visual Analysis for Agent Behaviors in LLM-Based Autonomous Systems
Lu, J., Pan, B., Chen, J., Feng, Y., Hu, J., Peng, Y., & Chen, W. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3394053 · Cited by 46
StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects
Liu, W., Du, Y., Hermans, T., Chernova, S., & Paxton, C. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.031 · Cited by 72
Gong, R., Huang, J., Zhao, Y.-Z., Geng, H., Gao, X.-F., Wu, Q., Ai, W., Zhou, Z.-H., Terzopoulos, D., Zhu, S.-C., et al. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.01873 · Cited by 96
GRID: Scene-Graph-based Instruction-driven Robotic Task Planning
Ni, Z., Deng, X., Tai, C., Zhu, X., Xie, Q., Huang, W., Wu, X., & Zeng, L. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801291 · Cited by 42
Exploring Embodied Multimodal Large Models: Development, datasets, and future directions
Chen, S., Wu, Z., Zhang, K., Li, C., Zhang, B., Ma, F., Yu, F. R., & Li, Q. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103198 · Cited by 39
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Ha, H., Florence, P., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2307.14535 · Cited by 24
“When providing action labels, humans can either provide task-specific [9, 10, 15, 18], or task-agnostic (“play”) data [13, 14, 16, 19].”
BackgroundSelf-citationExploring the Robustness of Decision-Level Through Adversarial Attacks on LLM-Based Embodied Models
Liu, S., Chen, J.-W., Ruan, S.-W., Su, H., & Yin, Z.-X. (2024) · ACM Multimedia · doi:10.1145/3664647.3680616 · Cited by 64
RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (Early Version)
Mu, Y., Chen, T., Peng, S., Chen, Z., Gao, Z., Zou, Y., Lin, L., Xie, Z., & Luo, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91813-1_17 · Cited by 73
LLM-controller: Dynamic robot control adaptation using large language models
Zahedifar, R., Baghshah, M. S., & Taheri, A. (2025) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2024.104913 · Cited by 23
Music in the digital age: commodity, community, communion
Cross, I. R. M. (2023) · AI & Society · doi:10.1007/s00146-023-01670-9 · Cited by 28
DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning
Jiang, Z., Xie, Y., Lin, K., Xu, Z., Wan, W., Mandlekar, A., Fan, L. J., & Zhu, Y. (2025) · 2025 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra55743.2025.11127809 · Cited by 24
Take a shot! Natural language control of intelligent robotic X-ray systems in surgery
Killeen, B. D., Chaudhary, S., Osgood, G. M., & Unberath, M. (2024) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-024-03120-3 · Cited by 19
An Interactive Agent Foundation Model
Durante, Z., Gong, R., Sarkar, B., Wake, N., Taori, R., Tang, P. C., Lakshmikanth, S. K., Schulman, K. A., Milstein, A. S., Vo, H., et al. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00350 · Cited by 28
Learning to generate pointing gestures in situated embodied conversational agents
Deichler, A., Wang, S., Alexanderson, S., & Beskow, J. (2023) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1110534 · Cited by 16
CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation
Wang, J., Qin, Y., Kuang, K., Korkmaz, Y., Gurumoorthy, A., Su, H., & Wang, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01700 · Cited by 21
Unlocking Robotic Autonomy: A Survey on the Applications of Foundation Models
Jang, D.-S., Cho, D.-H., Lee, W.-C., Ryu, S.-K., Jeong, B.-M., Hong, M., Jung, M., Kim, M., Lee, M., Lee, S., et al. (2024) · International Journal of Control Automation and Systems · doi:10.1007/s12555-024-0438-7 · Cited by 17
SUGAR : Pre-training 3D Visual Representations for Robotics
Chen, S., García, R., Laptev, I., & Schmid, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01709 · Cited by 20
Lightweight Language-driven Grasp Detection using Conditional Consistency Model
Nguyen, N., Vu, M. N., Huang, B., Vuong, A., Le, N., Vo, T., & Nguyen, A. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802007 · Cited by 17
Learning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 346
“We use datasets previously collected from these simulators, i.e., Habitat object navigation with HM3D [16] and Language Table Data from [17] to train the UniSim.”
MethodSelf-citation(2024) · doi:10.1109/icra57147.2024.10611448 · Cited by 12
TalkWithMachines: Enhancing Human-Robot Interaction Through Large/Vision Language Models
Abbas, A. N., & Beleznai, C. (2024) · IEEE International Conference on Robotic Computing (IRC) · doi:10.1109/irc63610.2024.00039 · Cited by 15
IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learning
Hoque, R., Mandlekar, A., Garrett, C. R., Goldberg, K. Y., & Fox, D. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801523 · Cited by 22
Transferring Foundation Models for Generalizable Robotic Manipulation
Yang, J., Tan, W., Jin, C., Yao, K., Liu, B., Fu, J., Song, R., Wu, G., & Wang, L. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00201 · Cited by 35
Medical & healthcare robotics: a roadmap for enhanced precision, safety, and efficacy
Iakovidis, D. K., Vartholomeos, P., Gall, A. L., Cianchetti, M., Ozioko, O., Dahiya, R., Mazomenos, E. B., Vasconcelos, F., Stoyanov, D. V., Philpott, J., et al. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/ae09bf · Cited by 10
Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations
Heravi, N., Wahid, A., Lynch, C., Florence, P., Armstrong, T., Tompson, J., Sermanet, P., Bohg, J., & Dwibedi, D. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160888 · Cited by 9
Self-citationFine-Tuning Multimodal Transformer Models for Generating Actions in Virtual and Real Environments
Staroverov, A., Gorodetsky, A., Krishtopik, A. S., Izmesteva, U. A., Yudin, D. A., Kovalev, A. K., & Panov, A. I. (2023) · IEEE Access · doi:10.1109/access.2023.3334791 · Cited by 11
SPRINT: Scalable Policy Pre-Training via Language Instruction Relabeling
Zhang, J. M., Pertsch, K., Zhang, J., & Lim, J. J. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610606 · Cited by 16
HiCRISP: An LLM-Based Hierarchical Closed-Loop Robotic Intelligent Self-Correction Planner
Ming, C., Lin, J.-C., Fong, P.-K., Wang, H., Duan, X.-M., & He, J.-P. (2023) · ACM Cloud and Autonomic Computing Conference · doi:10.1109/cac63892.2024.10865457 · Cited by 7
Using Social Robotics to Identify Educational Behavior: A Survey
de Vaca, A. J. R.-C., Meléndez-Armenta, R. Á., & Ponce-Espinosa, H. (2024) · Electronics · doi:10.3390/electronics13193956 · Cited by 7
Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning
Yang, J., Zhu, H., Wang, Y., Wu, G., He, T., & Wang, L. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00653 · Cited by 7
This&That: Language-Gesture Controlled Video Generation for Robot Planning
Wang, B., Sridhar, N., Feng, C., der Merwe, M. V., Fishman, A., Fazeli, N., & Park, J. J. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11128780 · Cited by 32
A survey of robotic manipulation: From bottom-up approaches to end-to-end paradigms with LLMs
Peng, K., Li, Q., He, Z.-J., Zhang, B.-W., Fu, X., Li, B., Wang, X.-H., Cheng, Z.-Q., Yan, Y., & Peng, X.-J. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.132921 · Cited by 7
ICRT: In-Context Imitation Learning via Next-Token Prediction
Fu, M., Huang, H., Datta, G., Chen, L. Y., Panitch, W., Liu, F., Li, H., & Goldberg, K. Y. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11128272 · Cited by 74
Advancing Sustainable Construction: Discrete Modular Systems & Robotic Assembly
Liu, Y., Belousov, B., Schneider, T., Harsono, K., Cheng, T.-W., Shih, S.-G., Tessmann, O., & Peters, J. (2024) · Sustainability · doi:10.3390/su16156678 · Cited by 6
IRASim: A Fine-Grained World Model for Robot Manipulation
Zhu, F., Wu, H., Guo, S., Liu, Y., Cheang, C., & Kong, T. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00917 · Cited by 110
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
NVIDIA, :, Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L. "., Fang, Y., Fox, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14734 · Cited by 10
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., & Fox, D. (2023) · arXiv · doi:10.48550/arxiv.2310.17596 · Cited by 226
“One popular data source is human demonstrators that teleoperate robot arms [2–6, 23– 27 ], but collecting 3 Problem Setup Imitation Learning.”
BackgroundGuo, Y., Ni, Y., Jin, J., Jiang, Y., Li, D., Zhao, H., & Shen, Y. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3590433 · Cited by 5
PTQ4RIS: Post-Training Quantization for Referring Image Segmentation
Jiang, X., Yang, H., Zhu, K., Qiu, X., Zhao, S., & Zhou, S. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11128841 · Cited by 7
Instructing Hierarchical Tasks to Robots by Verbal Commands
Telkes, P., Angleraud, A., & Pieters, R. (2024) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii58957.2024.10417491 · Cited by 6
Intelligent control of robotic X-ray devices using a language-promptable digital twin
Killeen, B. D., Suresh, A., Gómez, C., Íñigo, B., Bailey, C., & Unberath, M. (2025) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-025-03351-y · Cited by 5
The linguistic feedback of tourism robots significantly influences visitors’ ecotourism behaviors
Chang, R., Yang, D., Lyu, D., Cao, J. J., Wei, Y., & Xiao, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-01045-8 · Cited by 5
Language-Conditioned Offline RL for Multi-Robot Navigation
Morad, S., Shankar, A., Blumenkamp, J., & Prorok, A. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11127288 · Cited by 15
Language as a cognitive and social tool at the time of large language models
Borghi, A. M., Livio, C. D., Gervasi, A. M., Mannella, F., Nolfi, S., & Tummolini, L. (2024) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-024-00152-8 · Cited by 9
Talk Through It: End User Directed Manipulation Learning
Winge, C., Imdieke, A., Aldeeb, B., Kang, D., & Desingh, K. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3433309 · Cited by 4
Detect2Interact: Localizing Object Key Field in Visual Question Answering with LLMs
Wang, J., Zhu, M., Li, Y., Li, H., Yang, L., & Woo, W. L. (2024) · IEEE Intelligent Systems · doi:10.1109/mis.2024.3384513 · Cited by 4
Fostering Interactive Language Use: Strategies and Challenges in English Language Teaching
Yuhan, S. (2024) · Lecture Notes in Education Psychology and Public Media · doi:10.54254/2753-7048/54/20241563 · Cited by 4
Navigating Education 5.0 Robotic Technique for Teaching Foreign Languages in Todays Classroom
Muthmainnah, M., Apriani, E., Marzuki, A. G., Yakin, A. M. A., Aeni, N., Azar, A. S., & Riyawi, M. R. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-3041-8.ch008 · Cited by 4
Multi-task real-robot data with gaze attention for dual-arm fine manipulation
Kim, H., Ohmura, Y., & Kuniyoshi, Y. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802034 · Cited by 10
PolarNet: 3D Point Clouds for Language-Guided Robotic ManipulationPreprint
Chen, S., García, R., Schmid, C., & Laptev, I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.15596 · Cited by 27
Intelligent Indoor Navigation for Home Robots based on Large Language Models
Chen, Z., Su, Z., Chesser, C., Bourlon, C., & Sheng, W. (2025) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case58245.2025.11163777 · Cited by 3
Controllable Traffic Simulation through LLM-Guided Hierarchical Reasoning and Refinement
Liu, Z., Li, L., Wang, Y., Lin, H., Cheng, H., Liu, Z., He, L., & Wang, J. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11247747 · Cited by 3
Song, S., Kang, D., & Park, C.-E. (2026) · IEEE Access · doi:10.1109/access.2026.3664145 · Cited by 4
Minimizing learning experiences in embodied agents language and action learning
Milano, N., & Nolfi, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129510 · Cited by 3
Chen, Z., Shi, Z., Lu, X., He, L., Qian, S., Zhou, E., Yin, Z., Ouyang, W., Shao, J., Qiao, Y., et al. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11246228 · Cited by 4
Policy Learning from Large Vision-Language Model Feedback Without Reward Modeling
Luu, T. M., Lee, D.-H., Lee, Y.-H., & Yoo, C. D. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11246902 · Cited by 2
FlowDreamer: A RGB-D World Model With Flow-Based Motion Representations for Robot Manipulation
Guo, J., Ma, X., Wang, Y., Yang, M., Liu, H., & Li, Q. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3653273 · Cited by 2
World Models for Robotic Manipulation: A Survey
Wang, F., Wang, Z., Pei, G., Zhang, M., 梁燦錫, Hu, J., Li, Z., Wu, J., Han, N., Zhang, Z., et al. (2026) · SmartBot · doi:10.1002/smb2.70053 · Cited by 11
Did You Mean...? Confidence-based Trade-offs in Semantic Parsing
Stengel-Eskin, E., & Durme, B. V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.159 · Cited by 2
Real-World Robot Control Based on Contrastive Deep Active Inference With Demonstrations
Fujii, K., Isomura, T., & Murata, S. (2024) · IEEE Access · doi:10.1109/access.2024.3477306 · Cited by 2
Toward Universal Embodied Planning in Scalable Heterogeneous Field Robots Collaboration and Control
Wan, H., Zhang, Y., Wang, J., Wu, D., Li, M., Chen, X., Deng, Y., Huang, Y., Sun, Z., Li, Z., et al. (2025) · Journal of Field Robotics · doi:10.1002/rob.22522 · Cited by 4
‘What Did the Robot Do in My Absence?’ Video Foundation Models to Enhance Intermittent Supervision
Katuwandeniya, K., Tian, L., & Kulić, D. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3539118 · Cited by 2
Robot Behavior Personalization From Sparse User Feedback
Patel, M., & Chernova, S. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3550833 · Cited by 2
InteLiPlan: An Interactive Lightweight LLM-Based Planner for Domestic Robot Autonomy
Ly, K., Lu, K., & Havoutis, I. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3662577 · Cited by 12
Zero-shot Object Navigation with Vision-Language Foundation Models Reasoning
Yuan, S., Shafique, M., Baghdadi, R., Khorrami, F., Tzes, A. P., & Fang, Y. (2024) · International Conference on Automation, Robotics and Applications · doi:10.1109/icara60736.2024.10553173 · Cited by 3
Nishimura, T., Kuyo, K., Kambara, M., & Sugiura, K. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802596 · Cited by 2
Working Backwards: Learning to Place by Picking
Limoyo, O., Konar, A., Ablett, T., Kelly, J. W., Hogan, F. R., & Dudek, G. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802173 · Cited by 1
LiLMaps: Learnable Implicit Language Maps
Kruzhkov, E., & Behnke, S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00749 · Cited by 1
Double-Feedback: Enhancing Large Language Models Reasoning in Robotic Tasks by Knowledge Graphs
Wang, H., Zhang, S., Wang, S., Jiang, T., & Ge, Y. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3562776 · Cited by 3
Interactive Agent Foundation Model: A Meta-Learning Strategy
Dwivedi, M., & Dwivedi, A. K. (2025) · International Conference on Intelligent Systems, Advanced Computing and Communication (ISACC) · doi:10.1109/isacc65211.2025.10969383 · Cited by 1
Visual Instruction as an Intuitive Interface for Robotic Sorting
Müller, A. T., & Stache, N. C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93429-2_12 · Cited by 1
Kube, D., Hadwiger, S., & Meisen, T. (2025) · Biomimetic Intelligence and Robotics · doi:10.1016/j.birob.2025.100249 · Cited by 2
Development of a Mobile Assistive Robot for Daily Living Support*
Bui, T. M., Kim, Y.-O., Moon, J., Cho, M.-Y., Seo, M., & Shin, D. (2025) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/ur65550.2025.11078095 · Cited by 1
Metadata
- Authors
- Lynch, C., Wahid, A., Tompson, J., Ding, T., Betker, J., Baruch, R., Armstrong, T., & Florence, P.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 1-8
- ISSN
- 2377-3766 (ISSN-L)
- arXiv
- 2210.06407
- OpenAlex
- W4391759936
- Semantic Scholar
- 0e34addae55a571d7efd3a5e2543e86dd7d41a83
- Cited by
- 113
- References
- 0
- Citations by year
- 2026: 10 · 2025: 52 · 2024: 38 · 2023: 13
- FWCI
- 15.67
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- natural language interaction, human-robot interaction, real-time robot control, behavior cloning, robot datasets, long-horizon manipulation, conversational robots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
