The DOI matches its DataCite record; 46 of the 46 references it cites were found in a registry or an index.
Interactive Language: Talking to Robots in Real Time
Open accessAbstract
We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuo-linguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. "make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Several related efforts [3-6, 27] also collect large datasets (e.g. 1000s of demonstrations) by using a large number of human operators over extended periods of time.”
From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 15
- M. Shridhar, L. Manuelli, and D. Fox, "Cliport: What and where pathways for robotic manipulation," in Conference on Robot Learning. PMLR, 2022, pp. 894-906.
- S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn et al., "Learning language- conditioned robot behavior from offline data and crowd-sourced annotation," in Conference on Robot Learning. PMLR, 2022, pp. 1303-1315.
- E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, "Bc-z: Zero-shot task generalization with robotic imitation learning," in Conference on Robot Learning. PMLR, 2022, pp. 991-1002.
- S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies," The Journal of Machine Learning Research, vol. 17, no. 1, pp. 1334-1373, 2016.
- Y. Ding, C. Florensa, P. Abbeel, and M. Phielipp, "Goal-conditioned imitation learning," Advances in neural information processing systems, vol. 32, 2019.
- C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, "Learning latent plans from play," in Conference on robot learning. PMLR, 2020, pp. 1113-1132.
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learning from human preferences," Advances in neural information processing systems, vol. 30, 2017.
- A. Broad, J. Arkin, N. Ratliff, T. Howard, B. Argall, and D. C. Graph, "Towards real-time natural language corrections for assistive robots," in RSS Workshop on Model Learning for Human-Robot Communication, 2016.
- S. Karamcheti, M. Srivastava, P. Liang, and D. Sadigh, "Lila: Language- informed latent actions," in Conference on Robot Learning. PMLR, 2022, pp. 1379-1390.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervision," in International Conference on Machine Learning. PMLR, 2021, pp. 8748-8763.
- R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu, "On layer normalization in the transformer architecture," in International Conference on Machine Learning. PMLR, 2020, pp. 10 524-10 533.
- T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, "Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation," in IEEE International Conference on Robotics and Automation (ICRA), 2018.
- B. Wu, S. Nair, F.-F. Li, and C. Finn, "Example-driven model-based reinforcement learning for solving long-horizon visuomotor tasks," in 5th Annual Conference on Robot Learning, 2021. [Online]. Available: https://openreview.net/forum?id= daq0uh6yXr
- B. Hayes and B. Scassellati, "Challenges in shared-environment human-robot collaboration," learning, vol. 8, no. 9, 2013.
Cited by
Sorted by citations
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276
Self-citationDeep Learning Approaches to Grasp Synthesis: A Review
Newbury, R., Gu, M., Chumbley, L., Mousavian, A., Eppner, C., Leitner, J., Bohg, J., Morales, A., Asfour, T., Kragić, D., et al. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3280597 · Cited by 294
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., Deane, G., Fleming, S. M., Frith, C., Ji, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08708 · Cited by 196
Metadata
- Authors
- Lynch, C., Wahid, A., Tompson, J., Ding, T., Betker, J., Baruch, R., Armstrong, T., & Florence, P.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 1-8
- ISSN
- 2377-3766 (ISSN-L)
- arXiv
- 2210.06407
- OpenAlex
- W4391759936
- Semantic Scholar
- 0e34addae55a571d7efd3a5e2543e86dd7d41a83
- Cited by
- 113
- References
- 0
- Citations by year
- 2026: 10 · 2025: 52 · 2024: 38 · 2023: 13
- FWCI
- 15.67
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- natural language interaction, human-robot interaction, real-time robot control, behavior cloning, robot datasets, long-horizon manipulation, conversational robots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
