Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks
Open accessAbstract
General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse repertoire of general-purpose skills that allow composing long-horizon tasks by following unconstrained language instructions. In this letter, we present Composing Actions from Language and Vision (CALVIN) (ComposingActions fromLanguage andVision), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites. We evaluate the agents in zero-shot to novel language instructions and to novel environments. We show that a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents that learn to relate human language to their world models with this benchmark.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
During training the table blocks can start at any pose in a range, but the released scene-D evaluation fixes them at the same poses.
“During training the table blocks can start at any pose in a range, but the released scene-D evaluation fixes them at the same poses.”
From What Are We Actually Benchmarking in Robot Manipulation? (2026) · cites this paper as [3]
CALVIN's most commonly used protocol, ABC → D, trains on three table scenes (A, B, C) and tests on a fourth held-out scene D, where each evaluation episode is a chain of five language-conditioned tasks scored as the average number completed (ATC).
“CALVIN's most commonly used protocol, ABC → D, trains on three table scenes (A, B, C) and tests on a fourth held-out scene D, where each evaluation episode is a chain of five language-conditioned tasks scored as the average number completed (ATC).”
From What Are We Actually Benchmarking in Robot Manipulation? (2026) · cites this paper as [3]
The tasks MimicGen was demonstrated on have pose variation well beyond typical prior works using BC from human demos.
“The tasks also have pose variation well beyond typical prior works using BC from human demos [1,3-7,30,33,55,56].”
From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [56]
Knowledge distillation migrates knowledge from large teacher models to lightweight student models, significantly improving training efficiency while maintaining performance.
“on the other hand, the knowledge from the large-scale teacher model is migrated to the lightweight student model through knowledge distillation [310], which significantly improves the training efficiency while maintaining the performance of the while significantly improving the training efficiency.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [310]
Octo learns control mostly from self-supervised visual observations and reduces the burden on language annotation.
“This enables our model to learn control mostly from self-supervised visual observations and reduces the burden on language annotation, similar to prior work on multi-context imitation learning [54, 62, 61, 63].”
From Octo: An Open-Source Generalist Robot Policy (2024) · cites this paper as [62]
The citation lists haven’t been fetched yet
References 37 · Cited by 189
- Authors
- Mees, O., Hermann, L., Rosete-Beas, E., & Burgard, W. B.
- Affiliations
- University of Freiburg (DE)
- University of Freiburg, Freiburg, Germany
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2022-07
- Volume
- 7
- Issue
- 3
- Pages
- 7327-7334
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W4226073959
- Cited by
- 189
- References
- 37
- Citations by year
- 2026: 76 · 2025: 63 · 2024: 33 · 2023: 14 · 2022: 3
- FWCI
- 11.38
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- German Federal Ministry of Education and Research (01IS18040B-OML)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- benchmark, zero-shot generalization, human-robot interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
