IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Learning from offline task demonstrations is a problem of great interest in robotics. For simple short-horizon manipulation tasks with modest variation in task instances, offline learning from a small set of demonstrations can produce controllers that successfully solve the task. However, leveraging a fixed batch of data can be problematic for larger datasets and longer-horizon tasks with greater variations. The data can exhibit substantial diversity and consist of suboptimal solution approaches. In this paper, we propose Implicit Reinforcement without Interaction at Scale (IRIS), a novel framework for learning from large-scale demonstration datasets. IRIS factorizes the control problem into a goal-conditioned low-level controller that imitates short demonstration sequences and a high-level goal selection mechanism that sets goals for the low-level and selectively combines parts of suboptimal solutions leading to more successful task completions. We evaluate IRIS across three datasets, including the RoboTurk Cans dataset collected by humans via crowdsourcing, and show that performant policies can be learned from purely offline learning. Additional results at https://sites.google.com/stanford.edu/iris/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationKnow What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340
Deep Q-learning From Demonstrations
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11757 · Cited by 889
Overcoming Exploration in Reinforcement Learning with Demonstrations
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463162 · Cited by 712
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.10293 · Cited by 697
Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
Vecerik, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Rothörl, T., Lampe, T., & Riedmiller, M. (2017) · arXiv · doi:10.48550/arxiv.1707.08817 · Cited by 478
Data-efficient hierarchical reinforcement learning
Nachum, O., Gu, S., Lee, H., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.08296 · Cited by 446 · cited 3×
“The high-level goal selection mechanism chooses goal states for the low-level to try and reach (similar to [27]).”
BackgroundROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
Mandlekar, A., Zhu, Y., Garg, A., Booher, J., Spero, M., Tung, A., Gao, J., Emmons, J., Gupta, A., Orbay, E., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.02790 · Cited by 134 · cited 10×
“3: Manipulation Results: We present a comparison of IRIS against several baselines on the Robosuite Lift and RoboTurk Cans datasets (left two plots).”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 27
- 10.15607/RSS.2018.XIV.009 · DOI · Paper page
- ross, 2014, Reinforcement and imitation learning via interactive no-regret learning, arXiv preprint arXiv 1406 5979
- pong, 2018, Temporal difference models: Model-free deep rl for model-based control, arXiv preprint arXiv 1802 09085
- 10.1109/IROS.2016.7758091 · DOI · Paper page
- 10.1109/ICRA.2013.6630809 · DOI · Paper page
- 10.15607/RSS.2018.XIV.012 · DOI
- fujimoto, 2018, Off-policy deep reinforcement learning without exploration, arXiv preprint arXiv 1812 02588
- gao, 2018, Reinforcement learning from imperfect demonstrations, arXiv preprint arxiv 1802 05807
- 10.1109/ROBOT.2009.5152709 · DOI · Paper page
- higgins, 2017, Beta-vae: Learning basic visual concepts with a constrained variational framework, ICLRE, 2, 6
- 10.1177/0278364912445831 · DOI · Paper page
- kingma, 2013, Auto-encoding variational bayes, arXiv preprint arXiv 1312 6114
- andrychowicz, 2017, Hindsight experience replay, Advances in neural information processing systems, 5048
- agarwal, 2019, Striving for simplicity in off-policy deep reinforcement learning, arXiv preprint arXiv 1907 09977
- 10.1109/ICRA.2016.7487517 · DOI · Paper page
- bhatt, 2019, Crossnorm: Normalization for off-policy td reinforcement learning, arXiv preprint arXiv 1902 05023
- dundar, 2020, Unsupervised disentanglement of pose, appearance and background from images and videos, arXiv preprint arXiv 2001 09518
- ding, 2019, Goal-conditioned imitation learning, arXiv preprint arXiv 1906 03008
- achiam, 2019, Towards characterizing divergence in deep q-learning, arXiv preprint arXiv 1903 00066
- fan, 2018, Surreal: Open-source reinforcement learning framework and robot manipulation benchmark, Conference on Robot Learning
- abbeel, 2011, Inverse reinforcement learning, Encyclopedia of Machine Learning, 554
- 10.1177/0278364918784350 · DOI · Paper page
- levine, 2016, Learning hand-eye coordination for robotic grasping with large-scale data collection, ISER, 173
- kumar, 2019, Stabilizing off-policy q-learning via bootstrapping error reduction, arXiv preprint arXiv 1906 03008
- lynch, 2019, Learning latent plans from play, arXiv preprint arXiv 1903 00066
- liu, 2019, Off-policy policy gradient with state distribution correction, arXiv preprint arXiv 1904 01870
- 10.15607/RSS.2017.XIII.058 · DOI · Paper page
Cited by
Diffusion policy: Visuomotor policy learning via action diffusion
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., & Song, S. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649241273668 · Cited by 613
“This surprising result stands in contrast to the majority of recent behavior cloning work that generally relies on velocity control (Mandlekar et al., 2021; Sha fi ullah et al., 2022; Zhang et al., 2018; Florence et al., 2020; Mandlekar et al., 2020a, 2020b).”
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., & Song, S. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.026 · Cited by 530
Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
Nasiriany, S., Liu, H., & Zhu, Y. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812140 · Cited by 91
GTI: Learning to Generalize across Long-Horizon Tasks from Human Demonstrations
Mandlekar, A., Xu, D., Martín-Martín, R., Savarese, S., & Fei-Fei, L. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.061 · Cited by 90
Self-citationWhat Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., & Martín-Martín, R. (2021) · arXiv · doi:10.48550/arxiv.2108.03298 · Cited by 66
“We chose to evaluate 6 algorithms in this study – Behavioral Cloning (BC), BC with an RNN policy (BC-RNN), Hierarchical Behavioral Cloning (HBC) [10], Batch-Constrained Q-Learning (BCQ) [26], Conservative Q-Learning (CQL) [27], and IRIS [20].”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Mandlekar, A., Ramos, F., Boots, B., Savarese, S., Fei-Fei, L., Garg, A., & Fox, D.
- Venue
- 2020 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2020-05
- Pages
- 4414-4420
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2986616324
- Cited by
- 78
- References
- 12
- FWCI
- 3.73
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data. https://app.cochat.ai/papers/10.1109/icra40945.2020.9196935
- The paper
- Mandlekar, A., Ramos, F., Boots, B., Savarese, S., Fei-Fei, L., Garg, A., & Fox, D. (2020). IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data. 2020 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra40945.2020.9196935
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.