Scaling data-driven robotics with reward sketching and batch reinforcement learning
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
We present a framework for data-driven robotics that makes use of a large dataset of recorded robot experience and scales to several tasks using learned reward functions. We show how to apply this framework to accomplish three different object manipulation tasks on a real robot platform. Given demonstrations of a task together with task-agnostic recorded experience, we use a special form of human annotation as supervision to learn a reward function, which enables us to deal with real-world tasks where the reward signal cannot be acquired directly. Learned rewards are used in combination with a large dataset of experience from different tasks to learn a robot policy offline using batch RL. We show that using our approach it is possible to train agents to perform a variety of challenging manipulation tasks including stacking rigid objects and handling cloth.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 2×
“In simulated environments like video games, where experience and rewards are easy to obtain, deep RL is tremendously successful in outperforming top skilled humans by ingesting huge amounts of data [63, 69, 9].”
BackgroundBackpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879 · cited 1×
“We then update φ and θ using BPTT [70].”
Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Learning dexterous in-hand manipulation
Andrychowicz, O. M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364919887447 · Cited by 1,643
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621 · cited 1×
“Learning reward functions using inverse RL [52] achieved tremendous success [20, 27, 44, 21, 48, 73, 6].”
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847
Reinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831
Cited by
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824
“Cabi et al. (2019) propose to use reward learning from human preferences combined with offline RL to provide a user-friendly method for controlling robots for object manipulation tasks.”
MethodChallenges of real-world reinforcement learning: definitions, benchmarks and analysis
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021) · Machine Learning · doi:10.1007/s10994-021-05961-4 · Cited by 645
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
“Similar behavior has been observed in learned reward functions in the robotics domain [5].”
BackgroundA Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
Figueiredo Prudencio, R., Maximo, M. R. O. A., & Colombini, E. L. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3250269 · Cited by 361
D4RL: Datasets for Deep Data-Driven Reinforcement Learning
Fu, J., Kumar, A., Nachum, O., Tucker, G., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2004.07219 · Cited by 310
Inductive biases for deep learning of higher-level cognition
Goyal, A., & Bengio, Y. (2022) · Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rspa.2021.0068 · Cited by 300
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M., Chen, L., Ellis, K., et al. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.120 · Cited by 159
Wang, Z., Novikov, A., Zolna, K., Springenberg, J. T., Reed, S., Shahriari, B., Siegel, N., Merel, J., Gulcehre, C., Heess, N., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.15134 · Cited by 128
“Recently, it was shown that distributional off-policy deep RL techniques [5, 4], unlike their non-distributional counterparts, work well for offline RL in Atari [2] and robot manipulation [6].”
BackgroundMethodSelf-citationRobotGPT: Robot Manipulation Learning From ChatGPT
Jin, Y., Li, D., A, Y., Shi, J., Hao, P., Sun, F., Zhang, J., & Fang, B. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3357432 · Cited by 103
“Deep Learning advances the development of robot learning when the state includes image [13].”
BackgroundMorales, E. F., Murrieta-Cid, R., Becerra, I., & Esquivel-Basaldua, M. A. (2021) · Intelligent Service Robotics · doi:10.1007/s11370-021-00398-z · Cited by 101
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
“To address this, Wu et al. [2023] and Cabi et al. [2019] use feedback on specific portions of examples and Wu et al. [2023] use feedback with respect to different goals of the model (e.g., correctness, relevance).”
BackgroundAn empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2020) · arXiv · doi:10.48550/arxiv.2003.11881 · Cited by 78
“However, distributional deep RL approaches [Dabney et al. 2018; Bellemare et al. 2017; Barth-Maron et al. 2018] have been shown to produce better performance in the offline setting in both Atari [Agarwal et al. 2019] and robot manipulation [Cabi et al. 2019].”
BackgroundWhat Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., & Martín-Martín, R. (2021) · arXiv · doi:10.48550/arxiv.2108.03298 · Cited by 66
“…Prior work has tried using human supervision in the form of human demonstrations provided both by teleoperating the robot [9, 10] and via kinesthetic teaching [54–56], corrections to actions taken by the robot [21, 57–59], and positive and negative feedback for robot actions [60–62, 13, 63, 64].”
BackgroundMethodLearning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience
Lee, R., Ward, D., Cosgun, A., Dasagi, V., Corke, P., & Leitner, J. (2020) · arXiv · doi:10.48550/arxiv.2010.03209 · Cited by 47
“We use Batch RL for this work: once the robot has collected the dataset of random experience, we utilize this data to train our Q-network in an offline manner [24, 25, 26].”
MethodOpenVLA: An Open-Source Vision-Language-Action Model
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09246 · Cited by 62
A Survey of Reinforcement Learning from Human Feedback
Kaufmann, T., Weng, P., Bengs, V., & Hüllermeier, E. (2023) · arXiv · doi:10.48550/arxiv.2312.14925 · Cited by 43
Imitating Interactive Intelligence
Abramson, J., Ahuja, A., Barr, I., Brussee, A., Carnevale, F., Cassin, M., Chhaparia, R., Clark, S., Damoc, B., Dudzik, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.05672 · Cited by 43
Human-in-the-Loop Imitation Learning using Remote Teleoperation
Mandlekar, A., Xu, D., Martín-Martín, R., Zhu, Y., Fei-Fei, L., & Savarese, S. (2020) · arXiv · doi:10.48550/arxiv.2012.06733 · Cited by 35
“One paradigm is Reinforcement Learning (RL) with human feedback [40], where a human provides rewards during agent training [8, 11, 24, 25, 34, 36], but this suffers from the same limitations as IRL due to the need for extensive agent interaction.”
BackgroundManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
Mu, T., Ling, Z., Xiang, F., Yang, D., Li, X., Tao, S., Huang, Z., Jia, Z., & Su, H. (2021) · arXiv · doi:10.48550/arxiv.2107.14483 · Cited by 34
The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
Stone, A., Ramirez, O., Konolige, K., & Jonschkowski, R. (2021) · arXiv · doi:10.48550/arxiv.2101.02722 · Cited by 31
“Reinforcement learning has also been successfully applied to robotics training in the real world [8], [11]–[13].”
MethodOffline Meta-Reinforcement Learning with Online Self-Supervision
Pong, V. H., Nair, A., Smith, L., Huang, C., & Levine, S. (2021) · arXiv · doi:10.48550/arxiv.2107.03974 · Cited by 25
PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection
Dass, S., Pertsch, K., Zhang, H., Lee, Y., Lim, J. J., & Nikolaidis, S. (2022) · arXiv · doi:10.48550/arxiv.2212.04708 · Cited by 16
“Research into teleoperation systems has focused on exploring different interfaces like VR headsets [4], [5], joysticks [2] and smartphones [1].”
Background$π_0$: A Vision-Language-Action Flow Model for General Robot Control
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.24164 · Cited by 13
“Early work in this area often utilized self-supervised or autonomous data collection [26, 22, 8], providing a tractable data source for simple tasks such as grasping [18, 37] or pushing [56], but without the complexity of more dexterous behaviors.”
S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
Sinha, S., Mandlekar, A., & Garg, A. (2021) · arXiv · doi:10.48550/arxiv.2103.06326 · Cited by 5
On The Fragility of Learned Reward Functions
McKinney, L., Duan, Y., Krueger, D., & Gleave, A. (2023) · arXiv · doi:10.48550/arxiv.2301.03652 · Cited by 1
Metadata
- Authors
- Cabi, S., Gómez Colmenarejo, S., Novikov, A., Konyushova, K., Reed, S., Jeong, R., Zolna, K., Aytar, Y., Budden, D., Vecerik, M., Sushkov, O., Barker, D., Scholz, J., Denil, M., de Freitas, N., & Wang, Z.
- Venue
- Robotics: Science and Systems XVI
- Published
- 2020-07-12
- OpenAlex
- W3040490156
- Cited by
- 89
- References
- 20
- FWCI
- 6.85
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.