The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
End-To-End Robotic Reinforcement Learning without Reward Engineering
Open accessAbstract
The combination of deep neural network models and reinforcement learning algorithms can make it possible to learn policies for robotic behaviors that directly read in raw sensory inputs, such as camera images, effectively subsuming both estimation and control into one model.However, realworld applications of reinforcement learning must specify the goal of the task by means of a manually programmed reward function, which in practice requires either designing the very same perception pipeline that end-to-end reinforcement learning promises to avoid, or else instrumenting the environment with additional sensors to determine if the task has been performed successfully.In this paper, we propose an approach for removing the need for manual engineering of reward specifications by enabling a robot to learn from a modest number of examples of successful outcomes, followed by actively solicited queries, where the robot shows the user a state and asks for a label to determine whether that state represents successful completion of the task.While requesting labels for every single state would amount to asking the user to manually provide the reward signal, our method requires labels for only a tiny fraction of the states seen during training, making it an efficient and practical approach for learning skills without manually engineered rewards.We evaluate our method on real-world robotic manipulation tasks where the observations consist of images viewed by the robot's camera.In our experiments, our method effectively learns to arrange objects, place books, and drape cloth, directly from images and without any manually specified reward functions, and with only 1-4 hours of interaction with the real world.Videos of learned behavior are available at sites.google.com/view/reward-learning-rl/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594
Self-citationSoft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994
Self-citationLevine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971
Self-citationGenerative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847
Self-citationSim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385
Cited by
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570
Self-citationSurvey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174
Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
Johns, E. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9560942 · Cited by 119
Language Conditioned Imitation Learning Over Unstructured Data
Lynch*, C., & Sermanet*, P. (2021) · Robotics: Science and Systems XVII · doi:10.15607/rss.2021.xvii.047 · Cited by 110
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos
Smith, L., Dhawan, N., Zhang, M., Abbeel, P., & Levine, S. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.024 · Cited by 102
Self-citationUncertainty-Aware Action Advising for Deep Reinforcement Learning Agents
Da Silva, F. L., Hernandez-Leal, P., Kartal, B., & Taylor, M. E. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6036 · Cited by 69
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., & Martín-Martín, R. (2021) · arXiv · doi:10.48550/arxiv.2108.03298 · Cited by 66
XIRL: Cross-embodiment inverse reinforcement learning
Zakka, K., Zeng, A., Florence, P., Tompson, J., Bohg, J., & Dwibedi, D. (2021) · arXiv · doi:10.48550/arxiv.2106.03911 · Cited by 38
Latent Imagination Facilitates Zero-Shot Transfer in Autonomous Racing
Brunnbauer, A., Berducci, L., Brandstatter, A., Lechner, M., Hasani, R., Rus, D., & Grosu, R. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9811650 · Cited by 36
Human-in-the-Loop Imitation Learning using Remote Teleoperation
Mandlekar, A., Xu, D., Martín-Martín, R., Zhu, Y., Fei-Fei, L., & Savarese, S. (2020) · arXiv · doi:10.48550/arxiv.2012.06733 · Cited by 35
Affordance Learning from Play for Sample-Efficient Policy Learning
Borja-Diaz, J., Mees, O., Kalweit, G., Hermann, L., Boedecker, J., & Burgard, W. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9811889 · Cited by 32
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 27
Language Conditioned Imitation Learning over Unstructured Data
Lynch, C., & Sermanet, P. (2020) · arXiv · doi:10.48550/arxiv.2005.07648 · Cited by 18
Metadata
- Authors
- Singh, A., Yang, L., Finn, C., & Levine, S.
- Venue
- Robotics: Science and Systems XV
- Published
- 2019-06-22
- OpenAlex
- W2938421504
- Cited by
- 196
- References
- 12
- FWCI
- 12.22
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.