Residual Policy Learning
Open accessAbstract
We present Residual Policy Learning (RPL): a simple method for improving nondifferentiable policies using model-free deep reinforcement learning. RPL thrives in complex robotic manipulation tasks where good but imperfect controllers are available. In these tasks, reinforcement learning from scratch remains data-inefficient or intractable, but learning a residual on top of the initial controller can yield substantial improvements. We study RPL in six challenging MuJoCo tasks involving partial observability, sensor noise, model misspecification, and controller miscalibration. For initial controllers, we consider both hand-designed policies and model-predictive controllers with known or learned transition models. By combining learning with control algorithms, RPL can perform long-horizon, sparse-reward tasks for which reinforcement learning alone fails. Moreover, we find that RPL consistently and substantially improves on the initial controllers. We argue that RPL is a promising approach for combining the complementary strengths of deep reinforcement learning and robotic control, pushing the boundaries of what either can achieve independently. Video and code at https://k-r-allen.github.io/residual-policy-learning/.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A single residual-RL policy trained jointly on all demonstrations repairs kinematics instead of learning tasks, so it needs no per-task training and can generate additional physically consistent variants of each demonstration.
“the key design is a single residual-RL policy (Johannink et al., 2019;Silver et al., 2018) trained jointly across all demonstrations, which repairs kinematics rather than learning tasks, and therefore needs no per-task training and can synthesize additional physically consistent variants of each demonstration.”
From Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning (2026) · cites this paper as Silver et al., 2018
Section IV (p. 3) supports the general idea of learning a residual policy that corrects an initial policy. But the paper describes training residuals for manipulation tasks, not one policy trained jointly on demonstrations. It does not establish kinematics-only repair, freedom from per-task training, or generation of physically consistent demonstration variants. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037 · see citing works
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · see citing works
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · see citing works
The unscented Kalman filter for nonlinear estimation
Wan, E. A., & van der Merwe, R. (2002) · IEEE Adaptive Systems for Signal Processing, Communications, and Control Symposium (AS-SPCC) · doi:10.1109/asspcc.2000.882463 · Cited by 4,373 · see citing works
Apprenticeship learning via inverse reinforcement learning
Abbeel, P., & Ng, A. Y. (2004) · doi:10.1145/1015330.1015430 · Cited by 2,873 · see citing works
Soft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994 · see citing works
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941 · see citing works
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,868 · see citing works
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815 · see citing works
Learning dexterous in-hand manipulation
Andrychowicz, O. M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364919887447 · Cited by 1,643 · see citing works
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989385 · Cited by 1,485 · see citing works
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,348 · see citing works
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8463189 · Cited by 898 · see citing works
Neural networks for self-learning control systems
Nguyen, D., & Widrow, B. C. (1990) · IEEE Control Systems Magazine · doi:10.1109/37.55119 · Cited by 859 · see citing works
Continuous Deep Q-Learning with Model-based AccelerationPreprint
Gu, S., Lillicrap, T., Sutskever, I., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00748 · Cited by 770 · see citing works
Gaussian Processes for Data-Efficient Learning in Robotics and Control
Deisenroth, M. P., Fox, D., & Rasmussen, C. E. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.218 · Cited by 659 · see citing works
Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning
Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., & Funkhouser, T. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593986 · Cited by 606 · see citing works
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.12114 · Cited by 525 · see citing works
Residual Reinforcement Learning for Robot Control
Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J. A., Solowjow, E., & Levine, S. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794127 · Cited by 480 · see citing works
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for ResearchPreprint
Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej, M., Welinder, P., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09464 · Cited by 372 · see citing works
Learning Continuous Control Policies by Stochastic Value GradientsPreprint
Heess, N., Wayne, G., Silver, D., Lillicrap, T., Tassa, Y., & Erez, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.09142 · Cited by 359 · see citing works
One-Shot Visual Imitation Learning via Meta-Learning
Finn, C., Yu, T., Zhang, T., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1709.04905 · Cited by 341 · see citing works
Recurrent Experience Replay in Distributed Reinforcement Learning.
Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., & Dabney, W. (2018) · International Conference on Learning Representations · Cited by 292 · see citing works
Probabilistic policy reuse in a reinforcement learning agent
Fernández, F., & Veloso, M. (2006) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/1160633.1160762 · Cited by 274 · see citing works
Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis
Reuther, A., Kepner, J., Byun, C., Samsi, S., Arcand, W., Bestor, D., Bergeron, B., Gadepally, V. N., Houle, M., Hubbell, M., et al. (2018) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec.2018.8547629 · Cited by 251 · see citing works
Automatic gait optimization with Gaussian process regression
Lizotte, D. J., Wang, T., Bowling, M., & Schuurmans, D. (2007) · Cited by 237 · see citing works
Temporal Difference Models: Model-Free Deep RL for Model-Based ControlPreprint
Pong, V. H., Gu, S., Dalal, M., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09081 · Cited by 164 · see citing works
Automatic LQR tuning based on Gaussian process global optimization
Marco, A., Hennig, P., Bohg, J., Stefan Schaal, A., & Trimpe, S. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487144 · Cited by 145 · see citing works
Benchmarking Reinforcement Learning Algorithms on Real-World RobotsPreprint
Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., & Bergstra, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.07731 · Cited by 129 · see citing works
Marco, A., Berkenkamp, F., Hennig, P., Schoellig, A. P., Krause, A., Stefan Schaal, A., & Trimpe, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989186 · Cited by 114 · see citing works
Ajay, A., Wu, J., Fazeli, N., Bauzá, M., Kaelbling, L. P., Tenenbaum, J. B., & Rodríguez, A. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593995 · Cited by 102 · see citing works
Self-citationEX2: Exploration with Exemplar Models for Deep Reinforcement Learning
Fu, J., Co-Reyes, J. D., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.01260 · Cited by 95 · see citing works
Using response surfaces and expected improvement to optimize snake robot gait parameters
Tesch, M., Schneider, J., & Choset, H. (2011) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2011.6095076 · Cited by 86 · see citing works
Active Model Learning and Diverse Action Sampling for Task and Motion Planning
Wang, Z., Garrett, C. R., Kaelbling, L. P., & Lozano‐Pérez, T. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594027 · Cited by 54 · see citing works
Self-citationCombining model-based policy search with online model learning for control of physical humanoids
Mordatch, I., Mishra, N., Eppner, C., & Abbeel, P. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487140 · Cited by 49 · see citing works
Combining learned and analytical models for predicting action effects.Preprint
Kloss, A., Stefan Schaal, A., & Bohg, J. (2017) · arXiv (Cornell University) · Cited by 47 · see citing works
Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural Networks.Preprint
Houthooft, R., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · Cited by 44 · see citing works
Physically consistent state estimation and system identification for contacts
Kolev, S., & Todorov, E. V. (2015) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2015.7363481 · Cited by 40 · see citing works
Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning
Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., & Funkhouser, T. (2018) · arXiv · doi:10.48550/arxiv.1803.09956
Ajay, A., Wu, J., Fazeli, N., Bauza, M., Kaelbling, L. P., Tenenbaum, J. B., & Rodriguez, A. (2018) · arXiv · doi:10.48550/arxiv.1808.03246
Self-citationCombining Learned and Analytical Models for Predicting Action Effects from Sensory Data
Kloss, A., Schaal, S., & Bohg, J. (2017) · arXiv · doi:10.48550/arxiv.1710.04102
Residual Reinforcement Learning for Robot Control
Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J. A., Solowjow, E., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1812.03201
Flagged first, then by citations
Research works printed but not linked: 5
- [2][2] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for visionbased robotic manipulation. In Conference on Robot Learning , pages 651-673, 2018. -
- [5][5] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems , pages 50485058, 2017. -
- [9][9] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep Q-learning with modelbased acceleration. In International Conference on Machine Learning , pages 2829-2838, 2016. -
- [13][13] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for modelbased deep reinforcement learning with model-free finetuning. In IEEE International Conference on Robotics and Automation (ICRA) , pages 7559-7566. IEEE, 2018. -
- [24][24] Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-toend differentiable physics for learning and control. In Advances in Neural Information Processing Systems , pages 7176-7187, 2018. -
Cited by
Sorted by citations
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570 · see citing works
Residual Reinforcement Learning for Robot Control
Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J. A., Solowjow, E., & Levine, S. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794127 · Cited by 480 · see citing works
TossingBot: Learning to Throw Arbitrary Objects With Residual Physics
Zeng, A., Song, S., Lee, J., Rodríguez, A., & Funkhouser, T. A. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2020.2988642 · Cited by 299 · see citing works
Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards
Schoettler, G., Nair, A., Luo, J., Bahl, S., Ojea, J. A., Solowjow, E., & Levine, S. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341714 · Cited by 191 · see citing works
Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks
Zhang, J. O., (Sasha), A. S., Zamir, A., Guibas, L. J., & Malik, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_41 · Cited by 190 · see citing works
Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
Johns, E. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9560942 · Cited by 119 · see citing works
TossingBot: Learning to Throw Arbitrary Objects with Residual Physics
Zeng, A., Song, S., Lee, J., Rodriquez, A., & A.Funkouser, T. (2019) · Robotics: Science and Systems · doi:10.15607/rss.2019.xv.004 · Cited by 110 · see citing works
Mohammed, M. Q., Kwek, L. C., & Chua, S. C. (2020) · IEEE Access · doi:10.1109/access.2020.3027923 · Cited by 108 · see citing works
Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks
Schoettler, G., Nair, A., Ojea, J. A., Levine, S., & Solowjow, E. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340848 · Cited by 88 · see citing works
Injecting knowledge in data-driven vehicle trajectory predictors
Bahari, M., Nejjar, I., & Alahi, A. (2021) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2021.103010 · Cited by 74 · see citing works
Ohn-Bar, E., Prakash, A., Behl, A., Chitta, K., & Geiger, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01131 · Cited by 75 · see citing works
Karkaria, V. N., Tsai, Y.-K., Chen, Y.-P., & Chen, W. (2025) · Engineering Optimization · doi:10.1080/0305215x.2024.2434201 · Cited by 78 · see citing works
Bi-Directional Domain Adaptation for Sim2Real Transfer of Embodied Navigation Agents
Truong, J., Chernova, S., & Batra, D. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3062303 · Cited by 78 · see citing works
Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning
Garcia-Hernando, G., Johns, E., & Kim, T. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340947 · Cited by 61 · see citing works
Development of residual learning in deep neural networks for computer vision: A survey
Xu, G., Wang, X., Wu, X., Leng, X., & Xu, Y. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109890 · Cited by 90 · see citing works
Lee, M. A., Florensa, C., Tremblay, J., Ratliff, N., Garg, A., Ramos, F. T., & Fox, D. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197125 · Cited by 57 · see citing works
Differentiable Algorithm Networks for Composable Robot Learning
Karkus, P., Ma, X., Hsu, D., Kaelbling, L. P., Lee, W. S., & Lozano‐Pérez, T. (2019) · Robotics: Science and Systems · doi:10.15607/rss.2019.xv.039 · Cited by 56 · see citing works
Self-citationInsertionNet - A Scalable Solution for Insertion
Spector, O., & Castro, D. D. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3076971 · Cited by 51 · see citing works
Challenges of Reinforcement Learning
Ding, Z., & Dong, H. (2020) · Springer eBooks · doi:10.1007/978-981-15-4095-0_7 · Cited by 52 · see citing works
Robot Learning of Mobile Manipulation With Reachability Behavior Priors
Jauhri, S., Peters, J., & Chalvatzaki, G. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3188109 · Cited by 49 · see citing works
Luo, Y., Wang, Y., Dong, K., Zhang, Q., Cheng, E., Sun, Z., & Song, B. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126620 · Cited by 43 · see citing works
Residual Policy Learning Facilitates Efficient Model-Free Autonomous Racing
Zhang, R., Hou, J. C., Chen, G., Li, Z., Chen, J., & Knoll, A. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3192770 · Cited by 42 · see citing works
Machine learning meets advanced robotic manipulation
Nahavandi, S., Alizadehsani, R., Nahavandi, D., Lim, C. P., Kelly, K., & Bello, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102221 · Cited by 42 · see citing works
A Comparison of Action Spaces for Learning Manipulation Tasks
Varin, P., Grossman, L., & Kuindersma, S. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8967946 · Cited by 37 · see citing works
Learning Hierarchical Control for Robust In-Hand Manipulation
Li, T., Srinivasan, K., Meng, M. Q. ‐., Yuan, W., & Bohg, J. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197343 · Cited by 38 · see citing works
Sample-efficient Reinforcement Learning in Robotic Table Tennis
Tebbe, J., Krauch, L., Gao, Y., & Zell, A. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560764 · Cited by 37 · see citing works
Adaptive Control of a Mechatronic System Using Constrained Residual Reinforcement Learning
Staessens, T., Lefebvre, T., & Crevecoeur, G. (2022) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3144565 · Cited by 33 · see citing works
TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task TransferPreprint
Allevato, A., Short, E. S., Pryor, M., & Thomaz, A. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.11200 · Cited by 34 · see citing works
Physical enhanced residual learning (PERL) framework for vehicle trajectory prediction
Long, K., Sheng, Z., Shi, H., Li, X., Chen, S., & Ahn, S. (2025) · Communications in Transportation Research · doi:10.1016/j.commtr.2025.100166 · Cited by 30 · see citing works
Affordance Learning from Play for Sample-Efficient Policy Learning
Diaz, J. L. B., Mees, O., Kalweit, G., Hermann, L., Boedecker, J., & Burgard, W. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811889 · Cited by 30 · see citing works
Bayesian controller fusion: Leveraging control priors in deep reinforcement learning for robotics
Rana, K., Dasagi, V., Haviland, J., Talbot, B., Milford, M. J., & Sünderhauf, N. (2023) · The International Journal of Robotics Research · doi:10.1177/02783649231167210 · Cited by 30 · see citing works
Residual Policy Learning for Shared Autonomy
Schaff, C., & Walter, M. R. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.072 · Cited by 30 · see citing works
Naunheim, S., Kuhl, Y., Schug, D., Schulz, V., & Mueller, F. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3323131 · Cited by 24 · see citing works
Funk, N., Schaff, C., Madan, R., Yoneda, T., Urain, J., Watson, J., Gordon, E. K., Widmaier, F., Bauer, S., Srinivasa, S. S., et al. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3129139 · Cited by 24 · see citing works
Evolving Rewards to Automate Reinforcement LearningPreprint
Faust, A., Francis, A. G., Jr., & Mehta, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07628 · Cited by 27 · see citing works
Rana, K., Talbot, B., Dasagi, V., Milford, M. J., & Sünderhauf, N. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197386 · Cited by 23 · see citing works
Hierarchical Policies for Cluttered-Scene Grasping With Latent Plans
Wang, L., Meng, X., Yu, X., & Fox, D. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3143198 · Cited by 22 · see citing works
Iterative residual tuning for system identification and sim-to-real robot learning
Allevato, A., Short, E. S., Pryor, M., & Thomaz, A. L. (2020) · Autonomous Robots · doi:10.1007/s10514-020-09925-w · Cited by 21 · see citing works
Deep Residual Reinforcement Learning based Autonomous Blimp Control
Liu, Y. T., Price, E., Black, M. J., & Ahmad, A. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981182 · Cited by 19 · see citing works
Search-based task and motion planning for hybrid systems: Agile autonomous vehicles
Ajanović, Z., Regolin, E., Shyrokau, B. N., Ćatić, H., Horn, M., & Ferrara, A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.105893 · Cited by 19 · see citing works
Hou, J. C., Chen, G., Li, Z., He, W., Gu, S., Knoll, A., & Jiang, C. (2023) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2023.3312647 · Cited by 19 · see citing works
Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
Wu, Y., Mozifian, M., & Shkurti, F. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561333 · Cited by 17 · see citing works
Residual Physics and Post-Posed Shielding for Safe Deep Reinforcement Learning Method
Zhang, Q., Mahbod, M. H. B., Chng, C.-B., Lee, P. S., & Chui, C. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3178084 · Cited by 17 · see citing works
Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Control
Dawood, M., Dengler, N., de Heuvel, J., & Bennewitz, M. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161492 · Cited by 14 · see citing works
Implicit Kinematic Policies: Unifying Joint and Cartesian Action Spaces in End-to-End Robot Learning
Ganapathi, A., Florence, P., Varley, J., Burns, K., Goldberg, K. Y., & Zeng, A. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812165 · Cited by 14 · see citing works
Residual Reinforcement Learning from DemonstrationsPreprint
Alakuijala, M., Dulac-Arnold, G., Mairal, J., Ponce, J., & Schmid, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08050 · Cited by 14 · see citing works
Review of Reinforcement Learning for Robotic Grasping: Analysis and Recommendations
Sekkat, H., Moutik, O., Ourabah, L., Elkari, B., Chaibi, Y., & Tchakoucht, T. A. (2023) · Statistics Optimization & Information Computing · doi:10.19139/soic-2310-5070-1797 · Cited by 17 · see citing works
RL-augmented MPC Framework for Agile and Robust Bipedal Footstep Locomotion Planning and Control
Bang, S. H., Jové, C. A., & Sentis, L. (2024) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids58906.2024.10769914 · Cited by 12 · see citing works
Sample-Efficient Learning for Industrial Assembly using Qgraph-bounded DDPG
Hoppe, S., Giftthaler, M., Krug, R., & Toussaint, M. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341390 · Cited by 12 · see citing works
Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertainty
Ranjbar, A., Vien, N. A., Ziesche, H., Boedecker, J., & Neumann, G. M. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636176 · Cited by 12 · see citing works
Lv, J., Feng, Y., Zhang, C., Zhao, S., Shao, L., & Lu, C. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.040 · Cited by 11 · see citing works
RESPRECT: Speeding-up Multi-Fingered Grasping With Residual Reinforcement Learning
Ceola, F., Rosasco, L., & Natale, L. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3363532 · Cited by 11 · see citing works
Pou, B., Smith, J., Quiñones, E., Martín, M., & Gratadour, . (2024) · Optics Express · doi:10.1364/oe.530254 · Cited by 11 · see citing works
Qu, G., Yu, C., Low, S. H., & Wierman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07476 · Cited by 11 · see citing works
Learning Vision-based Reactive Policies for Obstacle AvoidancePreprint
Aljalbout, E., Ji, C., Ritt, K., Ulmer, M., & Haddadin, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.16298 · Cited by 11 · see citing works
CusADi: A GPU Parallelization Framework for Symbolic Expressions and Optimal Control
Jeon, S. H., Hong, S., Lee, H. J., Khazoom, C., & Kim, S. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3512254 · Cited by 10 · see citing works
Adaptive policy learning for data-driven powertrain control with eco-driving
Kerbel, L., Ayalew, B., Ivančo, A., & Loiselle, K. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106489 · Cited by 10 · see citing works
Cascaded Compositional Residual Learning for Complex Interactive Behaviors
Kumar, K. N., Essa, I. A., & Ha, S. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3286171 · Cited by 10 · see citing works
Mulet, B. P., Smith, J., Quiñones, E., Martín, M., & Gratadour, . (2022) · Adaptive Optics Systems VIII · doi:10.1117/12.2627849 · Cited by 9 · see citing works
Wang, Y., Bai, Y., & Svinin, M. M. (2021) · International Journal of Control Automation and Systems · doi:10.1007/s12555-020-0412-y · Cited by 9 · see citing works
Leveraging control priors in deep reinforcement learning for robotics
Rana, K. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.238632 · Cited by 9 · see citing works
Research on Robot Massage Force Control Based on Residual Reinforcement Learning
Xiao, M., Zhang, T., Zou, Y., Chen, S., & Wu, W. R. (2023) · IEEE Access · doi:10.1109/access.2023.3347416 · Cited by 8 · see citing works
Liu, Q., Guo, Y., Deng, L., Liu, H., Li, D., & Sun, H. (2024) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2024.3454080 · Cited by 8 · see citing works
Design and Control of Roller Grasper V3 for In-Hand Manipulation
Yuan, S., Shao, L., Feng, Y., Sun, J., Xue, T., Yako, C. L., Bohg, J., & Salisbury, J. K. (2024) · IEEE Transactions on Robotics · doi:10.1109/tro.2024.3454388 · Cited by 8 · see citing works
Qu, G., Yu, C., Low, S. H., & Wierman, A. (2021) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc45484.2021.9683735 · Cited by 8 · see citing works
Grasp and Motion Planning for Dexterous Manipulation for the Real Robot ChallengePreprint
Yoneda, T., Schaff, C., Maeda, T., & Walter, M. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.02842 · Cited by 8 · see citing works
Deep Model Predictive Optimization
Sacks, J., Rana, R., Huang, K., Spitzer, A., Shi, G., & Boots, B. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611492 · Cited by 7 · see citing works
Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning
Urain, J., Liu, P., Li, A., D’Eramo, C., & Peters, J. (2021) · Robotics: Science and Systems · doi:10.15607/rss.2021.xvii.052 · Cited by 7 · see citing works
Trumpp, R., Javanmardi, E., Nakazato, J., Tsukada, M., & Caccamo, M. (2024) · 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801657 · Cited by 6 · see citing works
Bootstrapping Motor Skill Learning with Motion Planning
Abbatematteo, B., Rosen, E., Tellex, S., & Konidaris, G. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636661 · Cited by 6 · see citing works
Zhang, R., Chen, G., Hou, J. C., Li, Z., & Knoll, A. (2022) · IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI) · doi:10.1109/mfi55806.2022.9913862 · Cited by 6 · see citing works
Adapting Object-Centric Probabilistic Movement Primitives with Residual Reinforcement Learning
Carvalho, J., Koert, D., Daniv, M., & Peters, J. (2022) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids53995.2022.10000148 · Cited by 6 · see citing works
Reinforcement Learning Based User-Specific Shared Control Navigation in Crowds
Zhang, B., Holloway, C., & Carlson, T. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394139 · Cited by 6 · see citing works
Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative ModelsPreprint
Wu, Y., Mozifian, M., & Shkurti, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01298 · Cited by 6 · see citing works
Wang, C., Su, C., Sun, B., Chen, G., & Xie, L. (2024) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2024.1355170 · Cited by 5 · see citing works
Lv, J., Feng, Y., Zhang, C., Zhao, S., Shao, L., & Lu, C. (2024) · The International Journal of Robotics Research · doi:10.1177/02783649241284653 · Cited by 5 · see citing works
Residual Model Learning for Microrobot Control
Gruenstein, J., Chen, T., Doshi, N., & Agrawal, P. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560862 · Cited by 6 · see citing works
Learning hierarchical control for robust in-hand manipulation
Li, T. (2022) · Elsevier eBooks · doi:10.1016/b978-0-32-390445-2.00017-9 · Cited by 6 · see citing works
Tolerance-Guided Policy Learning for Adaptable and Transferrable Delicate Industrial InsertionPreprint
Niu, B., Wang, C., & Liu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02303 · Cited by 5 · see citing works
Abbas, A. N., Chasparis, G. C., & Kelleher, J. D. (2024) · Research Square · doi:10.21203/rs.3.rs-3918353/v1 · Cited by 5 · see citing works
Doan, H. T. L., Arita, H., & Tahara, K. (2024) · ROBOMECH Journal · doi:10.1186/s40648-024-00273-3 · Cited by 4 · see citing works
Mixtures of Experts for Scaling up Neural Networks in Order Execution
Li, K., Cucuringu, M., Sánchez-Betancourt, L., & Willi, T. (2024) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3677052.3698691 · Cited by 4 · see citing works
Lee, G., Hou, B., Choudhury, S., & Srinivasa, S. S. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636163 · Cited by 5 · see citing works
Efficient Multi-Task Learning via Iterated Single-Task Transfer
Zentner, K. R., Puri, U., Zhang, Y., Julian, R., & Sukhatme, G. S. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981244 · Cited by 4 · see citing works
Residual Policy Learning for Powertrain Control
Kerbel, L., Ayalew, B., Ivančo, A., & Loiselle, K. (2022) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2022.10.270 · Cited by 4 · see citing works
An offline learning of behavior correction policy for vision-based robotic manipulation
Dong, Q., Kaneko, T., & Sugiyama, M. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610177 · Cited by 3 · see citing works
Residual reinforcement learning for logistics cart transportation
Matsuo, R., Yasuda, S., Kumagai, T., Sato, N., Yoshida, H., & Yairi, T. (2022) · Advanced Robotics · doi:10.1080/01691864.2022.2046504 · Cited by 3 · see citing works
HR-Planner: A Hierarchical Highway Tactical Planner based on Residual Reinforcement Learning
Wu, H., Li, Y., Zhuang, H., Wang, C., & Yang, M. Z. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812400 · Cited by 3 · see citing works
Autonomous Blimp Control via $H_{\infty}$ Robust Deep Residual Reinforcement Learning
Zuo, Y., Liu, Y. T., & Ahmad, A. (2023) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case56687.2023.10260561 · Cited by 4 · see citing works
PeRP: Personalized Residual Policies For Congestion Mitigation Through Co-operative Advisory Systems
Hasan, A., Chakraborty, N., Chen, H., Cho, J.-H., Wu, C., & Driggs-Campbell, K. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10422444 · Cited by 3 · see citing works
Long-Horizon Prediction and Uncertainty Propagation with Residual Point Contact Learners
Fazeli, N., Ajay, A., & Rodríguez, A. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196511 · Cited by 3 · see citing works
Vessel-to-Vessel Motion Compensation with Reinforcement Learning
Herland, S., & Bach, K. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.26860 · Cited by 2 · see citing works
CLAMGen: Closed-Loop Arm Motion Generation via Multi-view Vision-Based RL
Akinola, I., Wang, Z., & Allen, P. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636369 · Cited by 2 · see citing works
Robust nonlinear set-point control with reinforcement learning
Zhang, R., Mattsson, P., & Wigren, T. (2023) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc55779.2023.10156038 · Cited by 2 · see citing works
Model Based Residual Policy Learning with Applications to Antenna Control
Möllerstedt, V. E., Russo, A., & Bouton, M. (2024) · IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN) · doi:10.1109/icmlcn59089.2024.10624756 · Cited by 2 · see citing works
Cooperative Advisory Residual Policies for Congestion Mitigation
Hasan, A., Chakraborty, N., Chen, H., Cho, J.-H., Wu, C., & Driggs-Campbell, K. (2024) · ACM Journal on Autonomous Transportation Systems · doi:10.1145/3699519 · Cited by 2 · see citing works
Towards a Manipulator System for Disposal of Waste from Patients Undergoing Chemotherapy
Li, H.-Y., Ho, L. S., Athukorala, A., Lu, W. Y., Dharmawan, A. G., Guo, J. L. F., Tan, M. M. L., Wong, K. C., Ng, N. S., Tan, M. M. X., et al. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636865 · Cited by 2 · see citing works
Aiding reinforcement learning for set point control
Zhang, R., Mattsson, P., & Wigren, T. (2023) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2023.10.1220 · Cited by 2 · see citing works
Optimizing Attention and Cognitive Control Costs Using Temporally Layered Architectures
Patel, D., B, T. J. S. A., & Siegelmann, H. T. (2024) · Neural Computation · doi:10.1162/neco_a_01718 · Cited by 2 · see citing works
Elaborating on Learned Demonstrations with Temporal Logic Specifications
Innes, C., & Ramamoorthy, S. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.004 · Cited by 2 · see citing works
- Authors
- Silver, T., Allen, K., Tenenbaum, J., & Kaelbling, L.
- Affiliations
- Massachusetts Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1812.06298
- OpenAlex
- W2905364877
- Cited by
- 113
- References
- 40
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 21 · 2023: 20 · 2022: 16 · 2021: 25 · 2020: 20 · 2019: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1723381, 1523767, 1231216)
- Companhia Brasileira de Metalurgia e Mineração (CCF-1231216)
- Natural Sciences and Engineering Research Council of Canada
- Office of Naval Research (N00014-18, N000141310333, N00014-18-1-2847, FA9550, N00014)
- Air Force Office of Scientific Research (FA9550-17, FA9550-, FA9550-17-1, FA9550-17-1-0165)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Keywords
- model-free deep reinforcement learning, robotic manipulation, model predictive control, sparse rewards, long-horizon tasks, sensor noise, model misspecification, partial observability, MuJoCo
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
