The DOI matches its DataCite record; 43 of the 43 references it cites were found in a registry or an index.
Residual Policy Learning
Open accessAbstract
We present Residual Policy Learning (RPL): a simple method for improving nondifferentiable policies using model-free deep reinforcement learning. RPL thrives in complex robotic manipulation tasks where good but imperfect controllers are available. In these tasks, reinforcement learning from scratch remains data-inefficient or intractable, but learning a residual on top of the initial controller can yield substantial improvements. We study RPL in six challenging MuJoCo tasks involving partial observability, sensor noise, model misspecification, and controller miscalibration. For initial controllers, we consider both hand-designed policies and model-predictive controllers with known or learned transition models. By combining learning with control algorithms, RPL can perform long-horizon, sparse-reward tasks for which reinforcement learning alone fails. Moreover, we find that RPL consistently and substantially improves on the initial controllers. We argue that RPL is a promising approach for combining the complementary strengths of deep reinforcement learning and robotic control, pushing the boundaries of what either can achieve independently. Video and code at https://k-r-allen.github.io/residual-policy-learning/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the key design is a single residual-RL policy (Johannink et al., 2019;Silver et al., 2018) trained jointly across all demonstrations, which repairs kinematics rather than learning tasks, and therefore needs no per-task training and can synthesize additional physically consistent variants of each demonstration.”
From Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning (2026) · cites this paper as Silver et al., 2018
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
The unscented Kalman filter for nonlinear estimation
Wan, E. A., & van der Merwe, R. (2002) · IEEE Adaptive Systems for Signal Processing, Communications, and Control Symposium (AS-SPCC) · doi:10.1109/asspcc.2000.882463 · Cited by 4,373
Apprenticeship learning via inverse reinforcement learning
Abbeel, P., & Ng, A. Y. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015430 · Cited by 2,855
Soft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,868
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815
Research works printed but not linked: 5
- [2][2] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for visionbased robotic manipulation. In Conference on Robot Learning , pages 651-673, 2018. -
- [5][5] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems , pages 50485058, 2017. -
- [9][9] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep Q-learning with modelbased acceleration. In International Conference on Machine Learning , pages 2829-2838, 2016. -
- [13][13] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for modelbased deep reinforcement learning with model-free finetuning. In IEEE International Conference on Robotics and Automation (ICRA) , pages 7559-7566. IEEE, 2018. -
- [24][24] Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-toend differentiable physics for learning and control. In Advances in Neural Information Processing Systems , pages 7176-7187, 2018. -
Cited by
Sorted by citations
Items 101–114 of 114
A Simple Approach to Continual Learning by Transferring Skill ParametersPreprint
Zentner, K. R., Julian, R., Puri, U., Zhang, Y., & Sukhatme, G. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10255 · Cited by 2
Solving the Real Robot Challenge using Deep Reinforcement LearningPreprint
McCarthy, R., Sánchez, F. R., Wang, Q., Bulens, D. C., McGuinness, K., O'Connor, N. E., & Redmond, S. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.15233 · Cited by 3
Machine Learning Meets Advanced Robotic ManipulationPreprint
Nahavandi, S., Alizadehsani, R., Nahavandi, D., Lim, C. P., Kelly, K., & Bello, F. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4587790 · Cited by 2
Generalizing Cooperative Eco-driving via Multi-residual Task Learning
Jayawardana, V., Li, S., Wu, C., Farid, Y. Z., & Oguchi, K. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610586 · Cited by 1
Bootstrapping Motor Skill Learning with Motion PlanningPreprint
Abbatematteo, B., Rosen, E., Tellex, S., & Konidaris, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04736 · Cited by 1
Yang, Z., Liu, C., Zhou, Y., Lv, R., Cui, T., Yi, S., Zhu, H., Lu, I., & Sun, J. (2026) · arXiv · doi:10.48550/arxiv.2609.24093 · Cited by 2
“…et al., 2017), letting the physics engine supply the missing contact forces; the key design is a single residual‑RL policy (Johannink et al., 2019; Silver et al., 2018) trained jointly across all demonstrations, which repairs kinematics rather than learning tasks, and therefore needs no…”
InsertionNet -- A Scalable Solution for InsertionPreprint
Spector, O., & Castro, D. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14223
Real Robot Challenge using Deep Reinforcement Learning.
McCarthy, R., Sánchez, F. R., McGuinness, K., O'Connor, N. E., & Redmond, S. J. (2021)
Soft Action Priors: Towards Robust Policy Transfer
Centa, M., & Preux, P. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25850
Self-Residual Deep Reinforcement Learning for Volt-Var Control in Active Distribution Networks
Liu, Y., Liu, Q., Chen, M., Luo, L., Guo, Y., & Wang, K. (2024) · International Conference on Energy Systems and Electrical Power (ICESEP) · doi:10.1109/icesep62218.2024.10651654
General Automatic Solution Generation for Social Problems
NIu, T., Huang, H., Du, Y. H., Zhang, W., Shi, L., & Zhao, R. J. (2025) · Machine Intelligence Research · doi:10.1007/s11633-024-1496-2
Adaptive Control for Energy Optimization in Data Center
Farinda, R., Palitai, K., Popiołek, K., & Seganti, A. (2026) · ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys) · doi:10.1145/3744256.3812552
Boase, D., Miah, M. S., & Gueaieb, W. (2026) · International Journal of Intelligent Robotics and Applications · doi:10.1007/s41315-026-00560-9
Metadata
- Authors
- Silver, T., Allen, K., Tenenbaum, J., & Kaelbling, L.
- Affiliations
- Massachusetts Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1812.06298
- OpenAlex
- W2905364877
- Cited by
- 113
- References
- 40
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 21 · 2023: 20 · 2022: 16 · 2021: 25 · 2020: 20 · 2019: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1723381, 1523767, 1231216)
- Companhia Brasileira de Metalurgia e Mineração (CCF-1231216)
- Natural Sciences and Engineering Research Council of Canada
- Office of Naval Research (N00014-18, N000141310333, N00014-18-1-2847, FA9550, N00014)
- Air Force Office of Scientific Research (FA9550-17, FA9550-, FA9550-17-1, FA9550-17-1-0165)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Keywords
- model-free deep reinforcement learning, robotic manipulation, model predictive control, sparse rewards, long-horizon tasks, sensor noise, model misspecification, partial observability, MuJoCo
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
