CoChat
Sign Up
Cited by 480Reinforcement Learning in RoboticsRobot Manipulation and LearningAdaptive Dynamic Programming Control

Residual Reinforcement Learning for Robot Control

Open access
Johannink, Tobias, Bahl, Shikhar, Nair, Ashvin, Luo, Jianlan, Kumar, Avinash, Loskyll, Matthias, Ojea, Juan Aparicio, Solowjow, Eugen +1 more·2019·2019 International Conference on Robotics and Automation (ICRA)
DOI 10.1109/icra.2019.8794127
Open this paper in CoChat — free accountOpen PDF

Abstract

Conventional feedback control methods can solve various types of robot control problems very efficiently by capturing the structure with explicit models, such as rigid body equations of motion. However, many control problems in modern manufacturing deal with contacts and friction, which are difficult to capture with first-order physical modeling. Hence, applying control design methodologies to these kinds of problems often results in brittle and inaccurate controllers, which have to be manually tuned for deployment. Reinforcement learning (RL) methods have been demonstrated to be capable of learning continuous robot controllers from interactions with the environment, even for problems that include friction and contacts. In this paper, we study how we can solve difficult control problems in the real world by decomposing them into a part that is solved efficiently by conventional feedback control methods, and the residual which is solved with RL. The final control policy is a superposition of both control signals. We demonstrate our approach by training an agent to successfully perform a real-world block assembly task involving contacts and unstable objects.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icra.2019.8794127

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A single residual-RL policy trained jointly on all demonstrations repairs kinematics instead of learning tasks, so it needs no per-task training and can generate additional physically consistent variants of each demonstration.

    “the key design is a single residual-RL policy (Johannink et al., 2019;Silver et al., 2018) trained jointly across all demonstrations, which repairs kinematics rather than learning tasks, and therefore needs no per-task training and can synthesize additional physically consistent variants of each demonstration.”

    From Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning (2026) · cites this paper as Johannink et al., 2019

    The Method Summary describes a learned residual policy added to a fixed controller, but not a policy trained jointly on demonstrations. The Related Work passage distinguishes this method from learning through demonstrations, while the Environment Variation results report training corrective behavior for a specific block-insertion task. Those passages contradict the claim that the policy repairs demonstrations rather than learning tasks and needs no per-task training; our copy does not describe generating variants of demonstrations. (LLM, conf=0.85)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Residual RL was introduced with [10] for a robot manipulator.

    “Introduced with [10] for a robot manipulator, this method consists of complementing a traditional type of controller with a RL policy”

    From RLPP: A Residual Method for Zero-Shot Real-World Autonomous Racing on Scaled Platforms (2025) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 26

  1. S. Fujimoto, H. van Hoof, and D. Meger, "Addressing Func- tion Approximation Error in Actor-Critic Methods," arXiv preprint arXiv:1802.09477, 2018. · arXiv
  2. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, "Continuous control with deep reinforcement learning," in International Conference on Learning Representations (ICLR), 2016.
  3. V. Pong, S. Gu, M. Dalal, and S. Levine, "Temporal Difference Models: Model-Free Deep RL For Model-Based Control," in International Conference on Learning Representations (ICLR), 2018.
  4. E. Todorov, T. Erez, and Y. Tassa, "MuJoCo: A physics engine for model-based control," in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012, pp. 5026-5033.
  5. J. Peters, K. Mülling, and Y. Altün, "Relative Entropy Policy Search," in AAAI Conference on Artificial Intelligence, 2010, pp. 1607-1612.
  6. M. P. Deisenroth and C. E. Rasmussen, "PILCO: A model-based and data-efficient approach to policy search," in International Conference on Machine Learning (ICML), 2011, pp. 465-472.
  7. S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, "Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection," International Journal of Robotics Research, 2017.
  8. S. Gu, E. Holly, T. Lillicrap, and S. Levine, "Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates," IEEE International Conference on Robotics and Automation (ICRA), 2017.
  9. S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-End Training of Deep Visuomotor Policies," Journal of Machine Learning Research (JMLR), vol. 17, no. 1, pp. 1334-1373, 2016.
  10. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor," in International Conference on Machine Learning, 2018.
  11. A. Y. Ng, D. Harada, and S. Russell, "Policy invariance under reward transformations: Theory and application to reward shaping," in International Conference on Machine Learning (ICML), 1999.
  12. I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-maron, M. Vecerik, T. Lampe, Y. Tassa, T. Erez, M. Riedmiller, and M. R. Deepmind, "Data-efficient Deep Reinforcement Learning for Dexterous Manipulation," CoRR, vol. abs/1704.0, 2017.
  13. G. Thomas, M. Chien, A. Tamar, J. A. Ojea, and P. Abbeel, "Learning Robotic Assembly from CAD," in IEEE International Conference on Robotics and Automation (ICRA), 2018.
  14. R. Loftin, J. Macglashan, B. Peng, M. E. Taylor, M. L. Littman, J. Huang, and D. L. Roberts, "A Strategy-Aware Technique for Learn- ing Behaviors from Discrete Human Feedback," in AAAI Conference on Artificial Intelligence, 2014, pp. 937-943.
  15. W. Saunders, G. Sastry, A. Stuhlmüller, and O. Evans, "Trial without Error: Towards Safe Reinforcement Learning via Human Intervention," in International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2018.
  16. L. Torrey and M. E. Taylor, "Teaching on a Budget: Agents Advising Agents in Reinforcement Learning," in International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2013.
  17. J. Frank, S. Mannor, and D. Precup, "Reinforcement Learning in the Presence of Rare Events," in International Conference on Machine Learning (ICML), 2008.
  18. P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learning from human preferences," in Advances in Neural Information Processing Systems (NIPS), 2017.
  19. J. Peters and S. Schaal, "Reinforcement learning of motor skills with policy gradients," Neural Networks, vol. 21, no. 4, pp. 682-697, 2008.
  20. J. Kober and J. Peter, "Policy search for motor primitives in robotics," in Advances in Neural Information Processing Systems (NIPS), vol. 97, 2008, pp. 83-117.
  21. T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, J. Z. Leibo, and A. Gruslys, "Learning from Demonstrations for Real World Reinforcement Learning," in AAAI Conference on Artificial Intelligence, 2018.
  22. M. Večerík, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller, "Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards," CoRR, vol. abs/1707.0, 2017.
  23. K. Subramanian, C. L. Isbell, and A. L. Thomaz, "Exploration from Demonstration for Interactive Reinforcement Learning," in International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2016.
  24. M. Srouji, J. Zhang, and R. Salakhutdinov, "Structured Control Nets for Deep Reinforcement Learning," in International Conference on Machine Learning (ICML), 2018.
  25. F. Wang and J. Decker, "Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator," arXiv preprint arXiv:1803.10228, 2018. · arXiv
  26. N. Sünderhauf, O. Brock, W. Scheirer, R. Hadsell, D. Fox, J. Leitner, B. Upcroft, P. Abbeel, W. Burgard, M. Milford, and P. Corke, "The Limits and Potentials of Deep Learning for Robotics," International Journal of Robotics Research, vol. 37, no. 4-5, pp. 405-420, 2018.

Cited by

Sorted by citations

Authors
Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J. A., Solowjow, E., & Levine, S.
Affiliations
  • Hamburg University of Technology (DE)
  • University of California, Berkeley (US)
ORCID
Venue
2019 International Conference on Robotics and Automation (ICRA)
Publisher
IEEE
Published
2019-05
Pages
6023-6029
ISSN
1050-4729 (ISSN-L)
DOI
10.1109/icra.2019.8794127
OpenAlex
W2904746163
Cited by
480
References
4
Citations by year
2026: 80 · 2025: 61 · 2024: 77 · 2023: 73 · 2022: 72 · 2021: 66 · 2020: 41 · 2019: 8 · 2018: 2
FWCI
17.89
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • tub.dok (Hamburg University of Technology) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Siemens USA
  • Office of Naval Research
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
Keywords
residual reinforcement learning, hybrid control, feedback control, robot control, friction, contact dynamics, block assembly, manufacturing robotics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review