CoChat
Sign Up
PreprintCited by 403Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

Sim-to-Real Robot Learning from Pixels with Progressive Nets

Open access
Rusu, Andrei A., Vecerik, Mel, Rothörl, Thomas, Heess, Nicolas, Pascanu, Razvan, Hadsell, Raia·2016·arXiv
DOI 10.48550/arxiv.1610.04286arXiv 1610.04286
Open this paper in CoChat — free accountOpen PDF

Abstract

Applying end-to-end learning to solve complex, interactive, pixel-driven control tasks on a robot is an unsolved problem. Deep Reinforcement Learning algorithms are too slow to achieve performance on a real robot, but their potential has been demonstrated in simulated environments. We propose using progressive networks to bridge the reality gap and transfer learned policies from simulation to the real world. The progressive net approach is a general framework that enables reuse of everything from low-level visual features to high-level policies for transfer to new tasks, enabling a compositional, yet simple, approach to building complex skills. We present an early demonstration of this approach with a number of experiments in the domain of robot manipulation that focus on bridging the reality gap. Unlike other proposed approaches, our real-world experiments demonstrate successful task learning from raw visual input on a fully actuated robot manipulator. Moreover, rather than relying on model-based trajectory optimisation, the task learning is accomplished using only deep reinforcement learning and sparse rewards.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1610.04286

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The assumption that real observations should match simulated ones when the consequences of a control input applied in simulation differ from those applied on the real robot is not always true.

    “However, the authors assume that, when the consequences of a control input applied in simulation differ from those applied on the real robot, real observations should match simulated ones, and that is not always true.”

    From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [49]

    The Experiments section (p. 4) says that visual discrepancies between simulated and real observations are obvious, rather than assuming they should match. The Introduction (p. 2) describes training a new network on the real robot after training in simulation. The paper does not make the claimed assumption about observations when control inputs have different consequences. (LLM, conf=0.83)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Christiano et al. [49] adopted TL to perform sim-to-real control input adaptation.

    “Christiano et al. [49] adopted TL to perform sim-to-real control input adaptation.”

    From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [49]

    The Introduction (pp. 1–2) says the paper uses transfer learning to move a pixel-to-action robot control policy from simulation to a real robot. But the title page (p. 1) names Rusu et al., not Christiano et al., as the authors, and the paper describes policy transfer rather than specifically calling it “control input adaptation.” (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Results in [49] highlight the effectiveness of such method and the relatively low number of samples required for convergence.

    “Results highlight the effectiveness of such method and the relatively low number of samples required for convergence.”

    From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [49]

  • The methods of imitation learning can be broadly classified into two key areas: behaviour cloning and inverse reinforcement learning.

    “The methods of imitation learning can be broadly classified into two key areas: behaviour cloning where an agent learns a mapping from observations to actions given demonstrations [26], [27] and inverse reinforcement learning where an agent attempts to estimate a reward function that describes the given demonstrations [28].”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [26]

    The Introduction (pp. 1–2) and Related Literature (pp. 3–4) discuss deep reinforcement learning and sim-to-real transfer, not a classification of imitation-learning methods. Searches for behaviour cloning, inverse reinforcement learning, and learning actions from demonstrations found no passage in our copy addressing the attributed claim. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 25

  1. [1][1] S. Levine and P. Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27 , pages 1071-1079. Curran Associates, Inc., 2014. URL http://papers.nips.cc/paper/ 5444-learning-neural-network-policies-with-guided-policy-search-under-unknown-dynamics. pdf . -
  2. [2][2] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , 2015. -
  3. [3][3] N. Heess, G. Wayne, D. Silver, T. P. Lillicrap, T. Erez, and Y. Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 2944-2952, 2015. URL http://papers.nips.cc/paper/ 5796-learning-continuous-control-policies-by-stochastic-value-gradients . -
  4. [4][4] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. Proceedings of the International Conference on Learning Representations (ICLR) , 2016. URL http://arxiv.org/abs/1509.02971 . - · arXiv
  5. [5][5] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR) , 2016. -
  6. [6][6] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Int'l Conf. on Machine Learning (ICML) , 2016. -
  7. [7][7] S. Gu, T. P. Lillicrap, I. Sutskever, and S. Levine. Continuous deep q-learning with model-based acceleration. In ICML 2016 , 2016. -
  8. [8][8] A. Rusu, N. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 , 2016. - · arXiv
  9. [9][9] X. Peng, B. Sun, K. Ali, and K. Saenko. Learning deep object detectors from 3d models. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015 , pages 1278-1286, 2015. -
  10. [10][10] H. Su, C. R. Qi, Y. Li, and L. J. Guibas. Render for CNN: viewpoint estimation in images using cnns trained with rendered 3d model views. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015 , pages 2686-2694, 2015. -
  11. [11][11] M. Long, Y. Cao, J. Wang, and M. I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 , pages 97-105, 2015. -
  12. [12][12] E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko. Simultaneous deep transfer across domains and tasks. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015 , pages 4068-4076, 2015. -
  13. [14][14] E. Tzeng, C. Devin, J. Hoffman, C. Finn, X. Peng, S. Levine, K. Saenko, and T. Darrell. Towards adapting deep visuomotor representations from simulated to real environments. CoRR , abs/1511.07111, 2015. URL http://arxiv.org/abs/1511.07111 . - · arXiv
  14. [15][15] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research , 17(59):1-35, 2016. -
  15. [16][16] H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, and M. Marchand. Domain-adversarial neural networks. CoRR , abs/1412.4446, 2014. URL http://arxiv.org/abs/1412.4446 . - · arXiv
  16. [17][17] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan. Domain separation networks. In Advances in Neural Information Processing Systems , pages 343-351, 2016. -
  17. [18][18] S. Barrett, M. E. Taylor, and P. Stone. Transfer learning for reinforcement learning on a physical robot. In Ninth International Conference on Autonomous Agents and Multiagent Systems - Adaptive Learning Agents Workshop (AAMAS - ALA) , 2010. -
  18. [19][19] S. James and E. Johns. 3D Simulation for Robot Arm Control with Deep Q-Learning. ArXiv e-prints , 2016. -
  19. [20][20] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Robotics and Automation (ICRA), 2017 IEEE International Conference on , pages 3357-3364. IEEE, 2017. -
  20. [21][21] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research , 17(39):1-40, 2016. -
  21. [22][22] S. Levine, N. Wagener, and P. Abbeel. Learning contact-rich manipulation skills with guided policy search. In IEEE International Conference on Robotics and Automation, ICRA 2015, Seattle, WA, USA, 26-30 May, 2015 , pages 156-163, 2015. -
  22. [23][23] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In ICRA 2016 , 2016. -
  23. [24][24] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research , page 0278364917710318, 2016. -
  24. [25][25] V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A. Graves, M. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015. -
  25. [26][26] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems IROS , 2012.

Cited by

Sorted by citations

Related

Authors
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2016
Pages
262-270
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1610.04286
arXiv
1610.04286
OpenAlex
W2534269850
Cited by
385
References
0
Citations by year
2026: 2 · 2025: 14 · 2024: 23 · 2023: 21 · 2022: 36 · 2021: 66 · 2020: 68 · 2019: 72 · 2018: 52 · 2017: 29 · 2016: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Conference on Robot Learning
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
sim-to-real transfer, robotic manipulation, visual features, policy transfer, reality gap, deep reinforcement learning, sparse rewards, end-to-end learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review