CoChat
Sign Up
PreprintCited by 208Robot Manipulation and LearningReinforcement Learning in RoboticsDomain Adaptation and Few-Shot Learning

Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task

Open access
James, Stephen, Davison, Andrew J., Johns, Edward·2017·arXiv
DOI 10.48550/arxiv.1707.02267arXiv 1707.02267
Open this paper in CoChat — free accountOpen PDF

Abstract

End-to-end control for robot manipulation and grasping is emerging as an attractive alternative to traditional pipelined approaches. However, end-to-end methods tend to either be slow to train, exhibit little or no generalisability, or lack the ability to accomplish long-horizon or multi-stage tasks. In this paper, we show how two simple techniques can lead to end-to-end (image to velocity) execution of a multi-stage task, which is analogous to a simple tidying routine, without having seen a single real image. This involves locating, reaching for, and grasping a cube, then locating a basket and dropping the cube inside. To achieve this, robot trajectories are computed in a simulator, to collect a series of control velocities which accomplish the task. Then, a CNN is trained to map observed images to velocities, using domain randomisation to enable generalisation to real world images. Results show that we are able to successfully accomplish the task in the real world with the ability to generalise to novel environments, including those with dynamic lighting conditions, distractor objects, and moving objects, including the basket itself. We believe our approach to be simple, highly scalable, and capable of learning long-horizon tasks that have until now not been shown with the state-of-the-art in end-to-end robot control.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1707.02267

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Obtained results in [54] suggest that simulators improvement is an essential requirement since, as the model becomes less adequate, not even a robust controller is able to avoid a large RG.

    “Obtained results suggest that simulators improvement is an essential requirement since, as the model becomes less adequate, not even a robust controller is able to avoid a large RG.”

    From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [54]

    The Data Collection section (p. 4) says domain randomisation can overcome the reality gap, rather than saying simulator improvement is essential. The Experiments section (p. 7) finds that a baseline without domain randomisation transfers poorly; it does not show that a robust controller cannot avoid a large gap as the simulator becomes less adequate. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Evaluations in [54] have shown that controllers learnt in simulation with large observation space lead to bad results when transferred to real robot.

    “On the contrary, their evaluations have shown that controllers learnt in simulation with large observation space lead to bad results when transferred to real robot.”

    From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [54]

    The ablation study (4 Experiments, pp. 7–8) found that a baseline controller trained in simulation failed the full task on the real robot. But it attributes that failure to the absence of domain randomisation, not to a large observation space; the fully randomised controller completed all standard real-world trials. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Authors of [54] pointed out that considering a large observation space does not always bring benefits.

    “The authors of the cited paper also pointed out that considering a large observation space does not always bring bene ts.”

    From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [54]

    The Experiments section (pp. 6–8) compares performance with and without visual distractors and reports that adding joint-angle inputs helps. Neither comparison addresses whether a larger observation space is beneficial. Searches for “observation space” and larger visual inputs found no passage making the attributed point. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. [1][1] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research , 2016. -
  2. [2][2] W. H. Montgomery and S. Levine. Guided policy search via approximate mirror descent. Advances in Neural Information Processing Systems (NIPS) , 2016. -
  3. [3][3] W. Montgomery, A. Ajay, C. Finn, P. Abbeel, and S. Levine. Reset-free guided policy search: Efficient deep reinforcement learning with stochastic initial states. International Conference on Robotics and Automation (ICRA) , 2016. -
  4. [4][4] I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-Maron, M. Vecerik, T. Lampe, Y. Tassa, T. Erez, and M. Riedmiller. Data-efficient deep reinforcement learning for dexterous manipulation. arXiv preprint arXiv:1704.03073 , 2017. - · arXiv
  5. [5][5] S. James and E. Johns. 3d simulation for robot arm control with deep q-learning. NIPS 2016 Workshop (Deep Learning for Action and Interaction) , 2016. -
  6. [6][6] F. Zhang, J. Leitner, M. Milford, B. Upcroft, and P. Corke. Towards vision-based deep reinforcement learning for robotic motion control. Australasian Conference on Robotics and Automation (ACRA) , 2015. -
  7. [7][7] I. Higgins, A. Pal, A. A. Rusu, L. Matthey, C. P. Burgess, A. Pritzel, M. Botvinick, C. Blundell, and A. Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. International Conference on Machine Learning (ICML) , 2018. -
  8. [8][8] M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. International Conference for Learning Representations (ICLR) , 2016. -
  9. [9][9] N. Dilokthanakul, C. Kaplanis, N. Pawlowski, and M. Shanahan. Feature control as intrinsic motivation for hierarchical reinforcement learning. arXiv preprint arXiv:1705.06769 , 2017. - · arXiv
  10. [10][10] F. Sadeghi and S. Levine. (cad)2rl: Real single-image flight without a single real image. Robotics: Science and Systems Conference (R:SS) , 2016. -
  11. [11][11] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. Intelligent Robots and Systems (IROS) , 2017. -
  12. [14][14] S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. International Conference on Robotics and Automation (ICRA) , 2017. -
  13. [16][16] Y. Duan, M. Andrychowicz, B. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba. One-shot imitation learning. arXiv preprint arXiv:1703.07326 , 2017. - · arXiv
  14. [17][17] A. X. Lee, A. Gupta, H. Lu, S. Levine, and P. Abbeel. Learning from multiple demonstrations using trajectory-aware non-rigid registration with applications to deformable object manipulation. Intelligent Robots and Systems (IROS) , 2015. -
  15. [18][18] K. Hausman, Y. Chebotar, S. Schaal, G. Sukhatme, and J. Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. Conference on Neural Information Processing Systems (NIPS) , 2017. -
  16. [19][19] B. C. Stadie, P. Abbeel, and I. Sutskever. Third-person imitation learning. International Conference on Learning Representations (ICLR) , 2017. -
  17. [20][20] S. Levine and V. Koltun. Guided policy search. International Conference on Machine Learning (ICML) , 2013. -
  18. [21][21] S. Levine and P. Abbeel. Learning neural network policies with guided policy search under unknown dynamics. Advances in Neural Information Processing Systems (NIPS) , 2014. -
  19. [22][22] E. Tzeng, C. Devin, J. Hoffman, C. Finn, P. Abbeel, S. Levine, K. Saenko, and T. Darrell. Adapting deep visuomotor representations with weak pairwise constraints. Workshop on the Algorithmic Foundations of Robotics (WAFR) , 2016. -
  20. [23][23] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. International Conference on Robotics and Automation (ICRA) , 2016. -
  21. [24][24] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. International Symposium on Experimental Robotics (ISER) , 2016. -
  22. [25][25] C. Finn and S. Levine. Deep visual foresight for planning robot motion. International Conference on Robotics and Automation (ICRA) , 2017. -
  23. [26][26] E. Johns, S. Leutenegger, and A. J. Davison. Deep learning a grasp function for grasping under gripper pose uncertainty. Intelligent Robots and Systems (IROS) , 2016. -
  24. [27][27] P. Christiano, Z. Shah, I. Mordatch, J. Schneider, T. Blackwell, J. Tobin, P. Abbeel, and W. Zaremba. Transfer from simulation to real world through learning deep inverse dynamics model. arXiv preprint arXiv:1610.03518 , 2016. - · arXiv
  25. [28][28] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. Robotics and Automation (ICRA) , 2017. -
  26. [29][29] A. Gupta, C. Devin, Y. Liu, P. Abbeel, and S. Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. International Conference on Learning Representations (ICLR) , 2017. -
  27. [30][30] M. F. E. Rohmer, S. P. N. Singh. V-rep: a versatile and scalable robot simulation framework. International Conference on Intelligent Robots and Systems (IROS) , 2013. -
  28. [31][31] K. Perlin. Improving noise. ACM Transactions on Graphics (TOG) , 2002. -
  29. [32][32] D. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference for Learning Representations (ICLR) , 2015.

Cited by

Sorted by citations

Authors
James, S., Davison, A. J., & Johns, E.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Pages
334-343
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1707.02267
arXiv
1707.02267
OpenAlex
W2725320964
Cited by
208
References
2
Citations by year
2025: 6 · 2024: 14 · 2023: 20 · 2022: 18 · 2021: 42 · 2020: 44 · 2019: 39 · 2018: 22 · 2017: 2 · 2016: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion, cc-by)
  • arXiv (Cornell University)
  • Conference on Robot Learning
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
robotic manipulation, robotic grasping, domain randomization, sim-to-real transfer, long-horizon tasks, dynamic lighting, moving objects

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review