Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces
Open accessAbstract
In this paper, we study the problem of zero-shot sim-to-real when the task requires both highly precise control with sub-millimetre error tolerance, and wide task space generalisation. Our framework involves a coarse-to-fine controller, where trajectories begin with classical motion planning using ICP-based pose estimation, and transition to a learned end-to-end controller which maps images to actions and is trained in simulation with domain randomisation. In this way, we achieve precise control whilst also generalising the controller across wide task spaces, and keeping the robustness of vision-based, end-to-end control. Real-world experiments on a range of different tasks show that, by exploiting the best of both worlds, our framework significantly outperforms purely motion planning methods, and purely learning-based methods. Furthermore, we answer a range of questions on best practices for precise sim-to-real transfer, such as how different image sensor modalities and image feature representations perform.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent sim2real efforts have been very promising; several works have transferred policies trained via imitation learning from sim to real.
“Recent sim2real efforts have been very promising - several works [61, 95-98] have been able to transfer policies trained via imitation learning from sim to real.”
From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [95]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 2× · see citing works
“2) Overcoming the Reality Gap: Operating in the zeroshot setting creates the problem of the “reality gap”, which we overcome by using domain randomisation [28], [30].”
Background
Flagged first, then by citations
Research works printed but not linked: 28
- M. S. Ahn, H. Chae, D. Noh, H. Nam, and D. Hong. Analysis and noise modeling of the intel realsense d435 for mobile robots. In IEEE 16th International Conference on Ubiquitous Robots (UR), 2019.
- R. Alghonaim and E. Johns. Benchmarking domain randomisation for visual sim-to-real transfer. In IEEE International Conference on Robotics and Automation, 2021.
- O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. Mc- Grew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 2020.
- P. BESL and N. MCKAY. A method for registration of 3d shapes. IEEE Transactions on pattern analysis and machine intelligence, 1992.
- C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. In IEEE International Conference on Robotics and Automation, 2016.
- C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 2021.
- R. L. Haugaard, J. Langaa, C. Sloth, and A. G. Buch. Fast robust peg-in-hole insertion with continuous visual servoing. 4th Conference on Robot Learning (CoRL 2020), CoRL, 2020.
- S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, 2015.
- T. Jakab, A. Gupta, H. Bilen, and A. Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In 32nd International Conference on Neural Information Processing Systems, 2018.
- S. James, A. J. Davison, and E. Johns. Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task. In CoRL, 2017.
- S. James, M. Freese, and A. J. Davison. Pyrep: Bringing v-rep to deep robot learning. arXiv preprint arXiv:1906.11176, 2019. · arXiv
- E. Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In IEEE International Conference on Robotics and Automation, 2021.
- E. Johns, S. Leutenegger, and A. J. Davison. Deep learning a grasp function for grasping under gripper pose uncertainty. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2016.
- D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt- opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293, 2018. · arXiv
- A. Lagrassa, S. Lee, and O. Kroemer. Learning skills to patch plans based on inaccurate models. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020.
- M. A. Lee, C. Florensa, J. Tremblay, N. Ratliff, A. Garg, F. Ramos, and D. Fox. Guided uncertainty-aware policy optimization: Combin- ing learning and model-based strategies for sample-efficient policy learning. In 2020 IEEE International Conference on Robotics and Automation, 2020.
- V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learn- ing, 2010.
- S. Paradis, M. Hwang, B. Thananjeyan, J. Ichnowski, D. Seita, D. Fer, T. Low, J. E. Gonzalez, and K. Goldberg. Intermittent visual servoing: Efficiently learning policies robust to instrument changes for high- precision surgical manipulation. arXiv preprint arXiv:2011.06163, 2020. · arXiv
- E. Y. Puang, K. P. Tee, and W. Jing. Kovis: Keypoint-based visual servoing with zero-shot sim-to-real transfer for robotics manipulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2020.
- R. Raj, V. Namboodiri, and L. Behera. Learning to switch cnns with model agnostic meta learning for fine precision visual servoing. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020.
- E. Rohmer, S. P. N. Singh, and M. Freese. Coppeliasim (formerly v- rep): a versatile and scalable robot simulation framework. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013.
- S. Salcudean and C. An. On the control of redundant coarse-fine manipulators. In IEEE International Conference on Robotics and Automation, 1989.
- G. Schoettler, A. Nair, J. A. Ojea, S. Levine, and E. Solowjow. Meta-reinforcement learning for robotic industrial insertion tasks. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020.
- A. Sharon and D. Hardt. Enhancement of robot accuracy using endpoint feedback and a macro-micro manipulator system. In IEEE American Control Conference, 1984.
- N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut- dinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 2014.
- J. C. Triyonoputro, W. Wan, and K. Harada. Quickly inserting pegs into uncertain holes using multi-view images and deep network trained on synthetic data. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2019.
- E. Valassakis, Z. Ding, and E. Johns. Crossing the gap: A deep dive into zero-shot sim-to-real transfer for dynamics. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020.
- S. Zakharov, B. Planche, Z. Wu, A. Hutter, H. Kosch, and S. Ilic. Keep it unreal: Bridging the realism gap for 2.5 d recognition with geometry priors only. In IEEE International Conference on 3D Vision, 2018.
Cited by
Sorted by citations
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., & Fox, D. (2023) · arXiv · doi:10.48550/arxiv.2310.17596 · Cited by 5 · see citing works
“Combining MimicGen with sim-to-real policy deployment methods [58–62, 95–98] is Improved Performance with More Flexible Policy Models.”
Method
- Authors
- Valassakis, E., Di Palo, N., & Johns, E.
- Affiliations
- Imperial College London (GB)
- ORCID
- Johns, Edward
- Venue
- 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2021-09-27
- Pages
- 5989-5996
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W3183526037
- Cited by
- 17
- References
- 1
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 4 · 2023: 4 · 2022: 3 · 2021: 2
- FWCI
- 1.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- sim-to-real transfer, visual servoing, domain randomization, motion planning, vision-based robot control, visual feature representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
