CoChat
Sign Up
PreprintCited by 1

Open-Sourced Reinforcement Learning Environments for Surgical Robotics

Richter, Florian, Orosco, Ryan K., Yip, Michael C.·2019·arXiv
DOI 10.48550/arxiv.1903.02090arXiv 1903.02090
Open this paper in CoChat — free account

Abstract

Reinforcement Learning (RL) is a machine learning framework for artificially intelligent systems to solve a variety of complex problems. Recent years has seen a surge of successes solving challenging games and smaller domain problems, including simple though non-specific robotic manipulation and grasping tasks. Rapid successes in RL have come in part due to the strong collaborative effort by the RL community to work on common, open-sourced environment simulators such as OpenAI's Gym that allow for expedited development and valid comparisons between different, state-of-art strategies. In this paper, we aim to start the bridge between the RL and the surgical robotics communities by presenting the first open-sourced reinforcement learning environments for surgical robots, called dVRL[3]{dVRL available at https://github.com/ucsdarclab/dVRL}. Through the proposed RL environments, which are functionally equivalent to Gym, we show that it is easy to prototype and implement state-of-art RL algorithms on surgical robotics problems that aim to introduce autonomous robotic precision and accuracy to assisting, collaborative, or repetitive tasks during surgery. Learned policies are furthermore successfully transferable to a real robot. Finally, combining dVRL with the over 40+ international network of da Vinci Surgical Research Kits in active use at academic institutions, we see dVRL as enabling the broad surgical robotics community to fully leverage the newest strategies in reinforcement learning, and for reinforcement learning scientists with no knowledge of surgical robotics to test and develop new algorithms that can solve the real-world, high-impact challenges in autonomous surgery.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1903.02090

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The dVRL environment was useful for suction and debris removal in a real surgical setting, although it resulted in simple and primitive actions of reaching and picking.

    “Although the proposed environment resulted in the simple and primitive actions of reaching and picking, it was useful for suction and debris removal in a real surgical setting.”

    From Advanced Applications of Industrial Robotics: New Trends and Possibilities (2021) · cites this paper as [63]

    The Discussion and Conclusion says the environments produce simple reaching and picking primitives and demonstrate their utility through suctioning and debris removal. But the Suction and Debris Removal section describes a simulated abdomen with fake blood, not an actual surgical setting; the paper calls it a “realistic surgical setting.” (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The dVRL control policy was effectively learned and could be transferred to a real robot with minimal effort.

    “During the experiments, the dVRL control policy was effectively learned, and it was found that it could be transferred to a realrobot-with minimal efforts.”

    From Advanced Applications of Industrial Robotics: New Trends and Possibilities (2021) · cites this paper as [63]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 31

  1. R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
  2. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al., "Mastering the game of go with deep neural networks and tree search," Nature, vol. 529, no. 7587, p. 484, 2016.
  3. J. Tobin, L. Biewald, R. Duan, M. Andrychowicz, A. Handa, V. Kumar, B. McGrew, A. Ray, J. Schneider, P. Welinder, et al., "Domain randomization and generative models for robotic grasping," in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3482-3489, IEEE, 2018.
  4. A. J. Hung, J. Chen, D. H. Anthony Jarc, H. Djaladat, and I. S. Gilla, "Development and validation of objective performance metrics for robot-assisted radical prostatectomy: A pilot study," The Journal of Urology, vol. 199, pp. 296-304, Jan 2018.
  5. F. Richter, R. K. Orosco, and M. C. Yip, "Motion scaling solutions for improved performance in high delay surgical teleoperation," arXiv preprint arXiv:1902.03290, 2019. · arXiv
  6. M. Yip and N. Das, ROBOT AUTONOMY FOR SURGERY, ch. Chap- ter 10, pp. 281-313. World Scientific, 2018.
  7. T. Osa, N. Sugita, and M. Mitsuishi, "Online trajectory planning in dynamic environments for surgical task automation.," in Robotics: Science and Systems, pp. 1-9, 2014.
  8. J. Van Den Berg, S. Miller, D. Duckworth, H. Hu, A. Wan, X.- Y. Fu, K. Goldberg, and P. Abbeel, "Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations," in 2010 IEEE International Conference on Robotics and Automation (ICRA), pp. 2074-2081, IEEE, 2010.
  9. F. Alambeigi, Z. Wang, R. Hegeman, Y.-H. Liu, and M. Armand, "A robust data-driven approach for online learning and manipulation of unmodeled 3-d heterogeneous compliant objects," Robotics and Automation Letters, vol. 3, no. 4, pp. 4140-4147, 2018.
  10. J. J. Ji, S. Krishnan, V. Patel, D. Fer, and K. Goldberg, "Learning 2d surgical camera motion from demonstrations," in 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE), pp. 35-42, IEEE, 2018.
  11. A. Murali, S. Sen, B. Kehoe, A. Garg, S. McFarland, S. Patil, W. D. Boyd, S. Lim, P. Abbeel, and K. Goldberg, "Learning by observation for surgical subtasks: Multilateral cutting of 3d viscoelastic and 2d orthotropic tissue phantoms," in 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 1202-1209, IEEE, 2015.
  12. C. D'Ettorre et al., "Automated pick-up of suturing needles for robotic surgical assistance," in Intl. Conf. on Robotics and Automation, pp. 1370-1377, IEEE, 2018.
  13. F. Zhong, Y. Wang, Z. Wang, and Y.-H. Liu, "Dual-arm robotic needle insertion with active tissue deformation for autonomous suturing," Robotics and Automation Letters, vol. 4, no. 3, pp. 2669-2676, 2019.
  14. B. Kehoe, G. Kahn, J. Mahler, J. Kim, A. Lee, A. Lee, K. Nakagawa, S. Patil, W. D. Boyd, P. Abbeel, et al., "Autonomous multilateral debridement with the raven surgical robot," in 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1432-1439, IEEE, 2014.
  15. D. Seita, S. Krishnan, R. Fox, S. McKinley, J. Canny, and K. Goldberg, "Fast and reliable autonomous surgical debridement with cable-driven robots using a two-phase calibration procedure," in 2018 IEEE Inter- national Conference on Robotics and Automation (ICRA), pp. 6651- 6658, IEEE, 2018.
  16. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, "Openai gym," CoRR, vol. abs/1606.01540, 2016. · arXiv
  17. P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor, and S. P. DiMaio, "An open-source research kit for the da vinci R surgical system," IEEE Intl. Conf. on Robotics and Automation, pp. 6434-6439, 2014.
  18. R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, "Policy gradient methods for reinforcement learning with function approxima- tion," in Advances in neural information processing systems, pp. 1057- 1063, 2000.
  19. J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, "Trust region policy optimization," in International Conference on Machine Learning, pp. 1889-1897, 2015.
  20. V. R. Konda and J. N. Tsitsiklis, "Actor-critic algorithms," in Advances in neural information processing systems, pp. 1008-1014, 2000.
  21. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, "Continuous control with deep reinforce- ment learning," arXiv preprint arXiv:1509.02971, 2015. · arXiv
  22. S. James, M. Freese, and A. J. Davison, "Pyrep: Bringing v-rep to deep robot learning," arXiv preprint arXiv:1906.11176, 2019. · arXiv
  23. T. Tang, C. Liu, W. Chen, and M. Tomizuka, "Robotic manipulation of deformable objects by tangent space mapping and non-rigid reg- istration," in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2689-2696, IEEE, 2016.
  24. A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. De Maria, V. Panneershelvam, M. Suleyman, C. Beattie, S. Pe- tersen, et al., "Massively parallel methods for deep reinforcement learning," arXiv preprint arXiv:1507.04296, 2015. · arXiv
  25. Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, "Benchmarking deep reinforcement learning for continuous control," in International Conference on Machine Learning, pp. 1329-1338, 2016.
  26. G. A. Fontanelli, F. Ficuciello, L. Villani, and B. Siciliano, "Modelling and identification of the da vinci research kit robotic arms," in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1464-1469, IEEE, 2017.
  27. Y. Wang, R. Gondokaryono, A. Munawar, and G. S. Fischer, "A convex optimization-based dynamic model identification package for the da vinci research kit," IEEE Robotics and Automation Letters, vol. 4, no. 4, pp. 3657-3664, 2019.
  28. M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder, V. Ku- mar, and W. Zaremba, "Multi-goal reinforcement learning: Challeng- ing robotics environments and request for research," arXiv preprint arXiv:1802.09464, 2018. · arXiv
  29. M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welin- der, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, "Hindsight experience replay," in Advances in Neural Information Processing Systems, 2017.
  30. J. Matas, S. James, and A. J. Davison, "Sim-to-real reinforce- ment learning for deformable object manipulation," arXiv preprint arXiv:1806.07851, 2018. · arXiv
  31. M. M. Mohamed, J. Gu, and J. Luo, "Modular design of neurosurgical robotic system," International Journal of Robotics and Automation, vol. 33, no. 5, 2018.

Cited by

Sorted by citations

Authors
Richter, F., Orosco, R. K., & Yip, M. C.
Venue
arXiv
Publisher
arXiv
Published
2019
DOI
10.48550/arxiv.1903.02090
arXiv
1903.02090
References
5
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review