CoChat
Sign Up
Cited by 275Surgical Simulation and TrainingMotor Control and AdaptationHand Gesture Recognition Systems

Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery

Open access
Wang, Ziheng, Majewicz Fey, Ann·2018·International Journal of Computer Assisted Radiology and Surgery
DOI 10.1007/s11548-018-1860-1
Open this paper in CoChat — free accountOpen PDF

Abstract

Purpose With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge.Methods We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series data of the motion kinematics to individual skill levels.Results We perform experiments on the public minimally invasive surgical robotic dataset, JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our proposed learning model achieved competitive accuracies of 92.5%, 95.4%, and 91.3%, in the standard training tasks: Suturing, Needle-passing, and Knot-tying, respectively. Without the need of engineered features or carefully tuned gesture segmentation, our model can successfully decode skill information from raw motion profiles via end-to-end learning. Meanwhile, the proposed model is able to reliably interpret skills within a 1-3 second window, without needing an observation of entire training trial.Conclusion This study highlights the potential of deep architectures for efficient online skill assessment in modern surgical training.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11548-018-1860-1

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI is expected to augment healthcare workflow through automated triage, improve the productivity of individual physicians, reduce human errors, discover better patterns of patient care, defray medical costs, perform minimally invasive surgery, and reduce mortality rates.

    “AI is expected to augment healthcare workflow through automated triage, improve the productivity of individual physicians, reduce human errors, discover better patterns of patient care, defray medical costs, perform minimally invasive surgery, and reduce mortality rates.8–14”

    From Educating Future Physicians in Artificial Intelligence (AI): An Integrative Review and Proposed Changes (2021) · cites this paper as [12]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 47 · Cited by 275

Research works printed but not linked: 51

  1. [1]1. Roberts K. E., Bell R. L., Duffy A. J. (2006) Evolution of surgical skills training. World journal of gastroenterology: WJG 12(20):3219
  2. [2]2. Reznick R. K., MacRae H. (2006) Teaching surgical skillschanges in the wind. New England Journal of Medicine 355(25):2664-2669
  3. [4]4. Birkmeyer J. D., Finks J. F., O'reilly A., Oerline M., Carlin A. M., Nunn A. R., Dimick J., Banerjee M., Birkmeyer N. J. (2013) Surgical skill and complication rates after bariatric surgery. New England Journal of Medicine 369(15):1434-1442
  4. [6]6. Bridgewater B., Grayson A. D., Jackson M., Brooks N., Grotte G. J., Keenan D. J., Millner R., Fabri B. M., Mark J. (2003) Surgeon specific mortality in adult cardiac surgery: comparison between crude and risk stratified data. British Medical Journal 327(7405):13-17
  5. [7]7. Goh A. C., Goldfarb D. W., Sander J. C., Miles B. J., Dunkin B. J. (2012) Global evaluative assessment of robotic skills: Validation of a clinical assessment tool to measure robotic surgical skills. The Journal of Urology 187(1):247-252
  6. [8]8. Aghazadeh M. A., Jayaratna I. S., Hung A. J., Pan M. M., Desai M. M., Gill I. S., Goh A. C. (2015) External validation of global evaluative assessment of robotic skills (gears). Surgical Endoscopy 29(11):3261-3266
  7. [9]9. Niitsu H., Hirabayashi N., Yoshimitsu M., Mimura T., Taomoto J., Sugiyama Y., Murakami S., Saeki S., Mukaida H., Takiyama W. (2013) Using the objective structured assessment of technical skills (osats) global rating scale to evaluate the skills of surgical trainees in the operating room. Surgery Today 43(3):271-275
  8. [10]10. Reiley C. E., Lin H. C., Yuh D. D., Hager G. D. (2011) Review of methods for objective surgical skill evaluation. Surgical Endoscopy 25(2):356-366
  9. [11]11. Vedula S. S., Ishii M., Hager G. D. (2017) Objective assessment of surgical technical skill and competency in the operating room. Annual review of biomedical engineering 19:301-325
  10. [12]12. Moustris G. P., Hiridis S. C., Deliparaschos K. M., Konstantinidis K. M. (2011) Evolution of autonomous and semi-autonomous robotic surgical systems: a review of the literature. The International Journal of Medical Robotics and Computer Assisted Surgery 7(4):375-392
  11. [13]13. Cheng C., Sa-Ngasoongsong A., Beyca O., Le T., Yang H., Kong Z., Bukkapatnam S. T. (2015) Time series forecasting for nonlinear and non-stationary processes: a review and comparative study. IIE Transactions 47(10):1053-1071
  12. [14]14. Klonowski W. (2009) Everything you wanted to ask about eeg but were afraid to get the right answer. Nonlinear Biomedical Physics 3(1):2
  13. [15]15. Reiley C. E., Hager G. D. (2009) Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. In: International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, pp. 435-442
  14. [16]16. Kassahun Y., Yu B., Tibebu A. T., Stoyanov D., Giannarou S., Metzen J. H., Vander Poorten E. (2016) Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions. International Journal of Computer Assisted Radiology and Surgery 11(4):553-568
  15. [17]17. Judkins T. N., Oleynikov D., Stergiou N. (2009) Objective evaluation of expert and novice performance during robotic surgical training tasks. Surgical endoscopy 23(3):590
  16. [19]19. Trejos A. L., Patel R. V., Malthaner R. A., Schlachta C. M. (2014) Development of force-based metrics for skills assessment in minimally invasive surgery. Surgical Endoscopy 28(7):2106-2119
  17. [20]20. Poursartip B., LeBel M.-E., Patel R., Naish M., Trejos A. L. (2018) Analysis of energy-based metrics for laparoscopic skills assessment. IEEE Transactions on Biomedical Engineering 65(7):1532-1542
  18. [21]21. Ershad M., Koesters Z., Rege R., Majewicz A. (2016) Meaningful assessment of surgical expertise: Semantic labeling with data and crowds. In: International Conference on Medical Image Computing and ComputerAssisted Intervention, Springer, pp. 508-515
  19. [22]22. Sharon Y., Lendvay T. S., Nisky I. (2017) Instrument orientation-based metrics for surgical skill evaluation in robot-assisted and open needle driving. arXiv preprint arXiv:170909452
  20. [23]23. Shackelford S., Bowyer M. (2017) Modern metrics for evaluating surgical technical skills. Current Surgery Reports 5(10):24
  21. [24]24. Fard M. J., Ameri S., Darin Ellis R., Chinnam R. B., Pandya A. K., Klein M. D. (2018) Automated robotassisted surgical skill evaluation: Predictive analytics approach. The International Journal of Medical Robotics and Computer Assisted Surgery 14(1):e1850
  22. [25]25. Stefanidis D., Scott D. J., Korndorffer Jr J. R. (2009) Do metrics matter? time versus motion tracking for performance assessment of proficiency-based laparoscopic skills training. Simulation in Healthcare 4(2):104-108
  23. [26]26. Chmarra M. K., Klein S., de Winter J. C., Jansen F.W., Dankelman J. (2010) Objective classification of residents based on their psychomotor laparoscopic skills. Surgical Endoscopy 24(5):1031-1039
  24. [27]27. Vedula S. S., Malpani A., Ahmidi N., Khudanpur S., Hager G., Chen C. C. G. (2016) Task-level vs. segmentlevel quantitative metrics for surgical skill assessment. Journal of Surgical Education 73(3):482-489
  25. [28]28. Poursartip B., LeBel M.-E., McCracken L. C., Escoto A., Patel R. V., Naish M. D., Trejos A. L. (2017) Energy-based metrics for arthroscopic skills assessment. Sensors 17(8):1808
  26. [29]29. Forestier G., Petitjean F., Senin P., Despinoy F., Jannin P. (2017) Discovering discriminative and interpretable patterns for surgical motion analysis. In: Conference on Artificial Intelligence in Medicine in Europe, Springer, pp. 136-145
  27. [30]30. Brown J. D., OBrien C. E., Leung S. C., Dumon K. R., Lee D. I., Kuchenbecker K. J. (2017) Using contact forces and robot arm accelerations to automatically rate surgeon skill at peg transfer. IEEE Transactions on Biomedical Engineering 64(9):2263-2275
  28. [31]31. Zia A., Essa I. (2018) Automated surgical skill assessment in rmis training. International Journal of Computer Assisted Radiology and Surgery DOI 10.1007/ s11548-018-1735-5
  29. [32]32. Tao L., Elhamifar E., Khudanpur S., Hager G. D., Vidal R. (2012) Sparse hidden markov models for surgical gesture classification and skill evaluation. In: IPCAI, Springer, pp. 167-177
  30. [33]33. LeCun Y., Bengio Y., Hinton G. (2015) Deep learning. Nature 521(7553):436-444
  31. [35]35. Silver D., Huang A., Maddison C. J., Guez A., Sifre L., van den Driessche G., Schrittwieser J., Antonoglou I., Panneershelvam V., Lanctot M., Dieleman S., Grewe D., Nham J., Kalchbrenner N., Sutskever I., Lillicrap T., Leach M., Kavukcuoglu K., Graepel T., Hassabis D. (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484-489
  32. [36]36. Graves A., Mohamed A.-r., Hinton G. (2013) Speech recognition with deep recurrent neural networks. In: Acoustics, speech and signal processing (icassp), 2013 ieee international conference on, IEEE, pp. 6645-6649
  33. [38]38. Rajpurkar P., Hannun A. Y., Haghpanahi M., Bourn C., Ng A. Y. (2017) Cardiologist-level arrhythmia detection with convolutional neural networks. arXiv preprint arXiv:170701836
  34. [39]39. DiPietro R., Lea C., Malpani A., Ahmidi N., Vedula S. S., Lee G. I., Lee M. R., Hager G. D. (2016) Recognizing surgical activities with recurrent neural networks. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp. 551-558
  35. [40]40. Gao Y., Vedula S. S., Reiley C. E., Ahmidi N., Varadarajan B., Lin H. C., Tao L., Zappella L., B´ ejar B., Yuh D. D., Chen C. C. G., Vidal R., Khudanpur S., Hager G. D. (2014) Jhu-isi gesture and skill assessment working set (jigsaws): A surgical activity dataset for human motion modeling. In: MICCAI Workshop: M2CAI, vol 3, p. 3
  36. [41]41. L¨ angkvist M., Karlsson L., Loutfi A. (2014) A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recognition Letters 42:11-24
  37. [42]42. Gamboa J. C. B. (2017) Deep learning for time-series analysis. arXiv preprint arXiv:170101887
  38. [43]43. Nair V., Hinton G. E. (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807-814
  39. [45]45. Li M., Zhang T., Chen Y., Smola A. J. (2014) Efficient mini-batch training for stochastic optimization. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, pp. 661-670
  40. [46]46. Kingma D., Ba J. (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980
  41. [47]47. Srivastava N., Hinton G., Krizhevsky A., Sutskever I., Salakhutdinov R. (2014) Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15(1):1929-1958
  42. [48]48. Wu H., Gu X. (2015) Max-pooling dropout for regularization of convolutional neural networks. In: International Conference on Neural Information Processing, Springer, pp. 46-54
  43. [49]49. Ahmidi N., Tao L., Sefati S., Gao Y., Lea C., Haro B. B., Zappella L., Khudanpur S., Vidal R., Hager G. D. (2017) A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery. IEEE Transactions on Biomedical Engineering 64(9):2025-2041
  44. [50]50. Krizhevsky A., Sutskever I., Hinton G. E. (2012) Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, Curran Associates Inc., NIPS, vol 1, pp. 1097-1105
  45. [52]52. Cui Z., Chen W., Chen Y. (2016) Multi-scale convolutional neural networks for time series classification. arXiv preprint arXiv:160306995
  46. [53]53. Le Guennec A., Malinowski S., Tavenard R. (2016) Data augmentation for time series classification using convolutional neural networks. In: ECML/PKDD Workshop on Advanced Analytics and Learning on Temporal Data
  47. [54]54. Um T. T., Pfister F. M., Pichler D., Endo S., Lang M., Hirche S., Fietzek U., Kuli´ c D. (2017) Data augmentation of wearable sensor data for parkinsons disease monitoring using convolutional neural networks. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, ACM, pp. 216-220
  48. [55]55. Sammut C., Webb G. I. (2011) Encyclopedia of machine learning. Springer Science & Business Media
  49. [56]56. Kumar R., Jog A., Malpani A., Vagvolgyi B., Yuh D., Nguyen H., Hager G., Chen C. C. G. (2012) Assessing system operation skills in robotic surgery trainees. The International Journal of Medical Robotics and Computer Assisted Surgery 8(1):118-124
  50. [57]57. Sun C., Shrivastava A., Singh S., Gupta A. (2017) Revisiting unreasonable effectiveness of data in deep learning era. In: 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, pp. 843-852
  51. [58]58. Dockter R. L., Lendvay T. S., Sweet R. M., Kowalewski T. M. (2017) The minimally acceptable classification criterion for surgical skill: intent vectors and separability of raw motion data. International journal of computer assisted radiology and surgery 12(7):1151-1159

Authors
Wang, Z., & Majewicz Fey, A.
Affiliations
The University of Texas at Dallas (US)
Corresponding author
Wang, Ziheng
ORCID
Wang, Ziheng
Venue
International Journal of Computer Assisted Radiology and Surgery
Publisher
Springer Science and Business Media LLC
Published
2018-09-25
Volume
13
Issue
12
Pages
1959-1970
ISSN
  • 1861-6410 (ISSN-L)
  • 1861-6429
DOI
10.1007/s11548-018-1860-1
OpenAlex
W2807902000
Cited by
275
References
47
Citations by year
2026: 20 · 2025: 32 · 2024: 37 · 2023: 49 · 2022: 40 · 2021: 52 · 2020: 26 · 2019: 18 · 2017: 1
FWCI
5.33
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Computer Assisted Radiology and Surgery (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1464432)
Topics
  • Health Sciences › Medicine › Surgery › Surgical Simulation and Training
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Motor Control and Adaptation
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
Keywords
robotic surgery, surgical skill assessment, convolutional neural networks, motion kinematics, minimally invasive surgery, surgical training, suturing, knot tying, gesture segmentation
MeSH terms
Machine Learning, Deep Learning, Biomechanical Phenomena, Clinical Competence, Gestures, Humans, Robotics, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review