Cited by 125Artificial Intelligence in Healthcare and EducationSurgical Simulation and TrainingTechnology and Human Factors in Education and Health

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions

Open access
Vercauteren, Tom, Unberath, Mathias, Padoy, Nicolas, Navab, Nassir·2020·Proceedings of the IEEE

Abstract

Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/jproc.2019.2946993

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Machine learning methods, including deep neural networks, have enjoyed growing popularity across image analysis problems precisely because they do not require explicit definitions of complex functional mappings.

    “Machine learning methods, including deep neural networks (NNs), have enjoyed a growing popularity across a variety of image analysis problems (Vercauteren et al., 2019), precisely because they do not require explicit definitions of complex functional mappings.”

    From The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective (2021) · cites this paper as Vercauteren et al., 2019

    The Introduction describes machine learning across multiple computer-assisted intervention tasks, and the Intelligent Imaging in Interventional Biophotonics section notes its increasing use in hyperspectral imaging. The Contextual Learning for Image Registration section gives an example of data-driven methods that avoid explicitly modeling image-intensity correlations. That example does not establish that avoiding explicit definitions of complex functional mappings is the general reason for machine learning’s growing popularity across image analysis. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • Because image-based navigation techniques are designed to seamlessly integrate into conventional surgical workflows, their use should-in theory-not cause frustration or prolonged procedure times.

    “Further, because image-based navigation techniques are designed to seamlessly integrate into conventional surgical workflows, their use should—in theory—not cause frustration or prolonged procedure times (Vercauteren et al., 2019).”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 171 · Cited by 125

Research works printed but not linked: 165

  1. [1]Simpson A. L., “ A large annotated medical image dataset for the development and evaluation of segmentation algorithms,” Feb. 2019, arXiv:1902.09063. [Online]. Available: https://arxiv.org/abs/1902.09063 · arXiv
  2. [3]Maier-Hein L., “ Surgical data science for next-generation interventions,” Nature Biomed. Eng., vol. 1, no. 9, pp. 691– 696, 2017. 31015666
  3. [4]Vedula S. S. and Hager G. D., “ Surgical data science: The new knowledge domain,” Innov. Surgical Sci., vol. 2, no. 3, pp. 109– 121, 2007. PMC5602563 28936475
  4. [5]Maier-Hein L., “ Surgical data science: A consensus perspective,” 2018, arXiv:1806.03184. [Online]. Available: https://arxiv.org/abs/1806.03184 · arXiv
  5. [6]Peters T. and Cleary K., Image-Guided Interventions Technology and Applications. Boston, MA, USA: Springer, 2008. 20415592
  6. [7]West J., “ Comparison and evaluation of retrospective intermodality brain image registration techniques,” J. Comput. Assist. Tomogr., vol. 21, no. 4, pp. 554– 568, 1997. 9216759
  7. [9]Iglesias J. E. and Sabuncu M. R., “ Multi-atlas segmentation of biomedical images: A survey,” Med. Image Anal., vol. 24, no. 1, pp. 205– 219, 2015. 26201875 PMC4532640
  8. [10]Comaniciu D., Engel K., Georgescu B., and Mansi T., “ Shaping the future through innovations: From medical imaging to precision medicine,” Med. Image Anal., vol. 33, pp. 19– 26, Oct. 2016. 27349829
  9. [11]Li W., Wang G., Fidon L., Ourselin S., Cardoso M. J., and Vercauteren T., “ On the compactness, efficiency, and representation of 3D convolutional networks: Brain parcellation as a pretext task,” in Information Processing in Medical Imaging (Lecture Notes in Computer Science), vol. 10265 Cham, Switzerland: Springer-Verlag, Jun. 2017, pp. 348– 360.
  10. [12]Litjens G., “ A survey on deep learning in medical image analysis,” Med. Image Anal., vol. 42, pp. 60– 88, Dec. 2017. 28778026
  11. [13]Bakas S., “ Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BraTS challenge,” 2018, arXiv:1811.02629. [Online]. Available: https://arxiv.org/abs/1811.02629 · arXiv
  12. [14]Sheller M. J., Reina G. A., Edwards B., Martin J., and Bakas S., “ Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation,” in Proc. BrainLes Challenge MICCAI, 2018, pp. 92– 104. PMC6589345 31231720
  13. [15]Roy A. G., Siddiqui S., Pölsterl S., Navab N., and Wachinger C., “ BrainTorrent: A peer-to-peer environment for decentralized federated learning,” 2019, arXiv:1905.06731. [Online]. Available: https://arxiv.org/abs/1905.06731 · arXiv
  14. [16]Kamnitsas K., “ Unsupervised domain adaptation in brain lesion segmentation with adversarial networks,” in Proc. IPMI, 2017, pp. 597– 609.
  15. [17]Wang P., “ Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: A prospective randomised controlled study,” in Proc. Gut, 2019, pp. 1813– 1819. PMC6839720 30814121
  16. [19]Dorent R., Li W., Ekanayake J., Ourselin S., and Vercauteren T., “ Learning joint lesion and tissue segmentation from task-specific hetero-modal datasets,” in Proceedings of Machine Learning Research, vol. 102, Cardoso M. J. Eds. Cambridge, MA, USA: PMLR, 2019, pp. 164– 174.
  17. [20]Dorent R., Joutard S., Modat M., Ourselin S., and Vercauteren T., “ Hetero-modal variational encoder-decoder for joint modality completion and segmentation,” in Proc. MICCAI, 2019, pp. 74– 82.
  18. [21]Digioia M. D. A. M. III, Jaramaz B., Nikou C., Labarca R. S., Moody J. E., and Colgan B. D., “ Surgical navigation for total hip replacement with the use of HipNav,” Operative Techn. Orthopaedics, vol. 10, no. 1, pp. 3– 8, 2000.
  19. [22]Boudissa M., Courvoisier A., Chabanas M., and Tonetti J., “ Computer assisted surgery in preoperative planning of acetabular fracture surgery: State of the art,” Expert Rev. Med. Devices, vol. 15, no. 1, pp. 81– 89, 2018. 29206497
  20. [23]Kulyk P., Vlachopoulos L., Fürnstahl P., and Zheng G., “ Fully automatic planning of total shoulder arthroplasty without segmentation: A deep learning based approach,” in Computational Methods and Clinical Applications in Musculoskeletal Imaging, Vrtovec T., Yao J., Zheng G., and Pozo J. M., Eds. Cham, Switzerland: Springer, 2019, pp. 22– 34.
  21. [24]Carrillo F., Vlachopoulos L., Schweizer A., Nagy L., Snedeker J., and Fürnstahl P., “ A time saver: Optimization approach for the fully automatic 3D planning of forearm osteotomies,” in Medical Image Computing and Computer Assisted Intervention—MICCAI, Descoteaux M., Maier-Hein L., Franz A., Jannin P., Collins D. L., and Duchesne S., Eds. Cham, Switzerland: Springer, 2017, pp. 488– 496.
  22. [25]Sparks R., “ Automated multiple trajectory planning algorithm for the placement of stereo-electroencephalography (SEEG) electrodes in epilepsy treatment,” Int. J. Comput. Assist. Radiol. Surg., vol. 12, no. 1, pp. 123– 136, Jan. 2017. 27368184 PMC5216164
  23. [26]Vakharia V. N., “ Automated trajectory planning for laser interstitial thermal therapy in mesial temporal lobe epilepsy,” Epilepsia, vol. 59, no. 4, pp. 814– 824, 2018. 29528488 PMC5901027
  24. [27]Granados A., “ A machine learning approach to predict instrument bending in stereotactic neurosurgery,” in Medical Image Computing and Computer Assisted Intervention—MICCAI, Frangi A. F., Schnabel J. A., Davatzikos C., Alberola-López C., and Fichtinger G., Eds. Cham, Switzerland: Springer, 2018, pp. 238– 246.
  25. [28]Giffard-Roisin S., “ Transfer learning from simulations on a reference anatomy for ECGI in personalized cardiac resynchronization therapy,” IEEE Trans. Biomed. Eng., vol. 66, no. 2, pp. 343– 353, Feb. 2018. 29993409
  26. [29]Ramalhinho J., “ A pre-operative planning framework for global registration of laparoscopic ultrasound to CT images,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 8, pp. 1177– 1186, Aug. 2018. 29860550 PMC6096745
  27. [30]Johnsen S. F., “ Database-based estimation of liver deformation under pneumoperitoneum for surgical image-guidance and simulation,” in Medical Image Computing and Computer Assisted Intervention—MICCAI, Navab N., Hornegger J., Wells W. M., and Frangi A., Eds. Cham, Switzerland: Springer, 2015, pp. 450– 458.
  28. [31]März K., “ Toward knowledge-based liver surgery: Holistic information processing for surgical decision support,” Int. J. Comput. Assist. Radiol. Surg., vol. 10, no. 6, pp. 749– 759, Jun. 2015. 25847671
  29. [32]Koivukangas T., Katisko J. P., and Koivukangas J. P., “ Technical accuracy of optical and the electromagnetic tracking systems,” SpringerPlus, vol. 2, no. 1, p. 90, 2013. 23586003 PMC3622743
  30. [33]Jolesz F. A., Intraoperative Imaging and Image-Guided Therapy. New York, NY, USA: Springer, 2014.
  31. [34]Pluim J. P. W. and Fitzpatrick J. M., “ Image registration,” IEEE Trans. Med. Imag., vol. 22, no. 11, pp. 1341– 1343, Nov. 2003. 14606668
  32. [35]Liao R., Zhang L., Sun Y., Miao S., and Chefd C., “ A review of recent advances in registration techniques applied to minimally invasive therapy,” IEEE Trans. Multimedia, vol. 15, no. 5, pp. 983– 1000, Aug. 2013.
  33. [36]Joskowicz L. and Hazan E. J., “ Computer aided orthopaedic surgery: Incremental shift or paradigm change?” Med. Image Anal., vol. 33, pp. 84– 90, Oct. 2016. 27407004
  34. [37]Fuerst B., Wein W., Müller M., and Navab N., “ Automatic ultrasound’MRI registration for neurosurgery using the 2D and 3D LC2 metric,” Med. Image Anal., vol. 18, no. 8, pp. 1312– 1319, 2014. 24842859
  35. [38]Wein W., “ Brain-shift correction with image-based registration and landmark accuracy evaluation,” in Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation. Cham, Switzerland: Springer, 2018, pp. 146– 151.
  36. [40]Balakrishnan G., Zhao A., Sabuncu M. R., Guttag J., and Dalca A. V., “ VoxelMorph: A learning framework for deformable medical image registration,” IEEE Trans. Med. Imag., vol. 38, no. 8, pp. 1788– 1800, Aug. 2019. 30716034
  37. [41]Hu Y., “ Weakly-supervised convolutional neural networks for multimodal image registration,” Med. Image Anal., vol. 49, pp. 1– 13, Oct. 2018. 30007253 PMC6742510
  38. [42]Hu Y., “ Adversarial deformation regularization for training image registration neural networks,” in Medical Image Computing and Computer Assisted Intervention—MICCAI (Lecture Notes in Computer Science), vol. 11070 Cham, Switzerland: Springer, 2018, pp. 774– 782.
  39. [43]Pfeiffer M., Riediger C., Weitz J., and Speidel S., “ Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 7, pp. 1147– 1155, Jul. 2019. 30993520
  40. [44]Brunet J.-N., Mendizabal A., Petit A., Golse N., Vibert E., and Cotin S., “ Physics-based deep neural network for augmented reality during liver surgery,” in Medical Image Computing and Computer Assisted Intervention—MICCAI (Lecture Notes in Computer Science). Cham, Switzerland: Springer, 2019.
  41. [45]Hu Y., Gibson E., Barratt D. C., Emberton M., Noble J. A., and Vercauteren T., “ Conditional segmentation in lieu of image registration,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2019.
  42. [48]Bui M., Albarqouni S., Schrapp M., Navab N., and Ilic S., “ X-Ray PoseNet: 6 DoF pose estimation for mobile X-Ray devices,” in Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), Mar. 2017, pp. 1036– 1044.
  43. [50]Prevost R., “ 3D freehand ultrasound without external tracking using deep learning,” Med. Image Anal., vol. 48, pp. 187– 202, Aug. 2018. 29936399
  44. [51]Bier B., “ X-ray-transform invariant anatomical landmark detection for pelvic trauma surgery,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2018, pp. 55– 63.
  45. [52]Bier B., “ Detecting anatomical landmarks for motion estimation in weight-bearing imaging of knees,” in Proc. Int. Workshop Mach. Learn. Med. Image Reconstruct. Cham, Switzerland: Springer, 2018, pp. 83– 90.
  46. [53]Gao C., Unberath M., Taylor R., and Armand M., “ Localizing dexterous surgical tools in X-ray for image-based navigation,” in Proc. IPCAI. Cham, Switzerland: Springer, 2019, pp. 1– 4.
  47. [54]Kügler D., Stefanov A., and Mukhopadhyay A., “ i3PosNet: Instrument pose estimation from X-ray,” 2018, arXiv:1802.09575. [Online]. Available: https://arxiv.org/abs/1802.09575 PMC7316684 32440956 · arXiv
  48. [55]Esfandiari H., Newell R., Anglin C., Street J., and Hodgson A. J., “ A deep learning framework for segmentation and pose estimation of pedicle screw implants based on C-arm fluoroscopy,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 8, pp. 1269– 1282, 2018. 29808466
  49. [56]Liao H., Lin W.-A., Zhang J., Zhang J., Luo J., and Zhou S. K., “ Multiview 2D/3D rigid registration via a point-of-interest network for tracking and triangulation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2019, pp. 12638– 12647.
  50. [57]Unberath M., “ Enabling machine learning in X-ray-based procedures via realistic simulation of image formation,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 9, pp. 1517– 1528, 2019. 31187399 PMC7297499
  51. [58]Clunie D. A., “ Technical challenges of enterprise imaging: HIMSS-SIIM collaborative white paper,” J. Digit. Imag., vol. 29, no. 5, pp. 583– 614, 2016. PMC5023533 27576909
  52. [59]Murad F. M., “ Image management systems,” Gastrointestinal Endoscopy, vol. 79, no. 1, pp. 15– 22, 2014. 24239253
  53. [60]Deganello A., Sellars M. E., Yusuf G. T., and Sidhu P. S., “ How much should i record during a CEUS examination? Practical aspects of the ‘real-time’ feature of a contrast ultrasound study,” Ultraschall Med., vol. 39, no. 5, pp. 484– 486, 2018. 30300906
  54. [61]Milletari F., Birodkar V., and Sofka M., “ Straight to the point: Reinforcement learning for user guidance in ultrasound,” 2019, arXiv:1903.00586. [Online]. Available: https://arxiv.org/abs/1903.00586 · arXiv
  55. [62]Hu Y., “ Freehand ultrasound image simulation with spatially-conditioned generative adversarial networks,” in Proc. RAMBO Workshop MICCAI (Lecture Notes in Computer Science), vol. 10555 Cham, Switzerland: Springer-Verlag, Sep. 2017, pp. 105– 115.
  56. [63]Engelhardt S., De Simone R., Full P. M., Karck M., and Wolf I., “ Improving surgical training phantoms by hyperrealism: Deep unpaired image-to-image translation from real surgeries,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. 2018, pp. 747– 755.
  57. [64]Luengo I., Flouty E., Giataganas P., Wisanuvej P., Nehme J., and Stoyanov D., “ SurReal: Enhancing surgical simulation realism using style transfer,” in Proc. Brit. Mach. Vis. Conf., Newcastle, U.K., Sep. 2018, p. 116.
  58. [65]Mahmood F., Chen R., and Durr N. J., “ Unsupervised reverse domain adaptation for synthetic medical images via adversarial training,” IEEE Trans. Med. Imag., vol. 37, no. 12, pp. 2572– 2581, Dec. 2018. 29993538
  59. [66]Zhang Y., Miao S., Mansi T., and Liao R., “ Task driven generative modeling for unsupervised domain adaptation: Application to X-ray image segmentation,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2018.
  60. [67]Unberath M., “ DeepDRR—A catalyst for machine learning in fluoroscopy-guided procedures,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2018.
  61. [68]Mahmood F., Chen R., Sudarsky S., Yu D., and Durr N. J., “ Deep learning with cinematic rendering: Fine-tuning deep neural networks using photorealistic medical images,” 2018, arXiv:1805.08400. [Online]. Available: https://arxiv.org/abs/1805.08400 30113015 · arXiv
  62. [69]Nair A. A., Tran T. D., Reiter A., and Bell M. A. L., “ A deep learning based alternative to beamforming ultrasound images,” in Proc. ICASSP, 2018, pp. 3359– 3363.
  63. [70]Toth D., Cimen S., Ceccaldi P., Kurzendorfer T., Rhode K., and Mountney P., “ Training deep networks on domain randomized synthetic X-ray data for cardiac interventions,” in Proceedings of Machine Learning Research, vol. 102, Cardoso M. J. Eds. Cambridge, MA, USA: PMLR, 2019, pp. 468– 482.
  64. [71]Jermyn M., “ Intraoperative brain cancer detection with Raman spectroscopy in humans,” Sci. Transl. Med., vol. 7, no. 274, p. 274ra19, 2015. 25673764
  65. [72]Jermyn M., “ Neural networks improve brain cancer detection with Raman spectroscopy in the presence of operating room light artifacts,” J. Biomed. Opt., vol. 21, no. 9, p. 094002, 2016. 27604560
  66. [73]Banbury C., “ Development of the self optimising Kohonen index network (SKiNET) for Raman spectroscopy based detection of anatomical eye tissue,” Sci. Rep., vol. 9, no. 1, p. 10812, 2019. 31346227 PMC6658481
  67. [74]Shapey J., “ Intraoperative multispectral and hyperspectral label-free imaging: A systematic review of in vivo clinical studies,” J. Biophoton., vol. 12, no. 9, p. e201800455, 2019. PMC6736677 30859757
  68. [75]Adler T. J., “ Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 997– 1007, 2019. 30903566
  69. [76]Ravì D., Fabelo H., Callic G. M., and Yang G.-Z., “ Manifold embedding and semantic segmentation for intraoperative guidance with hyperspectral brain imaging,” IEEE Trans. Med. Imag., vol. 36, no. 9, pp. 1845– 1857, Sep. 2017. 28436854
  70. [77]Lin J., “ Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks,” Med. Image Anal., vol. 48, pp. 162– 176, Aug. 2018. 29933116
  71. [78]Ravì D., Szczotka A. B., Pereira S. P., and Vercauteren T., “ Adversarial training with cycle consistency for unsupervised super-resolution in endomicroscopy,” Med. Image Anal., vol. 53, pp. 123– 131, Apr. 2019. 30769327 PMC6873642
  72. [79]Szczotka A. B., Ravì D., Shakir D. I., Pereira S. P., and Vercauteren T., “ Effective deep learning training for single-image super-resolution in endomicroscopy exploiting video-registration-based reconstruction,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 6, pp. 917– 924, Jun. 2018. 29687176 PMC5973979
  73. [80]Allman D., Assis F., Chrispin J., and Bell M. A. L., “ Deep neural networks to remove photoacoustic reflection artifacts in ex vivo and in vivo tissue,” in Proc. IEEE Int. Ultrason. Symp. (IUS), Oct. 2018, pp. 1– 4.
  74. [81]Ambrosini P., Ruijters D., Niessen W. J., Moelker A., and van Walsum T., “ Fully automatic and real-time catheter segmentation in X-ray fluoroscopy,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2017, pp. 577– 585.
  75. [82]Breininger K., “ Multiple device segmentation for fluoroscopic imaging using multi-task learning,” in Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis. Cham, Switzerland: Springer, 2018, pp. 19– 27.
  76. [83]Sa R., “ Intervertebral disc detection in X-ray images using faster R-CNN,” in Proc. 39th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC), Jul. 2017, pp. 564– 567. 29059935
  77. [84]Matviychuk Y., “ Learning a multiscale patch-based representation for image denoising in X-ray fluoroscopy,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Sep. 2016, pp. 2330– 2334.
  78. [85]Hariharan S. G., “ Learning-based X-ray image denoising utilizing model-based image simulations,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2019.
  79. [86]Anas E. M. A., Mousavi P., and Abolmaesumi P., “ A deep learning approach for real time prostate segmentation in freehand ultrasound guided biopsy,” Med. Image Anal., vol. 48, pp. 107– 116, Aug. 2018. 29886268
  80. [87]Mwikirize C., Nosher J. L., and Hacihaliloglu I., “ Convolution neural networks for real-time needle detection and localization in 2D ultrasound,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 5, pp. 647– 657, 2018. 29512006
  81. [88]Pesteie M., Lessoway V., Abolmaesumi P., and Rohling R. N., “ Automatic localization of the needle target for ultrasound-guided epidural injections,” IEEE Trans. Med. Imag., vol. 37, no. 1, pp. 81– 92, Jan. 2017. 28809679
  82. [89]Luchies A. C. and Byram B. C., “ Deep neural networks for ultrasound beamforming,” IEEE Trans. Med. Imag., vol. 37, no. 9, pp. 2010– 2021, Sep. 2018. PMC6109603 29994441
  83. [90]Simson W., “ End-to-end learning-based ultrasound reconstruction,” 2019, arXiv:1904.04696. [Online]. Available: https://arxiv.org/abs/1904.04696 · arXiv
  84. [91]Simson W., Paschali M., Navab N., and Zahnd G., “ Deep learning beamforming for sub-sampled ultrasound data,” in Proc. IEEE Int. Ultrason. Symp. (IUS), Oct. 2018, pp. 1– 4.
  85. [92]Zaech J.-N., “ Learning to avoid poor images: Towards task-aware C-arm cone-beam CT trajectories,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2019.
  86. [93]Milletari F., Rieke N., Baust M., Esposito M., and Navab N., “ CFCM: Segmentation via coarse to fine context memory,” 2018, arXiv:1806.01413. [Online]. Available: https://arxiv.org/abs/1806.01413 · arXiv
  87. [94]Blum T., Padoy N., Feußner H., and Navab N., “ Modeling and online recognition of surgical phases using hidden Markov models,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2008, pp. 627– 635. 18982657
  88. [95]Lalys F., Riffaud L., Bouget D., and Jannin P., “ A framework for the recognition of high-level surgical tasks from video images for cataract surgeries,” IEEE Trans. Biomed. Eng., vol. 59, no. 4, pp. 966– 976, Dec. 2012. 22203700 PMC3432023
  89. [96]Quellec G., Lamard M., Cochener B., and Cazuguel G., “ Real-time segmentation and recognition of surgical tasks in cataract surgery videos,” IEEE Trans. Med. Imag., vol. 33, no. 12, pp. 2352– 2360, Dec. 2014. 25055383
  90. [97]Twinanda A. P., Shehata S., Mutter D., Marescaux J., de Mathelin M., and Padoy N., “ EndoNet: A deep architecture for recognition tasks on laparoscopic videos,” IEEE Trans. Med. Imag., vol. 36, no. 1, pp. 86– 97, Jan. 2017. 27455522
  91. [98]Volkov M., Hashimoto D. A., Rosman G., Meireles O. R., and Rus D., “ Machine learning and coresets for automated real-time video segmentation of laparoscopic and robot-assisted surgery,” in Proc. ICRA, 2017, pp. 754– 759.
  92. [99]Bodenstedt S., “ Active learning using deep Bayesian networks for surgical workflow analysis,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 1079– 1087, 2019. 30968355
  93. [100]Padoy N., Blum T., Ahmadi S.-A., Feussner H., Berger M.-O., and Navab N., “ Statistical modeling and recognition of surgical workflow,” Med. Image Anal., vol. 16, no. 3, pp. 632– 641, 2012. 21195015
  94. [101]Twinanda A. P., Yengera G., Mutter D., Marescaux J., and Padoy N., “ RSDNet: Learning to predict remaining surgery duration from laparoscopic videos without manual annotations,” IEEE Trans. Med. Imag., vol. 38, no. 4, pp. 1069– 1078, Apr. 2019. 30371356
  95. [102]Bodenstedt S., “ Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 1089– 1095, 2019. 30968352
  96. [103]Al Hajj H., Lamard M., Conze P.-H., Cochener B., and Quellec G., “ Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks,” Med. Image Anal., vol. 47, pp. 203– 218, Jul. 2018. 29778931
  97. [104]Nwoye C. I., Mutter D., Marescaux J., and Padoy N., “ Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 1059– 1067, 2019. 30968356
  98. [105]Leibetseder A., Primus M. J., and Schoeffmann K., “ Automatic smoke classification in endoscopic video,” in Proc. 24th Int. Conf. MultiMedia Modeling (MMM), Bangkok, Thailand, Feb. 2018, pp. 362– 366.
  99. [106]Okamoto T., Ohnishi T., Kawahira H., Dergachyava O., Jannin P., and Haneishi H., “ Real-time identification of blood regions for hemostasis support in laparoscopic surgery,” Signal, Image Video Process., vol. 13, no. 2, pp. 405– 412, 2019.
  100. [107]Kannan S., Yengera G., Mutter D., Marescaux J., and Padoy N., “ Future-state predicting LSTM for early surgery type recognition,” IEEE Trans. Med. Imag., to be published. 31352339
  101. [108]Funke I., Jenke A., Mees S. T., Weitz J., Speidel S., and Bodenstedt S., “ Temporal coherence-based self-supervised learning for laparoscopic workflow analysis,” in Proc. 1st Int. Workshop, OR 2.0 5th Int. Workshop (CARE), 7th Int. Workshop (CLIP), 3rd Int. Workshop (ISIC), Granada, Spain, Sep. 2018, pp. 85– 93.
  102. [109]Yengera G., Mutter D., Marescaux J., and Padoy N., “ Less is more: Surgical phase recognition with less annotations through self-supervised pre-training of CNN-LSTM networks,” 2018, arXiv:1805.08569. [Online]. Available: https://arxiv.org/abs/1805.08569 · arXiv
  103. [110]Roß T., “ Exploiting the potential of unlabeled endoscopic video data with self-supervised learning,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 6, pp. 925– 933, 2018. 29704196
  104. [111]Yu T., Mutter D., Marescaux J., and Padoy N., “ Learning from a tiny dataset of manual annotations: A teacher/student approach for surgical phase recognition,” in Proc. IPCAI, 2019, pp. 1– 13.
  105. [112]Liu X., “ Self-supervised learning for dense depth estimation in monocular endoscopy,” in OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis. Cham, Switzerland: Springer, 2018, pp. 128– 138.
  106. [113]Ntourakis D., Memeo R., Soler L., Marescaux J., Mutter D., and Pessaux P., “ Augmented reality guidance for the resection of missing colorectal liver metastases: An initial experience,” World J. Surg., vol. 40, no. 2, pp. 419– 426, 2015. 26316112
  107. [114]Bouget D., Allan M., Stoyanov D., and Jannin P., “ Vision-based and marker-less surgical tool detection and tracking: A review of the literature,” Med. Image Anal., vol. 35, pp. 633– 654, Jan. 2017. 27744253
  108. [115]Sarikaya D., Corso J. J., and Guru K. A., “ Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection,” IEEE Trans. Med. Imag., vol. 36, no. 7, pp. 1542– 1549, 2017. 28186883
  109. [116]Kurmann T., “ Simultaneous recognition and pose estimation of instruments in minimally invasive surgery,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2017, pp. 505– 513.
  110. [117]Jin A., “ Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks,” in Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), Lake Tahoe, NV, USA, Mar. 2018, pp. 691– 699.
  111. [119]Laina I., “ Concurrent segmentation and localization for tracking of surgical instruments,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2017, pp. 664– 672.
  112. [120]Bodenstedt S., “ Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery,” 2018 arXiv:1805.02475. [Online]. Available: https://arxiv.org/abs/1805.02475 · arXiv
  113. [121]Allan M., “ 2017 robotic instrument segmentation challenge,” 2019, arXiv:1902.06426. [Online]. Available: https://arxiv.org/abs/1902.06426 · arXiv
  114. [122]Lejeune L., Grossrieder J., and Sznitman R., “ Iterative multi-path tracking for video and volume segmentation with sparse point supervision,” Med. Image Anal., vol. 50, pp. 65– 81, Dec. 2018. 30212738
  115. [123]Fuentes-Hurtado F., Kadkhodamohammadi A., Flouty E., Barbarisi S., Luengo I., and Stoyanov D., “ EasyLabels: Weak labels for scene segmentation in laparoscopic videos,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 7, pp. 1247– 1257, 2019. 31165349
  116. [124]Flouty E., “ CaDIS: Cataract dataset for image segmentation,” (2019), arXiv:1906.11586. [Online]. Available: https://arxiv.org/abs/1906.11586 · arXiv
  117. [125]Gibson E., “ Deep residual networks for automatic segmentation of laparoscopic videos of the liver,” Proc. SPIE, vol. 10135, p. 101351M, Mar. 2017.
  118. [126]Everson M. A., “ Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: A proof-of-concept study,” United Eur. Gastroenterol. J., vol. 7, no. 2, pp. 297– 306, 2019. PMC6498793 31080614
  119. [127]Misawa M., “ Artificial intelligence-assisted polyp detection for colonoscopy: Initial experience,” Gastroenterology, vol. 154, no. 8, pp. 2027– 2029, 2018. 29653147
  120. [128]Bernal J., “ Comparative validation of polyp detection methods in video colonoscopy: Results from the MICCAI 2015 endoscopic vision challenge,” IEEE Trans. Med. Imag., vol. 36, no. 6, pp. 1231– 1249, Feb. 2017. 28182555
  121. [129]Mahmood F., Yang Z., Chen R., Borders D., Xu W., and Durr N. J., “ Polyp segmentation and classification using predicted depth from monocular endoscopy,” Proc. SPIE, vol. 10950, p. 1095011, Mar. 2019.
  122. [130]Leonard S., “ Evaluation and stability analysis of video-based navigation system for functional endoscopic sinus surgery on in vivo clinical data,” IEEE Trans. Med. Imag., vol. 37, no. 10, pp. 2185– 2195, 5 2018. 29993881
  123. [131]Visentini-Scarzanella M., Sugiura T., Kaneko T., and Koto S., “ Deep monocular 3D reconstruction for assisted navigation in bronchoscopy,” Int. J. Comput. Assist. Radiol. Surg., vol. 12, no. 7, pp. 1089– 1099, 2017. 28508345
  124. [132]Wang J., Song S., Ren H., Lim C. M., and Meng M. Q.-H., “ Surgical instrument tracking by multiple monocular modules and a sensor fusion approach,” IEEE Trans. Autom. Sci. Eng., vol. 16, no. 2, pp. 629– 639, Apr. 2019.
  125. [133]Liu X., “ Self-supervised learning for dense depth estimation in monocular endoscopy,” 2019, arXiv:1902.07766. [Online]. Available: https://arxiv.org/abs/1806.09521 PMC7289272 31689184 · arXiv
  126. [134]Rodas N. L., Barrera F., and Padoy N., “ See it with your own eyes: Markerless mobile augmented reality for radiation awareness in the hybrid room,” IEEE Trans. Biomed. Eng., vol. 64, no. 2, pp. 429– 440, Feb. 2017. 27164565
  127. [135]James J. T., “ A new, evidence-based estimate of patient harms associated with hospital care,” J. Patient Saf., vol. 9, no. 3, pp. 122– 128, 2013. 23860193
  128. [136]Suliburk J. W., Buck Q. M., and Pirko C. J., “ Analysis of human performance deficiencies associated with surgical adverse events,” JAMA Netw. Open, vol. 2, no. 7, p. e198067, 2019. 31365107 PMC6669897
  129. [138]Padoy N., “ Machine and deep learning for workflow recognition during surgery,” Minimally Invasive Therapy Allied Technol., vol. 28, no. 2, pp. 82– 90, 2019. 30849261
  130. [139]Jannin P. and Morandi X., “ Surgical models for computer-assisted neurosurgery,” NeuroImage, vol. 37, no. 3, pp. 783– 791, 2007. 17613249
  131. [140]Blum T., Padoy N., Feußner H., and Navab N., “ Workflow mining for visualization and analysis of surgeries,” Int. J. Comput. Assist. Radiol. Surg., vol. 3, no. 5, pp. 379– 386, 2008.
  132. [141]Gibaud B., “ Toward a standard ontology of surgical process models,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 9, pp. 1397– 1408, 2018. 30006820
  133. [142]Haque A., “ Towards vision-based smart hospitals: A system for tracking and monitoring hand hygiene compliance,” in Proc. Mach. Learn. Res. Mach. Learn. Healthcare Conf. (MLHC), 2017, pp. 75– 87.
  134. [143]Ma A. J., “ Measuring patient mobility in the ICU using a novel noninvasive sensor,” Crit. care Med., vol. 45, no. 4, pp. 630– 636, 2017. 28291092 PMC5697716
  135. [145]Haque A., Peng B., Luo Z., Alahi A., Yeung S., and Li F., “ Viewpoint invariant 3D human pose estimation with recurrent error feedback,” in Proc. ECCV, 2016, pp. 160– 177.
  136. [146]Belagiannis V., “ Parsing human skeletons in an operating room,” Mach. Vis. Appl., vol. 27, no. 7, p. 1035– 1046, Oct. 2016.
  137. [147]Kadkhodamohammadi A., Gangi A., de Mathelin M., and Padoy N., “ A multi-view RGB-D approach for human pose estimation in operating rooms,” in Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), Mar. 2017, pp. 363– 372.
  138. [148]Chakraborty I., Elgammal A., and Burd R. S., “ Video based activity recognition in trauma resuscitation,” in Proc. 10th IEEE Int. Conf. Workshops Autom. Face Gesture Recognit. (FG), Apr. 2013, pp. 1– 8.
  139. [149]Lea C., Facker J. C., Hager G. D., Taylor R. H., and Saria S., “ 3D sensing algorithms towards building an intelligent intensive care unit,” in Proc. AMIA Summits Translational Sci., 2013, p, 136. PMC3845759 24303253
  140. [150]Twinanda A. P., Alkan E. O., Gangi A., and de Mathelin M., “ Data-driven spatio-temporal RGBD feature encoding for action recognition in operating rooms,” Int. J. Comput. Assist. Radiol. Surg., vol. 10, no. 6, pp. 737– 747, Jun. 2015. 25847670
  141. [151]Twinanda A. P., Winata P., Gangi A., Mathelin M. D., and Padoy N., “ Multi-stream deep architecture for surgical phase recognition on multi-view RGBD videos,” in Proc. M2CAI Workshop MICCAI, 2016, pp. 1– 8.
  142. [152]Srivastav V., Issenhuth T., Abdolrahim K., de Mathelin M., Gangi A., and Padoy N., “ MVOR: A multi-view RGB-D operating room dataset for 2D and 3D human pose estimation,” in Proc. MICCAI-LABELS, 2018, pp. 1– 10.
  143. [153]Issenhuth T., Srivastav V., Gangi A., and Padoy N., “ Face detection in the operating room: Comparison of state-of-the-art methods and a self-supervised approach,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 1049– 1058, 2019. 30968353
  144. [154]Srivastav V., Gangi A., and Padoy N., “ Privacy-preserving human pose estimation on low-resolution depth images,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2019.
  145. [155]Malpani A., Lea C., Chen C. C. G., and Hager G. D., “ System events: Readily accessible features for surgical phase detection,” Int. J. Comput. Assist. Radiol. Surg., vol. 11, no. 6, pp. 1201– 1209, 2016. 27177760
  146. [156]Gu Y., “ Language-based process phase detection in the trauma resuscitation,” in Proc. IEEE ICHI, Aug. 2017, pp. 239– 247. PMC6196035 30357019
  147. [157]Navab N., Hennersperger C., Frisch B., and Fuerst B., “ Personalized, relevance-based multimodal robotic imaging and augmented reality for computer assisted interventions,” Med. Image Anal., vol. 33, pp. 64– 71, Oct. 2016. 27475417
  148. [158]Lalys F. and Jannin P., “ Surgical process modelling: A review,” Int. J. Comput. Assist. Radiol. Surg., vol. 9, no. 3, pp. 495– 511, 2014. 24014322
  149. [159]Reiley C. E. and Hager G. D., “ Using robots to train the surgeons of tomorrow,” IEEE Spectr., to be published.
  150. [160]Lin H. C., Shafran I., Murphy T. E., Okamura A. M., Yuh D. D., and Hager G. D., “ Automatic detection and segmentation of robot-assisted surgical motions,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2005, pp. 802– 810. 16685920
  151. [161]Lin H. C., Shafran I., Yuh D., and Hager G. D., “ Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions,” Comput. Aided Surg., vol. 11, no. 5, pp. 220– 230, 2006. 17127647
  152. [162]Despinoy F., “ Unsupervised trajectory segmentation for surgical gesture recognition in robotic training,” IEEE Trans. Biomed. Eng., vol. 63, no. 6, pp. 1280– 1291, Oct. 2015. 26513773
  153. [163]Varadarajan B., Reiley C. E., Lin H., Khudanpur S., and Hager G. D., “ Data-derived models for segmentation with application to surgical assessment and training,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2009, pp. 426– 434. 20426016
  154. [164]Padoy N. and Hager G. D., “ Human-Machine Collaborative surgery using learned models,” in Proc. IEEE Int. Conf. Robot. Automat. (ICRA), Shanghai, China, May 2011, pp. 5285– 5292.
  155. [165]Birkmeyer J. D., “ Surgical skill and complication rates after bariatric surgery,” New England J. Med., vol. 369, no. 15, pp. 1434– 1442, 2013. 24106936
  156. [167]Kim T. S., “ Objective assessment of intraoperative technical skill in capsulorhexis using videos of cataract surgery,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 1097– 1105, 2019. 30977091
  157. [168]Fotouhi J., “ Interactive flying frustums (IFFs): Spatially aware surgical data visualization,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 913– 922, 2019. 30863981 PMC7357290
  158. [169]Matinfar S., “ Surgical soundtracks: Automatic acoustic augmentation of surgical procedures,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 9, pp. 1345– 1355, Sep. 2018. 30054775
  159. [170]Pauly O., Diotte B., Fallavollita P., Weidert S., Euler E., and Navab N., “ Machine learning-based augmented reality for improved surgical scene understanding,” Comput. Med. Imag. Graph., vol. 41, pp. 55– 60, Apr. 2015. 24998759
  160. [171]Malpani A., Vedula S. S., Lin H. C., Hager G. D., and Taylor R. H., “ Real-time teaching cues for automated surgical coaching,” 2017, arXiv:1704.07436. [Online]. Available: https://arxiv.org/abs/1704.07436 · arXiv
  161. [172]Reiter A., Allen P. K., and Zhao T., “ Feature classification for tracking articulated surgical tools,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2012, pp. 592– 600. 23286097
  162. [173]Du X., “ Articulated multi-instrument 2-D pose estimation using fully convolutional networks,” IEEE Trans. Med. Imag., vol. 37, no. 5, pp. 1276– 1287, 5 2018. PMC6051486 29727290
  163. [174]Pakhomov D., Premachandran V., Allan M., Azizian M., and Navab N., “ Deep residual learning for instrument segmentation in robotic surgery,” 2017, arXiv:1703.08580. [Online]. Available: https://arxiv.org/abs/1703.08580 · arXiv
  164. [175]Rieke N., “ Real-time localization of articulated surgical instruments in retinal microsurgery,” Med. Image Anal., vol. 34, pp. 82– 100, Dec. 2016. 27237604
  165. [176]Erickson T., “ Some problems with the notion of context-aware computing,” Commun. ACM, vol. 45, no. 2, pp. 102– 104, Feb. 2002.

Authors
Vercauteren, T., Unberath, M., Padoy, N., & Navab, N.
Affiliations
  • King's College London (GB)
  • Johns Hopkins University (US)
  • Centre National de la Recherche Scientifique (FR)
  • Technical University of Munich (DE)
Corresponding author
Vercauteren, Tom
Venue
Proceedings of the IEEE
Published
2020-01
Volume
108
Issue
1
Pages
198-214
ISSN
  • 0018-9219 (ISSN-L)
  • 1558-2256
OpenAlex
W2980852587
Cited by
125
References
171
Citations by year
2026: 14 · 2025: 17 · 2024: 9 · 2023: 24 · 2022: 17 · 2021: 24 · 2020: 20
FWCI
4.85
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the IEEE (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Portal (King's College London) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, cc-by)
  • Lirias (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Funders
  • Royal Academy of Engineering (RCSRF1819\7\34)
  • Wellcome Trust (203148/Z/16/Z, WT101957)
  • Engineering and Physical Sciences Research Council (NS/A000049/1, NS/A000027/1)
  • National Institute of Biomedical Imaging and Bioengineering (R01 EB0223939)
  • Johns Hopkins University
  • Fellowship of the Malone Center for Engineering in Healthcare, Johns Hopkins University
  • Agence Nationale de la Recherche (ANR-16-CE33-0009 (DeepSurg), ANR-18-CE45-0011-03 (OptimiX), ANR-11-LABX-0004 (Labex CAMI))
  • BPI France through Project CONDOR
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Surgery › Surgical Simulation and Training
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
Keywords
computer-based intervention, interventional imaging, surgical data science, human-AI collaboration, shared control, decision making under uncertainty, clinical workflow integration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review