CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions
Open accessAbstract
Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Machine learning methods, including deep neural networks, have enjoyed growing popularity across image analysis problems precisely because they do not require explicit definitions of complex functional mappings.
“Machine learning methods, including deep neural networks (NNs), have enjoyed a growing popularity across a variety of image analysis problems (Vercauteren et al., 2019), precisely because they do not require explicit definitions of complex functional mappings.”
From The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective (2021) · cites this paper as Vercauteren et al., 2019
The Introduction describes machine learning across multiple computer-assisted intervention tasks, and the Intelligent Imaging in Interventional Biophotonics section notes its increasing use in hyperspectral imaging. The Contextual Learning for Image Registration section gives an example of data-driven methods that avoid explicitly modeling image-intensity correlations. That example does not establish that avoiding explicit definitions of complex functional mappings is the general reason for machine learning’s growing popularity across image analysis. (LLM, conf=0.84)
Experimental — accuracy not yet measuredBecause image-based navigation techniques are designed to seamlessly integrate into conventional surgical workflows, their use should-in theory-not cause frustration or prolonged procedure times.
“Further, because image-based navigation techniques are designed to seamlessly integrate into conventional surgical workflows, their use should—in theory—not cause frustration or prolonged procedure times (Vercauteren et al., 2019).”
From The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective (2021) · cites this paper as Vercauteren et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 171 · Cited by 125
Research works printed but not linked: 165
- [1]Simpson A. L., “ A large annotated medical image dataset for the development and evaluation of segmentation algorithms,” Feb. 2019, arXiv:1902.09063. [Online]. Available: https://arxiv.org/abs/1902.09063 · arXiv
- [3]Maier-Hein L., “ Surgical data science for next-generation interventions,” Nature Biomed. Eng., vol. 1, no. 9, pp. 691– 696, 2017. 31015666
- [4]Vedula S. S. and Hager G. D., “ Surgical data science: The new knowledge domain,” Innov. Surgical Sci., vol. 2, no. 3, pp. 109– 121, 2007. PMC5602563 28936475
- [5]Maier-Hein L., “ Surgical data science: A consensus perspective,” 2018, arXiv:1806.03184. [Online]. Available: https://arxiv.org/abs/1806.03184 · arXiv
- [6]Peters T. and Cleary K., Image-Guided Interventions Technology and Applications. Boston, MA, USA: Springer, 2008. 20415592
- [7]West J., “ Comparison and evaluation of retrospective intermodality brain image registration techniques,” J. Comput. Assist. Tomogr., vol. 21, no. 4, pp. 554– 568, 1997. 9216759
- [9]Iglesias J. E. and Sabuncu M. R., “ Multi-atlas segmentation of biomedical images: A survey,” Med. Image Anal., vol. 24, no. 1, pp. 205– 219, 2015. 26201875 PMC4532640
- [10]Comaniciu D., Engel K., Georgescu B., and Mansi T., “ Shaping the future through innovations: From medical imaging to precision medicine,” Med. Image Anal., vol. 33, pp. 19– 26, Oct. 2016. 27349829
- [11]Li W., Wang G., Fidon L., Ourselin S., Cardoso M. J., and Vercauteren T., “ On the compactness, efficiency, and representation of 3D convolutional networks: Brain parcellation as a pretext task,” in Information Processing in Medical Imaging (Lecture Notes in Computer Science), vol. 10265 Cham, Switzerland: Springer-Verlag, Jun. 2017, pp. 348– 360.
- [12]Litjens G., “ A survey on deep learning in medical image analysis,” Med. Image Anal., vol. 42, pp. 60– 88, Dec. 2017. 28778026
- [13]Bakas S., “ Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BraTS challenge,” 2018, arXiv:1811.02629. [Online]. Available: https://arxiv.org/abs/1811.02629 · arXiv
- [14]Sheller M. J., Reina G. A., Edwards B., Martin J., and Bakas S., “ Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation,” in Proc. BrainLes Challenge MICCAI, 2018, pp. 92– 104. PMC6589345 31231720
- [15]Roy A. G., Siddiqui S., Pölsterl S., Navab N., and Wachinger C., “ BrainTorrent: A peer-to-peer environment for decentralized federated learning,” 2019, arXiv:1905.06731. [Online]. Available: https://arxiv.org/abs/1905.06731 · arXiv
- [16]Kamnitsas K., “ Unsupervised domain adaptation in brain lesion segmentation with adversarial networks,” in Proc. IPMI, 2017, pp. 597– 609.
- [17]Wang P., “ Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: A prospective randomised controlled study,” in Proc. Gut, 2019, pp. 1813– 1819. PMC6839720 30814121
- [19]Dorent R., Li W., Ekanayake J., Ourselin S., and Vercauteren T., “ Learning joint lesion and tissue segmentation from task-specific hetero-modal datasets,” in Proceedings of Machine Learning Research, vol. 102, Cardoso M. J. Eds. Cambridge, MA, USA: PMLR, 2019, pp. 164– 174.
- [20]Dorent R., Joutard S., Modat M., Ourselin S., and Vercauteren T., “ Hetero-modal variational encoder-decoder for joint modality completion and segmentation,” in Proc. MICCAI, 2019, pp. 74– 82.
- [21]Digioia M. D. A. M. III, Jaramaz B., Nikou C., Labarca R. S., Moody J. E., and Colgan B. D., “ Surgical navigation for total hip replacement with the use of HipNav,” Operative Techn. Orthopaedics, vol. 10, no. 1, pp. 3– 8, 2000.
- [22]Boudissa M., Courvoisier A., Chabanas M., and Tonetti J., “ Computer assisted surgery in preoperative planning of acetabular fracture surgery: State of the art,” Expert Rev. Med. Devices, vol. 15, no. 1, pp. 81– 89, 2018. 29206497
- [23]Kulyk P., Vlachopoulos L., Fürnstahl P., and Zheng G., “ Fully automatic planning of total shoulder arthroplasty without segmentation: A deep learning based approach,” in Computational Methods and Clinical Applications in Musculoskeletal Imaging, Vrtovec T., Yao J., Zheng G., and Pozo J. M., Eds. Cham, Switzerland: Springer, 2019, pp. 22– 34.
- [24]Carrillo F., Vlachopoulos L., Schweizer A., Nagy L., Snedeker J., and Fürnstahl P., “ A time saver: Optimization approach for the fully automatic 3D planning of forearm osteotomies,” in Medical Image Computing and Computer Assisted Intervention—MICCAI, Descoteaux M., Maier-Hein L., Franz A., Jannin P., Collins D. L., and Duchesne S., Eds. Cham, Switzerland: Springer, 2017, pp. 488– 496.
- [25]Sparks R., “ Automated multiple trajectory planning algorithm for the placement of stereo-electroencephalography (SEEG) electrodes in epilepsy treatment,” Int. J. Comput. Assist. Radiol. Surg., vol. 12, no. 1, pp. 123– 136, Jan. 2017. 27368184 PMC5216164
- [26]Vakharia V. N., “ Automated trajectory planning for laser interstitial thermal therapy in mesial temporal lobe epilepsy,” Epilepsia, vol. 59, no. 4, pp. 814– 824, 2018. 29528488 PMC5901027
- [27]Granados A., “ A machine learning approach to predict instrument bending in stereotactic neurosurgery,” in Medical Image Computing and Computer Assisted Intervention—MICCAI, Frangi A. F., Schnabel J. A., Davatzikos C., Alberola-López C., and Fichtinger G., Eds. Cham, Switzerland: Springer, 2018, pp. 238– 246.
- [28]Giffard-Roisin S., “ Transfer learning from simulations on a reference anatomy for ECGI in personalized cardiac resynchronization therapy,” IEEE Trans. Biomed. Eng., vol. 66, no. 2, pp. 343– 353, Feb. 2018. 29993409
- [29]Ramalhinho J., “ A pre-operative planning framework for global registration of laparoscopic ultrasound to CT images,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 8, pp. 1177– 1186, Aug. 2018. 29860550 PMC6096745
- [30]Johnsen S. F., “ Database-based estimation of liver deformation under pneumoperitoneum for surgical image-guidance and simulation,” in Medical Image Computing and Computer Assisted Intervention—MICCAI, Navab N., Hornegger J., Wells W. M., and Frangi A., Eds. Cham, Switzerland: Springer, 2015, pp. 450– 458.
- [31]März K., “ Toward knowledge-based liver surgery: Holistic information processing for surgical decision support,” Int. J. Comput. Assist. Radiol. Surg., vol. 10, no. 6, pp. 749– 759, Jun. 2015. 25847671
- [32]Koivukangas T., Katisko J. P., and Koivukangas J. P., “ Technical accuracy of optical and the electromagnetic tracking systems,” SpringerPlus, vol. 2, no. 1, p. 90, 2013. 23586003 PMC3622743
- [33]Jolesz F. A., Intraoperative Imaging and Image-Guided Therapy. New York, NY, USA: Springer, 2014.
- [34]Pluim J. P. W. and Fitzpatrick J. M., “ Image registration,” IEEE Trans. Med. Imag., vol. 22, no. 11, pp. 1341– 1343, Nov. 2003. 14606668
- [35]Liao R., Zhang L., Sun Y., Miao S., and Chefd C., “ A review of recent advances in registration techniques applied to minimally invasive therapy,” IEEE Trans. Multimedia, vol. 15, no. 5, pp. 983– 1000, Aug. 2013.
- [36]Joskowicz L. and Hazan E. J., “ Computer aided orthopaedic surgery: Incremental shift or paradigm change?” Med. Image Anal., vol. 33, pp. 84– 90, Oct. 2016. 27407004
- [37]Fuerst B., Wein W., Müller M., and Navab N., “ Automatic ultrasound’MRI registration for neurosurgery using the 2D and 3D LC2 metric,” Med. Image Anal., vol. 18, no. 8, pp. 1312– 1319, 2014. 24842859
- [38]Wein W., “ Brain-shift correction with image-based registration and landmark accuracy evaluation,” in Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation. Cham, Switzerland: Springer, 2018, pp. 146– 151.
- [40]Balakrishnan G., Zhao A., Sabuncu M. R., Guttag J., and Dalca A. V., “ VoxelMorph: A learning framework for deformable medical image registration,” IEEE Trans. Med. Imag., vol. 38, no. 8, pp. 1788– 1800, Aug. 2019. 30716034
- [41]Hu Y., “ Weakly-supervised convolutional neural networks for multimodal image registration,” Med. Image Anal., vol. 49, pp. 1– 13, Oct. 2018. 30007253 PMC6742510
- [42]Hu Y., “ Adversarial deformation regularization for training image registration neural networks,” in Medical Image Computing and Computer Assisted Intervention—MICCAI (Lecture Notes in Computer Science), vol. 11070 Cham, Switzerland: Springer, 2018, pp. 774– 782.
- [43]Pfeiffer M., Riediger C., Weitz J., and Speidel S., “ Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 7, pp. 1147– 1155, Jul. 2019. 30993520
- [44]Brunet J.-N., Mendizabal A., Petit A., Golse N., Vibert E., and Cotin S., “ Physics-based deep neural network for augmented reality during liver surgery,” in Medical Image Computing and Computer Assisted Intervention—MICCAI (Lecture Notes in Computer Science). Cham, Switzerland: Springer, 2019.
- [45]Hu Y., Gibson E., Barratt D. C., Emberton M., Noble J. A., and Vercauteren T., “ Conditional segmentation in lieu of image registration,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2019.
- [48]Bui M., Albarqouni S., Schrapp M., Navab N., and Ilic S., “ X-Ray PoseNet: 6 DoF pose estimation for mobile X-Ray devices,” in Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), Mar. 2017, pp. 1036– 1044.
- [50]Prevost R., “ 3D freehand ultrasound without external tracking using deep learning,” Med. Image Anal., vol. 48, pp. 187– 202, Aug. 2018. 29936399
- [51]Bier B., “ X-ray-transform invariant anatomical landmark detection for pelvic trauma surgery,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2018, pp. 55– 63.
- [52]Bier B., “ Detecting anatomical landmarks for motion estimation in weight-bearing imaging of knees,” in Proc. Int. Workshop Mach. Learn. Med. Image Reconstruct. Cham, Switzerland: Springer, 2018, pp. 83– 90.
- [53]Gao C., Unberath M., Taylor R., and Armand M., “ Localizing dexterous surgical tools in X-ray for image-based navigation,” in Proc. IPCAI. Cham, Switzerland: Springer, 2019, pp. 1– 4.
- [54]Kügler D., Stefanov A., and Mukhopadhyay A., “ i3PosNet: Instrument pose estimation from X-ray,” 2018, arXiv:1802.09575. [Online]. Available: https://arxiv.org/abs/1802.09575 PMC7316684 32440956 · arXiv
- [55]Esfandiari H., Newell R., Anglin C., Street J., and Hodgson A. J., “ A deep learning framework for segmentation and pose estimation of pedicle screw implants based on C-arm fluoroscopy,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 8, pp. 1269– 1282, 2018. 29808466
- [56]Liao H., Lin W.-A., Zhang J., Zhang J., Luo J., and Zhou S. K., “ Multiview 2D/3D rigid registration via a point-of-interest network for tracking and triangulation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2019, pp. 12638– 12647.
- [57]Unberath M., “ Enabling machine learning in X-ray-based procedures via realistic simulation of image formation,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 9, pp. 1517– 1528, 2019. 31187399 PMC7297499
- [58]Clunie D. A., “ Technical challenges of enterprise imaging: HIMSS-SIIM collaborative white paper,” J. Digit. Imag., vol. 29, no. 5, pp. 583– 614, 2016. PMC5023533 27576909
- [59]Murad F. M., “ Image management systems,” Gastrointestinal Endoscopy, vol. 79, no. 1, pp. 15– 22, 2014. 24239253
- [60]Deganello A., Sellars M. E., Yusuf G. T., and Sidhu P. S., “ How much should i record during a CEUS examination? Practical aspects of the ‘real-time’ feature of a contrast ultrasound study,” Ultraschall Med., vol. 39, no. 5, pp. 484– 486, 2018. 30300906
- [61]Milletari F., Birodkar V., and Sofka M., “ Straight to the point: Reinforcement learning for user guidance in ultrasound,” 2019, arXiv:1903.00586. [Online]. Available: https://arxiv.org/abs/1903.00586 · arXiv
- [62]Hu Y., “ Freehand ultrasound image simulation with spatially-conditioned generative adversarial networks,” in Proc. RAMBO Workshop MICCAI (Lecture Notes in Computer Science), vol. 10555 Cham, Switzerland: Springer-Verlag, Sep. 2017, pp. 105– 115.
- [63]Engelhardt S., De Simone R., Full P. M., Karck M., and Wolf I., “ Improving surgical training phantoms by hyperrealism: Deep unpaired image-to-image translation from real surgeries,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. 2018, pp. 747– 755.
- [64]Luengo I., Flouty E., Giataganas P., Wisanuvej P., Nehme J., and Stoyanov D., “ SurReal: Enhancing surgical simulation realism using style transfer,” in Proc. Brit. Mach. Vis. Conf., Newcastle, U.K., Sep. 2018, p. 116.
- [65]Mahmood F., Chen R., and Durr N. J., “ Unsupervised reverse domain adaptation for synthetic medical images via adversarial training,” IEEE Trans. Med. Imag., vol. 37, no. 12, pp. 2572– 2581, Dec. 2018. 29993538
- [66]Zhang Y., Miao S., Mansi T., and Liao R., “ Task driven generative modeling for unsupervised domain adaptation: Application to X-ray image segmentation,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2018.
- [67]Unberath M., “ DeepDRR—A catalyst for machine learning in fluoroscopy-guided procedures,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2018.
- [68]Mahmood F., Chen R., Sudarsky S., Yu D., and Durr N. J., “ Deep learning with cinematic rendering: Fine-tuning deep neural networks using photorealistic medical images,” 2018, arXiv:1805.08400. [Online]. Available: https://arxiv.org/abs/1805.08400 30113015 · arXiv
- [69]Nair A. A., Tran T. D., Reiter A., and Bell M. A. L., “ A deep learning based alternative to beamforming ultrasound images,” in Proc. ICASSP, 2018, pp. 3359– 3363.
- [70]Toth D., Cimen S., Ceccaldi P., Kurzendorfer T., Rhode K., and Mountney P., “ Training deep networks on domain randomized synthetic X-ray data for cardiac interventions,” in Proceedings of Machine Learning Research, vol. 102, Cardoso M. J. Eds. Cambridge, MA, USA: PMLR, 2019, pp. 468– 482.
- [71]Jermyn M., “ Intraoperative brain cancer detection with Raman spectroscopy in humans,” Sci. Transl. Med., vol. 7, no. 274, p. 274ra19, 2015. 25673764
- [72]Jermyn M., “ Neural networks improve brain cancer detection with Raman spectroscopy in the presence of operating room light artifacts,” J. Biomed. Opt., vol. 21, no. 9, p. 094002, 2016. 27604560
- [73]Banbury C., “ Development of the self optimising Kohonen index network (SKiNET) for Raman spectroscopy based detection of anatomical eye tissue,” Sci. Rep., vol. 9, no. 1, p. 10812, 2019. 31346227 PMC6658481
- [74]Shapey J., “ Intraoperative multispectral and hyperspectral label-free imaging: A systematic review of in vivo clinical studies,” J. Biophoton., vol. 12, no. 9, p. e201800455, 2019. PMC6736677 30859757
- [75]Adler T. J., “ Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 997– 1007, 2019. 30903566
- [76]Ravì D., Fabelo H., Callic G. M., and Yang G.-Z., “ Manifold embedding and semantic segmentation for intraoperative guidance with hyperspectral brain imaging,” IEEE Trans. Med. Imag., vol. 36, no. 9, pp. 1845– 1857, Sep. 2017. 28436854
- [77]Lin J., “ Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks,” Med. Image Anal., vol. 48, pp. 162– 176, Aug. 2018. 29933116
- [78]Ravì D., Szczotka A. B., Pereira S. P., and Vercauteren T., “ Adversarial training with cycle consistency for unsupervised super-resolution in endomicroscopy,” Med. Image Anal., vol. 53, pp. 123– 131, Apr. 2019. 30769327 PMC6873642
- [79]Szczotka A. B., Ravì D., Shakir D. I., Pereira S. P., and Vercauteren T., “ Effective deep learning training for single-image super-resolution in endomicroscopy exploiting video-registration-based reconstruction,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 6, pp. 917– 924, Jun. 2018. 29687176 PMC5973979
- [80]Allman D., Assis F., Chrispin J., and Bell M. A. L., “ Deep neural networks to remove photoacoustic reflection artifacts in ex vivo and in vivo tissue,” in Proc. IEEE Int. Ultrason. Symp. (IUS), Oct. 2018, pp. 1– 4.
- [81]Ambrosini P., Ruijters D., Niessen W. J., Moelker A., and van Walsum T., “ Fully automatic and real-time catheter segmentation in X-ray fluoroscopy,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2017, pp. 577– 585.
- [82]Breininger K., “ Multiple device segmentation for fluoroscopic imaging using multi-task learning,” in Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis. Cham, Switzerland: Springer, 2018, pp. 19– 27.
- [83]Sa R., “ Intervertebral disc detection in X-ray images using faster R-CNN,” in Proc. 39th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC), Jul. 2017, pp. 564– 567. 29059935
- [84]Matviychuk Y., “ Learning a multiscale patch-based representation for image denoising in X-ray fluoroscopy,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Sep. 2016, pp. 2330– 2334.
- [85]Hariharan S. G., “ Learning-based X-ray image denoising utilizing model-based image simulations,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2019.
- [86]Anas E. M. A., Mousavi P., and Abolmaesumi P., “ A deep learning approach for real time prostate segmentation in freehand ultrasound guided biopsy,” Med. Image Anal., vol. 48, pp. 107– 116, Aug. 2018. 29886268
- [87]Mwikirize C., Nosher J. L., and Hacihaliloglu I., “ Convolution neural networks for real-time needle detection and localization in 2D ultrasound,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 5, pp. 647– 657, 2018. 29512006
- [88]Pesteie M., Lessoway V., Abolmaesumi P., and Rohling R. N., “ Automatic localization of the needle target for ultrasound-guided epidural injections,” IEEE Trans. Med. Imag., vol. 37, no. 1, pp. 81– 92, Jan. 2017. 28809679
- [89]Luchies A. C. and Byram B. C., “ Deep neural networks for ultrasound beamforming,” IEEE Trans. Med. Imag., vol. 37, no. 9, pp. 2010– 2021, Sep. 2018. PMC6109603 29994441
- [90]Simson W., “ End-to-end learning-based ultrasound reconstruction,” 2019, arXiv:1904.04696. [Online]. Available: https://arxiv.org/abs/1904.04696 · arXiv
- [91]Simson W., Paschali M., Navab N., and Zahnd G., “ Deep learning beamforming for sub-sampled ultrasound data,” in Proc. IEEE Int. Ultrason. Symp. (IUS), Oct. 2018, pp. 1– 4.
- [92]Zaech J.-N., “ Learning to avoid poor images: Towards task-aware C-arm cone-beam CT trajectories,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2019.
- [93]Milletari F., Rieke N., Baust M., Esposito M., and Navab N., “ CFCM: Segmentation via coarse to fine context memory,” 2018, arXiv:1806.01413. [Online]. Available: https://arxiv.org/abs/1806.01413 · arXiv
- [94]Blum T., Padoy N., Feußner H., and Navab N., “ Modeling and online recognition of surgical phases using hidden Markov models,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2008, pp. 627– 635. 18982657
- [95]Lalys F., Riffaud L., Bouget D., and Jannin P., “ A framework for the recognition of high-level surgical tasks from video images for cataract surgeries,” IEEE Trans. Biomed. Eng., vol. 59, no. 4, pp. 966– 976, Dec. 2012. 22203700 PMC3432023
- [96]Quellec G., Lamard M., Cochener B., and Cazuguel G., “ Real-time segmentation and recognition of surgical tasks in cataract surgery videos,” IEEE Trans. Med. Imag., vol. 33, no. 12, pp. 2352– 2360, Dec. 2014. 25055383
- [97]Twinanda A. P., Shehata S., Mutter D., Marescaux J., de Mathelin M., and Padoy N., “ EndoNet: A deep architecture for recognition tasks on laparoscopic videos,” IEEE Trans. Med. Imag., vol. 36, no. 1, pp. 86– 97, Jan. 2017. 27455522
- [98]Volkov M., Hashimoto D. A., Rosman G., Meireles O. R., and Rus D., “ Machine learning and coresets for automated real-time video segmentation of laparoscopic and robot-assisted surgery,” in Proc. ICRA, 2017, pp. 754– 759.
- [99]Bodenstedt S., “ Active learning using deep Bayesian networks for surgical workflow analysis,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 1079– 1087, 2019. 30968355
- [100]Padoy N., Blum T., Ahmadi S.-A., Feussner H., Berger M.-O., and Navab N., “ Statistical modeling and recognition of surgical workflow,” Med. Image Anal., vol. 16, no. 3, pp. 632– 641, 2012. 21195015
- [101]Twinanda A. P., Yengera G., Mutter D., Marescaux J., and Padoy N., “ RSDNet: Learning to predict remaining surgery duration from laparoscopic videos without manual annotations,” IEEE Trans. Med. Imag., vol. 38, no. 4, pp. 1069– 1078, Apr. 2019. 30371356
- [102]Bodenstedt S., “ Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 1089– 1095, 2019. 30968352
- [103]Al Hajj H., Lamard M., Conze P.-H., Cochener B., and Quellec G., “ Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks,” Med. Image Anal., vol. 47, pp. 203– 218, Jul. 2018. 29778931
- [104]Nwoye C. I., Mutter D., Marescaux J., and Padoy N., “ Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 1059– 1067, 2019. 30968356
- [105]Leibetseder A., Primus M. J., and Schoeffmann K., “ Automatic smoke classification in endoscopic video,” in Proc. 24th Int. Conf. MultiMedia Modeling (MMM), Bangkok, Thailand, Feb. 2018, pp. 362– 366.
- [106]Okamoto T., Ohnishi T., Kawahira H., Dergachyava O., Jannin P., and Haneishi H., “ Real-time identification of blood regions for hemostasis support in laparoscopic surgery,” Signal, Image Video Process., vol. 13, no. 2, pp. 405– 412, 2019.
- [107]Kannan S., Yengera G., Mutter D., Marescaux J., and Padoy N., “ Future-state predicting LSTM for early surgery type recognition,” IEEE Trans. Med. Imag., to be published. 31352339
- [108]Funke I., Jenke A., Mees S. T., Weitz J., Speidel S., and Bodenstedt S., “ Temporal coherence-based self-supervised learning for laparoscopic workflow analysis,” in Proc. 1st Int. Workshop, OR 2.0 5th Int. Workshop (CARE), 7th Int. Workshop (CLIP), 3rd Int. Workshop (ISIC), Granada, Spain, Sep. 2018, pp. 85– 93.
- [109]Yengera G., Mutter D., Marescaux J., and Padoy N., “ Less is more: Surgical phase recognition with less annotations through self-supervised pre-training of CNN-LSTM networks,” 2018, arXiv:1805.08569. [Online]. Available: https://arxiv.org/abs/1805.08569 · arXiv
- [110]Roß T., “ Exploiting the potential of unlabeled endoscopic video data with self-supervised learning,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 6, pp. 925– 933, 2018. 29704196
- [111]Yu T., Mutter D., Marescaux J., and Padoy N., “ Learning from a tiny dataset of manual annotations: A teacher/student approach for surgical phase recognition,” in Proc. IPCAI, 2019, pp. 1– 13.
- [112]Liu X., “ Self-supervised learning for dense depth estimation in monocular endoscopy,” in OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis. Cham, Switzerland: Springer, 2018, pp. 128– 138.
- [113]Ntourakis D., Memeo R., Soler L., Marescaux J., Mutter D., and Pessaux P., “ Augmented reality guidance for the resection of missing colorectal liver metastases: An initial experience,” World J. Surg., vol. 40, no. 2, pp. 419– 426, 2015. 26316112
- [114]Bouget D., Allan M., Stoyanov D., and Jannin P., “ Vision-based and marker-less surgical tool detection and tracking: A review of the literature,” Med. Image Anal., vol. 35, pp. 633– 654, Jan. 2017. 27744253
- [115]Sarikaya D., Corso J. J., and Guru K. A., “ Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection,” IEEE Trans. Med. Imag., vol. 36, no. 7, pp. 1542– 1549, 2017. 28186883
- [116]Kurmann T., “ Simultaneous recognition and pose estimation of instruments in minimally invasive surgery,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2017, pp. 505– 513.
- [117]Jin A., “ Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks,” in Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), Lake Tahoe, NV, USA, Mar. 2018, pp. 691– 699.
- [119]Laina I., “ Concurrent segmentation and localization for tracking of surgical instruments,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2017, pp. 664– 672.
- [120]Bodenstedt S., “ Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery,” 2018 arXiv:1805.02475. [Online]. Available: https://arxiv.org/abs/1805.02475 · arXiv
- [121]Allan M., “ 2017 robotic instrument segmentation challenge,” 2019, arXiv:1902.06426. [Online]. Available: https://arxiv.org/abs/1902.06426 · arXiv
- [122]Lejeune L., Grossrieder J., and Sznitman R., “ Iterative multi-path tracking for video and volume segmentation with sparse point supervision,” Med. Image Anal., vol. 50, pp. 65– 81, Dec. 2018. 30212738
- [123]Fuentes-Hurtado F., Kadkhodamohammadi A., Flouty E., Barbarisi S., Luengo I., and Stoyanov D., “ EasyLabels: Weak labels for scene segmentation in laparoscopic videos,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 7, pp. 1247– 1257, 2019. 31165349
- [124]Flouty E., “ CaDIS: Cataract dataset for image segmentation,” (2019), arXiv:1906.11586. [Online]. Available: https://arxiv.org/abs/1906.11586 · arXiv
- [125]Gibson E., “ Deep residual networks for automatic segmentation of laparoscopic videos of the liver,” Proc. SPIE, vol. 10135, p. 101351M, Mar. 2017.
- [126]Everson M. A., “ Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: A proof-of-concept study,” United Eur. Gastroenterol. J., vol. 7, no. 2, pp. 297– 306, 2019. PMC6498793 31080614
- [127]Misawa M., “ Artificial intelligence-assisted polyp detection for colonoscopy: Initial experience,” Gastroenterology, vol. 154, no. 8, pp. 2027– 2029, 2018. 29653147
- [128]Bernal J., “ Comparative validation of polyp detection methods in video colonoscopy: Results from the MICCAI 2015 endoscopic vision challenge,” IEEE Trans. Med. Imag., vol. 36, no. 6, pp. 1231– 1249, Feb. 2017. 28182555
- [129]Mahmood F., Yang Z., Chen R., Borders D., Xu W., and Durr N. J., “ Polyp segmentation and classification using predicted depth from monocular endoscopy,” Proc. SPIE, vol. 10950, p. 1095011, Mar. 2019.
- [130]Leonard S., “ Evaluation and stability analysis of video-based navigation system for functional endoscopic sinus surgery on in vivo clinical data,” IEEE Trans. Med. Imag., vol. 37, no. 10, pp. 2185– 2195, 5 2018. 29993881
- [131]Visentini-Scarzanella M., Sugiura T., Kaneko T., and Koto S., “ Deep monocular 3D reconstruction for assisted navigation in bronchoscopy,” Int. J. Comput. Assist. Radiol. Surg., vol. 12, no. 7, pp. 1089– 1099, 2017. 28508345
- [132]Wang J., Song S., Ren H., Lim C. M., and Meng M. Q.-H., “ Surgical instrument tracking by multiple monocular modules and a sensor fusion approach,” IEEE Trans. Autom. Sci. Eng., vol. 16, no. 2, pp. 629– 639, Apr. 2019.
- [133]Liu X., “ Self-supervised learning for dense depth estimation in monocular endoscopy,” 2019, arXiv:1902.07766. [Online]. Available: https://arxiv.org/abs/1806.09521 PMC7289272 31689184 · arXiv
- [134]Rodas N. L., Barrera F., and Padoy N., “ See it with your own eyes: Markerless mobile augmented reality for radiation awareness in the hybrid room,” IEEE Trans. Biomed. Eng., vol. 64, no. 2, pp. 429– 440, Feb. 2017. 27164565
- [135]James J. T., “ A new, evidence-based estimate of patient harms associated with hospital care,” J. Patient Saf., vol. 9, no. 3, pp. 122– 128, 2013. 23860193
- [136]Suliburk J. W., Buck Q. M., and Pirko C. J., “ Analysis of human performance deficiencies associated with surgical adverse events,” JAMA Netw. Open, vol. 2, no. 7, p. e198067, 2019. 31365107 PMC6669897
- [138]Padoy N., “ Machine and deep learning for workflow recognition during surgery,” Minimally Invasive Therapy Allied Technol., vol. 28, no. 2, pp. 82– 90, 2019. 30849261
- [139]Jannin P. and Morandi X., “ Surgical models for computer-assisted neurosurgery,” NeuroImage, vol. 37, no. 3, pp. 783– 791, 2007. 17613249
- [140]Blum T., Padoy N., Feußner H., and Navab N., “ Workflow mining for visualization and analysis of surgeries,” Int. J. Comput. Assist. Radiol. Surg., vol. 3, no. 5, pp. 379– 386, 2008.
- [141]Gibaud B., “ Toward a standard ontology of surgical process models,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 9, pp. 1397– 1408, 2018. 30006820
- [142]Haque A., “ Towards vision-based smart hospitals: A system for tracking and monitoring hand hygiene compliance,” in Proc. Mach. Learn. Res. Mach. Learn. Healthcare Conf. (MLHC), 2017, pp. 75– 87.
- [143]Ma A. J., “ Measuring patient mobility in the ICU using a novel noninvasive sensor,” Crit. care Med., vol. 45, no. 4, pp. 630– 636, 2017. 28291092 PMC5697716
- [145]Haque A., Peng B., Luo Z., Alahi A., Yeung S., and Li F., “ Viewpoint invariant 3D human pose estimation with recurrent error feedback,” in Proc. ECCV, 2016, pp. 160– 177.
- [146]Belagiannis V., “ Parsing human skeletons in an operating room,” Mach. Vis. Appl., vol. 27, no. 7, p. 1035– 1046, Oct. 2016.
- [147]Kadkhodamohammadi A., Gangi A., de Mathelin M., and Padoy N., “ A multi-view RGB-D approach for human pose estimation in operating rooms,” in Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), Mar. 2017, pp. 363– 372.
- [148]Chakraborty I., Elgammal A., and Burd R. S., “ Video based activity recognition in trauma resuscitation,” in Proc. 10th IEEE Int. Conf. Workshops Autom. Face Gesture Recognit. (FG), Apr. 2013, pp. 1– 8.
- [149]Lea C., Facker J. C., Hager G. D., Taylor R. H., and Saria S., “ 3D sensing algorithms towards building an intelligent intensive care unit,” in Proc. AMIA Summits Translational Sci., 2013, p, 136. PMC3845759 24303253
- [150]Twinanda A. P., Alkan E. O., Gangi A., and de Mathelin M., “ Data-driven spatio-temporal RGBD feature encoding for action recognition in operating rooms,” Int. J. Comput. Assist. Radiol. Surg., vol. 10, no. 6, pp. 737– 747, Jun. 2015. 25847670
- [151]Twinanda A. P., Winata P., Gangi A., Mathelin M. D., and Padoy N., “ Multi-stream deep architecture for surgical phase recognition on multi-view RGBD videos,” in Proc. M2CAI Workshop MICCAI, 2016, pp. 1– 8.
- [152]Srivastav V., Issenhuth T., Abdolrahim K., de Mathelin M., Gangi A., and Padoy N., “ MVOR: A multi-view RGB-D operating room dataset for 2D and 3D human pose estimation,” in Proc. MICCAI-LABELS, 2018, pp. 1– 10.
- [153]Issenhuth T., Srivastav V., Gangi A., and Padoy N., “ Face detection in the operating room: Comparison of state-of-the-art methods and a self-supervised approach,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 1049– 1058, 2019. 30968353
- [154]Srivastav V., Gangi A., and Padoy N., “ Privacy-preserving human pose estimation on low-resolution depth images,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2019.
- [155]Malpani A., Lea C., Chen C. C. G., and Hager G. D., “ System events: Readily accessible features for surgical phase detection,” Int. J. Comput. Assist. Radiol. Surg., vol. 11, no. 6, pp. 1201– 1209, 2016. 27177760
- [156]Gu Y., “ Language-based process phase detection in the trauma resuscitation,” in Proc. IEEE ICHI, Aug. 2017, pp. 239– 247. PMC6196035 30357019
- [157]Navab N., Hennersperger C., Frisch B., and Fuerst B., “ Personalized, relevance-based multimodal robotic imaging and augmented reality for computer assisted interventions,” Med. Image Anal., vol. 33, pp. 64– 71, Oct. 2016. 27475417
- [158]Lalys F. and Jannin P., “ Surgical process modelling: A review,” Int. J. Comput. Assist. Radiol. Surg., vol. 9, no. 3, pp. 495– 511, 2014. 24014322
- [159]Reiley C. E. and Hager G. D., “ Using robots to train the surgeons of tomorrow,” IEEE Spectr., to be published.
- [160]Lin H. C., Shafran I., Murphy T. E., Okamura A. M., Yuh D. D., and Hager G. D., “ Automatic detection and segmentation of robot-assisted surgical motions,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2005, pp. 802– 810. 16685920
- [161]Lin H. C., Shafran I., Yuh D., and Hager G. D., “ Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions,” Comput. Aided Surg., vol. 11, no. 5, pp. 220– 230, 2006. 17127647
- [162]Despinoy F., “ Unsupervised trajectory segmentation for surgical gesture recognition in robotic training,” IEEE Trans. Biomed. Eng., vol. 63, no. 6, pp. 1280– 1291, Oct. 2015. 26513773
- [163]Varadarajan B., Reiley C. E., Lin H., Khudanpur S., and Hager G. D., “ Data-derived models for segmentation with application to surgical assessment and training,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2009, pp. 426– 434. 20426016
- [164]Padoy N. and Hager G. D., “ Human-Machine Collaborative surgery using learned models,” in Proc. IEEE Int. Conf. Robot. Automat. (ICRA), Shanghai, China, May 2011, pp. 5285– 5292.
- [165]Birkmeyer J. D., “ Surgical skill and complication rates after bariatric surgery,” New England J. Med., vol. 369, no. 15, pp. 1434– 1442, 2013. 24106936
- [167]Kim T. S., “ Objective assessment of intraoperative technical skill in capsulorhexis using videos of cataract surgery,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 1097– 1105, 2019. 30977091
- [168]Fotouhi J., “ Interactive flying frustums (IFFs): Spatially aware surgical data visualization,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, no. 6, pp. 913– 922, 2019. 30863981 PMC7357290
- [169]Matinfar S., “ Surgical soundtracks: Automatic acoustic augmentation of surgical procedures,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, no. 9, pp. 1345– 1355, Sep. 2018. 30054775
- [170]Pauly O., Diotte B., Fallavollita P., Weidert S., Euler E., and Navab N., “ Machine learning-based augmented reality for improved surgical scene understanding,” Comput. Med. Imag. Graph., vol. 41, pp. 55– 60, Apr. 2015. 24998759
- [171]Malpani A., Vedula S. S., Lin H. C., Hager G. D., and Taylor R. H., “ Real-time teaching cues for automated surgical coaching,” 2017, arXiv:1704.07436. [Online]. Available: https://arxiv.org/abs/1704.07436 · arXiv
- [172]Reiter A., Allen P. K., and Zhao T., “ Feature classification for tracking articulated surgical tools,” in Medical Image Computing and Computer Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2012, pp. 592– 600. 23286097
- [173]Du X., “ Articulated multi-instrument 2-D pose estimation using fully convolutional networks,” IEEE Trans. Med. Imag., vol. 37, no. 5, pp. 1276– 1287, 5 2018. PMC6051486 29727290
- [174]Pakhomov D., Premachandran V., Allan M., Azizian M., and Navab N., “ Deep residual learning for instrument segmentation in robotic surgery,” 2017, arXiv:1703.08580. [Online]. Available: https://arxiv.org/abs/1703.08580 · arXiv
- [175]Rieke N., “ Real-time localization of articulated surgical instruments in retinal microsurgery,” Med. Image Anal., vol. 34, pp. 82– 100, Dec. 2016. 27237604
- [176]Erickson T., “ Some problems with the notion of context-aware computing,” Commun. ACM, vol. 45, no. 2, pp. 102– 104, Feb. 2002.
- Authors
- Vercauteren, T., Unberath, M., Padoy, N., & Navab, N.
- Affiliations
- King's College London (GB)
- Johns Hopkins University (US)
- Centre National de la Recherche Scientifique (FR)
- Technical University of Munich (DE)
- Corresponding author
- Vercauteren, Tom
- Venue
- Proceedings of the IEEE
- Published
- 2020-01
- Volume
- 108
- Issue
- 1
- Pages
- 198-214
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- PMC
- PMC6952279
- OpenAlex
- W2980852587
- Cited by
- 125
- References
- 171
- Citations by year
- 2026: 14 · 2025: 17 · 2024: 9 · 2023: 24 · 2022: 17 · 2021: 24 · 2020: 20
- FWCI
- 4.85
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the IEEE (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Portal (King's College London) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, cc-by)
- Lirias (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Funders
- Royal Academy of Engineering (RCSRF1819\7\34)
- Wellcome Trust (203148/Z/16/Z, WT101957)
- Engineering and Physical Sciences Research Council (NS/A000049/1, NS/A000027/1)
- National Institute of Biomedical Imaging and Bioengineering (R01 EB0223939)
- Johns Hopkins University
- Fellowship of the Malone Center for Engineering in Healthcare, Johns Hopkins University
- Agence Nationale de la Recherche (ANR-16-CE33-0009 (DeepSurg), ANR-18-CE45-0011-03 (OptimiX), ANR-11-LABX-0004 (Labex CAMI))
- BPI France through Project CONDOR
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Surgery › Surgical Simulation and Training
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
- Keywords
- computer-based intervention, interventional imaging, surgical data science, human-AI collaboration, shared control, decision making under uncertainty, clinical workflow integration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review