SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network
The DOI matches its Crossref record; 48 of the 50 references it cites were found in a registry or an index.
Abstract
We propose an analysis of surgical videos that is based on a novel recurrent convolutional network (SV-RCNet), specifically for automatic workflow recognition from surgical videos online, which is a key component for developing the context-aware computer-assisted intervention systems. Different from previous methods which harness visual and temporal information separately, the proposed SV-RCNet seamlessly integrates a convolutional neural network (CNN) and a recurrent neural network (RNN) to form a novel recurrent convolutional architecture in order to take full advantages of the complementary information of visual and temporal features learned from surgical videos. We effectively train the SV-RCNet in an end-to-end manner so that the visual representations and sequential dynamics can be jointly optimized in the learning process. In order to produce more discriminative spatio-temporal features, we exploit a deep residual network (ResNet) and a long short term memory (LSTM) network, to extract visual features and temporal dependencies, respectively, and integrate them into the SV-RCNet. Moreover, based on the phase transition-sensitive predictions from the SV-RCNet, we propose a simple yet effective inference scheme, namely the prior knowledge inference (PKI), by leveraging the natural characteristic of surgical video. Such a strategy further improves the consistency of results and largely boosts the recognition performance. Extensive experiments have been conducted with the MICCAI 2016 Modeling and Monitoring of Computer Assisted Interventions Workflow Challenge dataset and Cholec80 dataset to validate SV-RCNet. Our approach not only achieves superior performance on these two datasets but also outperforms the state-of-the-art methods by a significant margin.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Another line of work uses CV to recognize distinct phases of surgery during operations, towards developing context-aware computer assistance systems 126,127 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [126]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“Specifically, we exploit the very deep residual network, the ResNet [21], to extract highly discriminative visual features from the video frames.”
BackgroundMethodInfluentialGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 1×
“networks on an image and intercept the learned information of networks [37], [38].”
BackgroundCaffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · cited 1×
“Our framework is implemented with C++ and Python based on the Caffe [35] deep learning library, using a TITAN X GPU for acceleration.”
MethodShin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2×
“HMM, we propose to tap into the temporal dimension of the surgical video data using the LSTM [31], [32], which has been demonstrated as a very powerful tool to model temporal concepts.”
MethodDeep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · cited 1×
“HMM, we propose to tap into the temporal dimension of the surgical video data using the LSTM [31], [32], which has been demonstrated as a very powerful tool to model temporal concepts.”
Method
Research works printed but not linked: 2
- olah, 2015, Understanding LSTM networks, Proc GitHub Blog
- dergachyova, 0, Data-driven surgical workflow detection: Technical report for M2CAI 2016 surgical workflow challenge, M2CAI Challenge
Cited by
Deep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406
Machine Learning for Surgical Phase Recognition: A Systematic Review
Garrow, C. R., Kowalewski, K.-F., Li, L., Wagner, M., Schmidt, M. W., Engelhardt, S., Hashimoto, D. A., Kenngott, H. G., Bodenstedt, S., Speidel, S., et al. (2021) · Annals of Surgery · doi:10.1097/sla.0000000000004425 · Cited by 311
“39,42,46,48,56,57 Other studies focused on bariatric surgery and colorectal studies, 51,54 with both achieving a phase recognition accuracy over 90% The same ML model was compared between LC and colorectal surgery 1 study, achieving 74% and 67% accuracy, respectively.”
BackgroundResultTeCNO: Surgical Phase Recognition with Multi-stage Temporal Convolutional Networks
Czempiel, T., Paschali, M., Keicher, M., Simson, W., Feussner, H., Kim, S. T., & Navab, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-59716-0_33 · Cited by 223
Multi-task recurrent convolutional network with correlation loss for surgical video analysis
Jin, Y., Li, H., Dou, Q., Chen, H., Qin, J., Fu, C.-W., & Heng, P.-A. (2020) · Medical Image Analysis · doi:10.1016/j.media.2019.101572 · Cited by 204
Self-citationMachine and deep learning for workflow recognition during surgery
Padoy, N. (2019) · Minimally Invasive Therapy & Allied Technologies · doi:10.1080/13645706.2019.1584116 · Cited by 181
Kitaguchi, D., Takeshita, N., Matsuzaki, H., Takano, H., Owada, Y., Enomoto, T., Oda, T., Miura, H., Yamanashi, T., Watanabe, M., et al. (2020) · Surgical Endoscopy · doi:10.1007/s00464-019-07281-0 · Cited by 176
“Automatic surgical workflow recognition is a key component for developing context-aware computer-assisted surgery (CA-CAS) systems [1].”
BackgroundWard, T. M., Mascagni, P., Ban, Y., Rosman, G., Padoy, N., Meireles, O., & Hashimoto, D. A. (2021) · Surgery · doi:10.1016/j.surg.2020.10.039 · Cited by 160
A Survey on Multimodal Data-Driven Smart Healthcare Systems: Approaches and Applications
Cai, Q., Wang, H., Li, Z., & Liu, X. (2019) · IEEE Access · doi:10.1109/access.2019.2941419 · Cited by 164
Gao, X., Jin, Y., Long, Y., Dou, Q., & Heng, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87202-1_57 · Cited by 211
Self-citationKitaguchi, D., Takeshita, N., Matsuzaki, H., Oda, T., Watanabe, M., Mori, K., Kobayashi, E., & Ito, M. (2020) · International Journal of Surgery · doi:10.1016/j.ijsu.2020.05.015 · Cited by 138
Jin, Y., Cheng, K., Dou, Q., & Heng, P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32254-0_49 · Cited by 136
Self-citationComputer-Aided Cervical Cancer Diagnosis Using Time-Lapsed Colposcopic Images
Li, Y., Chen, J., Xue, P., Tang, C., Chang, J., Chu, C., Ma, K., Li, Q., Zheng, Y., & Qiao, Y. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2994778 · Cited by 141
“memory (LSTM) [27].”
BackgroundMethodInfluentialTemporal Memory Relation Network for Workflow Recognition From Surgical Video
Jin, Y.-M., Long, Y.-H., Chen, C., Zhao, Z., Dou, Q., & Heng, P.-A. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3069471 · Cited by 158
“For data preprocessing, we follow the previous works [6], [8] to downsample the original videos from 25fps to 1fps to enrich the temporal information for network training.”
BackgroundMethodResultInfluentialSelf-citationArtificial intelligence‐based computer vision in surgery: Recent advances and future perspectives
Kitaguchi, D., Takeshita, N., Hasegawa, H., & Ito, M. (2021) · Annals of Gastroenterological Surgery · doi:10.1002/ags3.12513 · Cited by 164
OperA: Attention-Regularized Transformers for Surgical Phase Recognition
Czempiel, T., Paschali, M., Ostler, D., Kim, S. T., Busam, B., & Navab, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87202-1_58 · Cited by 150
“ResNet-50 is the feature extraction backbone, ResLSTM [14] and MTRCNet-CL [15] utilize LSTMs for the temporal refinement.”
BackgroundMethodWagner, M., Müller-Stich, B. P., Kisilenko, A., Tran, D., Heger, P., Mündermann, L., Lubotsky, D. M., Müller, B., Davitashvili, T., Capek, M., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102770 · Cited by 171
Self-citationCheng, K., You, J., Wu, S., Chen, Z., Zhou, Z., Guan, J., Peng, B., & Wang, X. (2021) · Surgical Endoscopy · doi:10.1007/s00464-021-08619-3 · Cited by 98
Tanzi, L., Piazzolla, P., Porpiglia, F., & Vezzetti, E. (2021) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-021-02432-y · Cited by 94
Jia, X., Mai, X., Cui, Y., Yuan, Y., Xing, X., Seo, H., Xing, L., & Meng, M. Q. ‐. (2020) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2020.2964827 · Cited by 111
“Recent researches reveal that the depth of feature representation is of crucial importance [29], [30], and many leading results in recognition tasks have greatly benefited from the very deep networks [31], [32].”
BackgroundJia, X., Xing, X., Yuan, Y., Xing, L., & Meng, M. Q. ‐. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2950506 · Cited by 100
Ramesh, S., Dall’Alba, D., Gonzalez, C. A., Yu, T., Mascagni, P., Mutter, D., Marescaux, J., Fiorini, P., & Padoy, N. (2021) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-021-02388-z · Cited by 119
“MTMS-TCN is evaluated against popular surgical phase recognition networks, ResNetLSTM [14], and TeCNO [6].”
BackgroundMethodInfluentialDeep Learning in Surgical Workflow Analysis: A Review of Phase and Step Recognition
Demir, K. C., Schieber, H., Weise, T., Roth, D., May, M., Maier, A., & Yang, S. H. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3311628 · Cited by 90
Wang, Z., Lu, B., Long, Y., Zhong, F., Cheung, T., Dou, Q., & Liu, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16449-1_46 · Cited by 166
Self-citationAutomated operative phase identification in peroral endoscopic myotomy
Ward, T. M., Hashimoto, D., Ban, Y., Rattner, D., Inoue, H., Lillemoe, K., Rus, D., Rosman, G., & Meireles, O. (2020) · Surgical Endoscopy · doi:10.1007/s00464-020-07833-9 · Cited by 73
LoViT: Long Video Transformer for surgical phase recognition
Liu, Y., Boels, M., García-Peraza-Herrera, L. C., Vercauteren, T. K., Dasgupta, P., Granados, A., & Ourselin, S. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103366 · Cited by 137
“Since vision features and temporal context of surgical video are highly complex, these methods show limited representation capabilities with pre-defined dependencies [12, 13].”
BackgroundComputer Vision in the Surgical Operating Room
Chadebecq, F., Vasconcelos, F., Mazomenos, E. B., & Stoyanov, D. V. (2020) · Visceral Medicine · doi:10.1159/000511934 · Cited by 89
“Recently, AI models for activity recognition have been developed and tested on annotated datasets from real cases of robotic-assisted radical prostatectomy and ocular microsurgery [18–20].”
Method“Deep-Onto” network for surgical workflow and context recognition
Nakawala, H., Bianchi, R., Pescatori, L. E., COBELLI, O. D., Ferrigno, G., & Momi, E. D. (2018) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-018-1882-8 · Cited by 81
Dissecting self-supervised learning methods for surgical computer vision
Ramesh, S., Srivastav, V., Alapatt, D., Yu, T., Murali, A., Sestini, L., Nwoye, C. I., Hamoud, I., Sharma, S., Fleurentin, A., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102844 · Cited by 113
Artificial intelligence for phase recognition in complex laparoscopic cholecystectomy
Golany, T., Aides, A., Freedman, D., Rabani, N., Liu, Y., Rivlin, E., Corrado, G. S., Matias, Y., Khoury, W., Kashtan, H., et al. (2022) · Surgical Endoscopy · doi:10.1007/s00464-022-09405-5 · Cited by 72
“Previous studies on AI for interpreting laparoscopic videos [12, 17, 34–36] have focused on identifying procedure phases and instruments.”
Image Level Training and Prediction: Intracranial Hemorrhage Identification in 3D Non-Contrast CT
Patel, A., van de Leemput, S. C., Prokop, M., van Ginneken, B., & Manniesing, R. (2019) · IEEE Access · doi:10.1109/access.2019.2927792 · Cited by 68
Xu, L., Zhang, H., Wang, J., Li, A., Song, S., Ren, H., Qi, L., Gu, J. J., & Meng, M. Q. ‐. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.015 · Cited by 62
Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Search
Gao, X., Jin, Y., Dou, Q., & Heng, P. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196674 · Cited by 62
“Combining a deep ResNet [21] with an RNN network, recurrent convolutional network was proposed to identify the surgical phase from videos [22], [23].”
BackgroundMethodSelf-citationActive learning using deep Bayesian networks for surgical workflow analysis
Bodenstedt, S., Rivoir, D., Jenke, A., Wagner, M., Breucha, M., Müller-Stich, B., Mees, S. T., Weitz, J., & Speidel, S. (2019) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-019-01963-9 · Cited by 58
“Several methods in the state-of-the-art for video-based surgical workflow analysis utilize convolutional neural networks (CNNs) for interpreting the video stream [1,3,12,21, 24].”
MethodHard Frame Detection and Online Mapping for Surgical Phase Recognition
Yi, F., & Jiang, T. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32254-0_50 · Cited by 74
Deep Learning Based Image Processing for Robot Assisted Surgery: A Systematic Literature Survey
Hussain, S. M., Brunetti, A., Lucarelli, G. T., Memeo, R., Bevilacqua, V., & Buongiorno, D. (2022) · IEEE Access · doi:10.1109/access.2022.3223704 · Cited by 62
SKiT: a Fast Key Information Video Transformer for Online Surgical Phase Recognition
Liu, Y., Huo, J., Peng, J., Sparks, R. E., Prokar Dasgupta, A., Granados, A., & Sebastien Ourselin, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01927 · Cited by 63
Kawka, M., Gall, T., Fang, C., Liu, R., & Jiao, L. R. (2021) · Intelligent Surgery · doi:10.1016/j.isurg.2021.03.001 · Cited by 67
Gumbs, A. A., Grasso, V., Bourdel, N., Croner, R. S., Spolverato, G., Frigerio, I. M., Manríquez, A. I., Hilal, M. A., Park, A., & Elyan, E. (2022) · Sensors · doi:10.3390/s22134918 · Cited by 61
Yue, Z., Ding, S., Zhao, W., Wang, H., Ma, J., Zhang, Y., & Zhang, Y. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2922682 · Cited by 65
“The complementary information of visual and temporal features learned fromaCNNwithLSTMhas been utilized, producing additional discriminative spatio-temporal features to largely boost the surgical workflow recognition problem [26].”
BackgroundMethodTakeuchi, M., Collins, T., Ndagijimana, A., Kawakubo, H., Kitagawa, Y., Marescaux, J., Mutter, D., Perretta, S., Hostettler, A., & Dallemagne, B. (2022) · Hernia · doi:10.1007/s10029-022-02621-x · Cited by 59
“The second approach was the SV-RCNet one [20], which combined a deep residual network (ResNet) [21] CNN with a long short-term memory (LSTM) network [22].”
MethodExploring Segment-Level Semantics for Online Phase Recognition From Surgical Videos
Ding, X., & Li, X. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3182995 · Cited by 65
“[19] Y. Jin et al., “SV-RCNet: Workflow recognition from surgical videos using recurrent convolutional network,” IEEE Trans.”
BackgroundMethodInfluentialLiu, Y., Zhao, Z., Chang, F., & Hu, S. (2020) · IEEE Access · doi:10.1109/access.2020.2989807 · Cited by 57
Shinozuka, K., Turuda, S., Fujinaga, A., Nakanuma, H., Kawamura, M., Matsunobu, Y., Tanaka, Y., Kamiyama, T., Ebe, K., Endo, Y., et al. (2022) · Surgical Endoscopy · doi:10.1007/s00464-022-09160-7 · Cited by 55
“These metrics can be described by the following equations ((1)–(3), below) [12, 13, 28], where TP is true positive, FP is false positive, FN is false negative, and TN is true negative:”
BackgroundMethodInfluentialA Review on Deep Learning in Minimally Invasive Surgery
Rivas-Blanco, I., Pérez-del-Pulgar, C. J., Garcia-Morales, I., & Muñoz, V. F. (2021) · IEEE Access · doi:10.1109/access.2021.3068852 · Cited by 59
“17 Another custom framework is presented in [60].”
BackgroundPrivacy-preserving Time-series Medical Images Analysis Using a Hybrid Deep Learning Framework
Yue, Z., Ding, S., Zhao, L., Zhang, Y., Cao, Z., Tanveer, M., Jolfaei, A., & Zheng, X. (2021) · ACM Transactions on Internet Technology · doi:10.1145/3383779 · Cited by 49
“Therefore, it can be concluded that the combination of CNN and LSTM can encode discriminative spatio-temporal features towards time-series medical image sequence, which is the same as other researches reported [6, 24, 38].”
BackgroundResultA contextual detector of surgical tools in laparoscopic videos using deep learning
Namazi, B., Sankaranarayanan, G., & Devarajan, V. (2021) · Surgical Endoscopy · doi:10.1007/s00464-021-08336-x · Cited by 50
Learning multi-modal representations by watching hundreds of surgical video lectures
Yuan, K., Srivastav, V., Yu, T., Lavanchy, J. L., Marescaux, J., Mascagni, P., Navab, N., & Padoy, N. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103644 · Cited by 122
Temporal coherence-based self-supervised learning for laparoscopic workflow analysis
Funke, I., Jenke, A., Mees, S. T., Weitz, J., Speidel, S., & Bodenstedt, S. (2018) · OR 2.0/CARE/CLIP/ISIC@MICCAI · doi:10.1007/978-3-030-01201-4_11 · Cited by 51
“State-of-the-art video-based approaches for workflow analysis rely on deep neural networks [1,2,9,15,18].”
BackgroundNwoye, C. I., Yu, T., Sharma, S., Murali, A., Alapatt, D., Vardazaryan, A., Yuan, K., Hajek, J., Reiter, W., Yamlahi, A., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102888 · Cited by 71
Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources
Qin, Y., Pedram, S. A., Feyzabadi, S., Allan, M., McLeod, A. J., Burdick, J. W., & Azizian, M. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196560 · Cited by 45
“introduced the post-processing of predictions using prior knowledge inference [22].”
MethodShi, P., Zhao, Z., Hu, S., & Chang, F. (2020) · IEEE Access · doi:10.1109/access.2020.3046258 · Cited by 53
“It also has many potential applications, such as accurate tracking [4]–[6], pose estimation [7]–[9], optimization of surgical scheduling, real-time reminders of surgery, and integrated post-surgical assessment [10]–[12].”
BackgroundNovel evaluation of surgical activity recognition models using task-based efficiency metrics
Zia, A., Guo, L., Zhou, L., Essa, I. A., & Jarc, A. (2019) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-019-02025-w · Cited by 48
Graph Convolutional Nets for Tool Presence Detection in Surgical Videos
Wang, S., Xu, Z., Yan, C., & Huang, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20351-1_36 · Cited by 49
Long, Y., Wu, J. Y., Lu, B., Jin, Y., Unberath, M., Liu, Y., Heng, P., & Dou, Q. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561028 · Cited by 53
Self-citationYu, T., Mutter, D., Marescaux, J., & Padoy, N. (2018) · arXiv.org · doi:10.48550/arxiv.1812.00033 · Cited by 53
Hegde, S. R., Namazi, B., Iyengar, N., Cao, S., Desir, A. D., Marques, C., Mahnken, H., Dumas, R. P., & Sankaranarayanan, G. (2023) · Surgical Endoscopy · doi:10.1007/s00464-023-10482-3 · Cited by 39
“Inspired by their work and after the introduction of a large, annotated dataset called Cholec80, other researchers have attempted numerous techniques to improve the performance of phase detection [10–14].”
Latent Graph Representations for Critical View of Safety Assessment
Murali, A., Alapatt, D., Mascagni, P., Vardazaryan, A., Garcia, A., Okamoto, N., Mutter, D., & Padoy, N. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3333034 · Cited by 74
Generative AI & machine learning in surgical education
Hla, D. A., & Hindin, D. I. (2024) · Current Problems in Surgery · doi:10.1016/j.cpsurg.2024.101701 · Cited by 42
LRTD: long-range temporal dependency based active learning for surgical workflow recognition
Shi, X., Jin, Y., Dou, Q., & Heng, P. (2020) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-020-02198-9 · Cited by 37
“Existing successes achieved by deep learning models for workflow recognition are mostly based on fully supervised learning using frame-wise annotations [14,15, 21].”
BackgroundMethodSelf-citationCurrent applications of artificial intelligence-based computer vision in laparoscopic surgery
Guo, K., Tao, H., Zhu, Y., Li, B., Fang, C., Qian, Y., & Yang, J. (2023) · Laparoscopic Endoscopic and Robotic Surgery · doi:10.1016/j.lers.2023.07.001 · Cited by 36
Komatsu, M., Kitaguchi, D., Yura, M., Takeshita, N., Yoshida, M., Yamaguchi, M., Kondo, H., Kinoshita, T., & Ito, M. (2023) · Gastric Cancer · doi:10.1007/s10120-023-01450-w · Cited by 41
“…artificial intelligence (AI) has been actively applied in surgical research [12–15], and AI-based surgical phase recognition is an essential part of an expansion of context-aware computer-assisted surgery systems [16, 17], which could lead to improved surgical safety and efficiency [18, 19].”
MIcro-surgical anastomose workflow recognition challenge report
Huaulmé, A., Sarıkaya, D., Mut, K. L., Despinoy, F., Long, Y., Dou, Q., Chng, C.-B., Lin, W., Kondo, S., Bravo-Sánchez, L., et al. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106452 · Cited by 40
Self-citationDifferent molecular enumeration influences in deep learning: an example using aqueous solubility
Chen, J., & Tseng, Y. J. (2020) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa092 · Cited by 38
Surgical workflow recognition with 3DCNN for Sleeve Gastrectomy
Zhang, B., Ghanem, A. G., Simes, A., Choi, H., & Yoo, A. C. (2021) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-021-02473-3 · Cited by 34
Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognition
Yang, S., Luo, L., Wang, Q., & Chen, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72089-5_57 · Cited by 68
Self-citationShi, X., Jin, Y., Dou, Q., & Heng, P. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102158 · Cited by 43
Self-citationConcept Graph Neural Networks for Surgical Video Understanding
Ban, Y., Eckhoff, J. A., Ward, T. M., Hashimoto, D. A., Meireles, O. R., Rus, D. L., & Rosman, G. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3299518 · Cited by 47
“…laparoscopic cholecystectomy, as it has been frequently focused on in computer vision analysis and ML applications, due to its relatively standardized workflow [12], [15], [16], [17], [19], [20], [21], as well as conceptual notions such as CVS, as a safety measure for prevention of accidental…”
BackgroundMT4MTL-KD: A Multi-Teacher Knowledge Distillation Framework for Triplet Recognition
Gui, S., Wang, Z., Chen, J., Zhou, X., Zhang, C., & Cao, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3345736 · Cited by 42
“A CNN + RNN architecture is employed by Jin et al. to acquire complementary information on visual and temporal features in an end-to-end manner [19].”
MethodNot End-to-End: Explore Multi-Stage Architecture for Online Surgical Phase Recognition
Yi, F., Yang, Y., & Jiang, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26316-3_25 · Cited by 34
SUPR-GAN: SUrgical PRediction GAN for Event Anticipation in Laparoscopic and Robotic Surgery
Ban, Y., Rosman, G., Eckhoff, J. A., Ward, T. M., Hashimoto, D. A., Kondo, T., Iwaki, H., Meireles, O. R., & Rus, D. L. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3156856 · Cited by 37
Surgical Phase Recognition: From Public Datasets to Real-World Data
Kirtac, K., Aydın, N., Lavanchy, J. L., Beldi, G., Smit, M., Woods, M. S., & Aspart, F. (2022) · Applied Sciences · doi:10.3390/app12178746 · Cited by 28
Less Is More: Surgical Phase Recognition From Timestamp Supervision
Ding, X., Yan, X., Wang, Z., Zhao, W., Zhuang, J., Xu, X., & Li, X. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3242980 · Cited by 39
Surgical workflow recognition with temporal convolution and transformer for action segmentation
Zhang, B., Goel, B., Sarhan, M. H., Goel, V. K., Abukhalil, R., Kalesan, B., Stottler, N., & Petculescu, S. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02811-z · Cited by 37
“To achieve temporal modeling, researchers proposed using image classification networks and recurrent neural networks together to capture the spatial and temporal information from video clips accordingly [4–8].”
MethodUsing deep learning to identify the recurrent laryngeal nerve during thyroidectomy
Gong, J., Holsinger, F. C., Noel, J. E., Mitani, S., Jopling, J. K., Bedi, N., Koh, Y. W., Orloff, L. A., Cernea, C. R., & Yeung, S. Y. (2021) · Scientific Reports · doi:10.1038/s41598-021-93202-y · Cited by 33
Uncharted Waters of Machine and Deep Learning for Surgical Phase Recognition in Neurosurgery
Jumah, F., Raju, B., Nagaraj, A., Shinde, R., Lescott, C., Sun, H., Gupta, G., & Nanda, A. (2022) · World Neurosurgery · doi:10.1016/j.wneu.2022.01.020 · Cited by 28
Temporal-based Swin Transformer network for workflow recognition of surgical video
Pan, X., Gao, X., Wang, H., Zhang, W., Mu, Y., Bi, M., & He, X. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02785-y · Cited by 30
Computer vision algorithms in healthcare: Recent advancements and future challenges
Kabir, M. M., Rahman, A., Hasan, M. N., & Mridha, M. F. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109531 · Cited by 35
Sasaki, K., Ito, M., Kobayashi, S., Kitaguchi, D., Matsuzaki, H., Kudo, M., Hasegawa, H., Takeshita, N., Sugimoto, M., Mitsunaga, S., et al. (2022) · International Journal of Surgery · doi:10.1016/j.ijsu.2022.106856 · Cited by 25
Zang, C., Türkcan, M. K., Narasimhan, S., Cao, Y., Yarali, K., Xiang, Z., Szot, S. L., Ahmad, F., Choksi, S., Bitner, D. P., et al. (2023) · Bioengineering · doi:10.3390/bioengineering10060654 · Cited by 25
“The architecture features both CNN and LSTM appended with an attention layer for aggregation of long-range temporal dependencies.”
MethodInfluentialLoukas, C., Frountzas, M., & Schizas, D. (2020) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-020-02285-x · Cited by 24
Rivoir, D., Funke, I., & Speidel, S. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103126 · Cited by 31
“…multi-stage training procedures, where each component (CNN, LSTM (Hochreiter and Schmidhuber, 1997), TCN (Farha and Gall, 2019), Transformer (Vaswani et al., 2017), etc.) is trained individually (Bano et al., 2020; Gao et al., 2021; Kannan et al., 2019; Marafioti et al., 2021; Yuan et al., 2021).”
BackgroundMethodInfluentialTao, R., Zou, X., & Zheng, G. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3279838 · Cited by 34
“This is probably the reason why in several recent endoscopic vision benchmarking competitions, learning-based approaches, particularly those based on deep learning models [22], [38], [39], are on the top list of the associated leaderboard.”
BackgroundMethodInfluentialSelf-distillation for Surgical Action Recognition
Yamlahi, A., Tran, T. N., Godau, P., Schellenberg, M., Michael, D., Smidt, F.-H., Nölke, J.-H., Adler, T. J., Tizabi, M. D., Nwoye, C. I., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43996-4_61 · Cited by 24
Towards accurate surgical workflow recognition with convolutional networks and transformers
Zhang, B., Abbing, J. R., Ghanem, A. G., Fer, D., Barker, J., Abukhalil, R., Goel, V. K., & Milletarì, F. (2021) · Computer Methods in Biomechanics and Biomedical Engineering Imaging & Visualization · doi:10.1080/21681163.2021.2002191 · Cited by 23
“Recent studies try to overcome this specific problem by using both the temporal and spatial information with networks combining both a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN) instead of a single Convolutional Neural Network (Jin et al. 2017, 2020).”
MethodLiu, K., Zhao, Z., Shi, P., Li, F., & He, S. (2022) · Journal of Computational Design and Engineering · doi:10.1093/jcde/qwac049 · Cited by 24
Intraoperative surgery room management: A deep learning perspective
Tanzi, L., Piazzolla, P., & Vezzetti, E. (2020) · International Journal of Medical Robotics and Computer Assisted Surgery · doi:10.1002/rcs.2136 · Cited by 25
“20,22 Endoscopic videos are, in particular, fundamental for the applications we discussed in Section 3.1.2: phase recognition, tool recognition, segmentation and tracking.”
BackgroundInfluentialAlshirbaji, T. A., Jalal, N. A., & Möller, K. (2020) · Current Directions in Biomedical Engineering · doi:10.1515/cdbme-2020-0002 · Cited by 22
Deep learning for surgical workflow analysis: a survey of progresses, limitations, and trends
Li, Y., Zhao, Z., Li, R., & Li, F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10929-6 · Cited by 42
“…domain-specific prior knowledge, ensuring consistency, and employing innovative loss functions, etc. Calibration with predefined surgical orders (Jin et al. 2018; Zhang et al. 2021b; Pan et al. 2022) is more suitable for phase and step recognition, which are more procedural in the process.”
InfluentialHecVL: Hierarchical Video-Language Pretraining for Zero-Shot Surgical Phase Recognition
Yuan, K., Srivastav, V., Navab, N., & Padoy, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72089-5_29 · Cited by 67
Doughty, M., Singh, K. S., & Ghugre, N. R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87202-1_64 · Cited by 22
Artificial Intelligence: A Review of Progress and Prospects in Medicine and Healthcare
Mishra, S. (2022) · Journal of Electronics Electromedical Engineering and Medical Informatics · doi:10.35882/jeeemi.v4i1.1 · Cited by 33
“models are now able to identify the distinctive phases during a surgical procedure to enable development of context-aware computer assisted systems by analyzing visual and temporal features learned from surgical videos [23], [24].”
BackgroundHybrid Spatiotemporal Contrastive Representation Learning for Content-Based Surgical Video Retrieval
Kumar, V., Tripathi, V., Pant, B., Alshamrani, S. S., Dumka, A., Gehlot, A., Singh, R., Rashid, M. U., Alshehri, A., & AlGhamdi, A. S. (2022) · Electronics · doi:10.3390/electronics11091353 · Cited by 23
“…system (CB ‐ SVR) to address these tasks. retrieval, which are based on handcrafted features In contrast to previous methods for sur ‐ gical video retrieval, which are based on handcrafted features, we utilize a deep learning approach to extract discriminative features from surgical videos for…”
Cascade Multi-Level Transformer Network for Surgical Workflow Analysis
Yue, W., Liao, H., Xia, Y., Lam, V. W. T., Luo, J., & Wang, Z. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3265354 · Cited by 21
Aggregating Long-Term Context for Learning Laparoscopic and Robot-Assisted Surgical Workflows
Ban, Y., Rosman, G., Ward, T. M., Hashimoto, D., Kondo, T., Iwaki, H., Meireles, O., & Rus, D. (2020) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra48506.2021.9561770 · Cited by 24
“LSTM is a baseline of CNN-LSTM structure similar to [24].”
BackgroundMethodInfluentialGuzmán-García, C., Tomé, M. G., González, P. S., García, I. O., & Aguilera, E. J. G. (2021) · Sensors · doi:10.3390/s21041330 · Cited by 24
King, A., Fowler, G., Macefield, R. C., Walker, H., Thomas, C., Markar, S. R., Higgins, E., Blazeby, J. M., & Blencowe, N. (2025) · BJS Open · doi:10.1093/bjsopen/zraf073 · Cited by 26
Anticipation for surgical workflow through instrument interaction and recognized Signals
Yuan, K., Holden, M., Gao, S., & Lee, W. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102611 · Cited by 23
FetNet: a recurrent convolutional network for occlusion identification in fetoscopic videos
Bano, S., Vasconcelos, F., Poorten, E. V., Vercauteren, T. K., Sebastien Ourselin, A., Deprest, J. A., & Stoyanov, D. V. (2020) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-020-02169-0 · Cited by 19
“Deep learning architectures have shown promising results in temporal analysis of laparoscopic videos for tool detection and tracking, and surgical phase identification [5,14,24].”
BackgroundEndo3D: Online Workflow Analysis for Endoscopic Surgeries Based on 3D CNN and LSTM
Chen, W., Feng, J., Lu, J., & Zhou, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01201-4_12 · Cited by 21
OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding
Hu, M., Xia, P., Wang, L., Yan, S., Tang, F., Xu, Z., Luo, Y., Song, K., Leitner, J., Cheng, X., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73235-5_27 · Cited by 53
Metadata
- Authors
- Jin, Y., Dou, Q., Chen, H., Yu, L., Qin, J., Fu, C.-W., & Heng, P.-A.
- Affiliations
- Chinese University of Hong Kong (HK)
- Hong Kong Polytechnic University (HK)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2018-05
- Volume
- 37
- Issue
- 5
- Pages
- 1114-1126
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2777273430
- Cited by
- 316
- References
- 50
- Citations by year
- 2026: 32 · 2025: 55 · 2024: 37 · 2023: 45 · 2022: 45 · 2021: 50 · 2020: 30 · 2019: 17 · 2018: 4
- FWCI
- 14.22
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- PolyU Institutional Research Archive (Hong Kong Polytechnic University) (submittedVersion)
- Funders
- Research Grants Council of Hong Kong Special Administrative Region (CUHK 14202514, CUHK 14203115)
- Shenzhen Science and Technology Program (JCYJ20170413162617606)
- Hong Kong Polytechnic University (1-ZE8J)
- Topics
- Health Sciences › Medicine › Surgery › Surgical Simulation and Training
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- surgical workflow recognition, surgical video analysis, recurrent convolutional networks, computer-assisted surgery, spatio-temporal features, Cholec80 dataset, ResNet, LSTM
- MeSH terms
- Algorithms, Humans, Image Processing, Computer-Assisted, Software, Databases, Factual, Neural Networks, Computer, Video-Assisted Surgery, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.