SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network
The DOI matches its Crossref record; 48 of the 50 references it cites were found in a registry or an index.
Abstract
We propose an analysis of surgical videos that is based on a novel recurrent convolutional network (SV-RCNet), specifically for automatic workflow recognition from surgical videos online, which is a key component for developing the context-aware computer-assisted intervention systems. Different from previous methods which harness visual and temporal information separately, the proposed SV-RCNet seamlessly integrates a convolutional neural network (CNN) and a recurrent neural network (RNN) to form a novel recurrent convolutional architecture in order to take full advantages of the complementary information of visual and temporal features learned from surgical videos. We effectively train the SV-RCNet in an end-to-end manner so that the visual representations and sequential dynamics can be jointly optimized in the learning process. In order to produce more discriminative spatio-temporal features, we exploit a deep residual network (ResNet) and a long short term memory (LSTM) network, to extract visual features and temporal dependencies, respectively, and integrate them into the SV-RCNet. Moreover, based on the phase transition-sensitive predictions from the SV-RCNet, we propose a simple yet effective inference scheme, namely the prior knowledge inference (PKI), by leveraging the natural characteristic of surgical video. Such a strategy further improves the consistency of results and largely boosts the recognition performance. Extensive experiments have been conducted with the MICCAI 2016 Modeling and Monitoring of Computer Assisted Interventions Workflow Challenge dataset and Cholec80 dataset to validate SV-RCNet. Our approach not only achieves superior performance on these two datasets but also outperforms the state-of-the-art methods by a significant margin.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Another line of work uses CV to recognize distinct phases of surgery during operations, towards developing context-aware computer assistance systems 126,127 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [126]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“Specifically, we exploit the very deep residual network, the ResNet [21], to extract highly discriminative visual features from the video frames.”
BackgroundMethodInfluentialGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 1×
“networks on an image and intercept the learned information of networks [37], [38].”
BackgroundCaffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · cited 1×
“Our framework is implemented with C++ and Python based on the Caffe [35] deep learning library, using a TITAN X GPU for acceleration.”
MethodShin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2×
“HMM, we propose to tap into the temporal dimension of the surgical video data using the LSTM [31], [32], which has been demonstrated as a very powerful tool to model temporal concepts.”
MethodDeep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · cited 1×
“HMM, we propose to tap into the temporal dimension of the surgical video data using the LSTM [31], [32], which has been demonstrated as a very powerful tool to model temporal concepts.”
Method
Research works printed but not linked: 2
- olah, 2015, Understanding LSTM networks, Proc GitHub Blog
- dergachyova, 0, Data-driven surgical workflow detection: Technical report for M2CAI 2016 surgical workflow challenge, M2CAI Challenge
Cited by
Items 101–200 of 316
Surgical video workflow analysis via visual-language learning
Li, P., Shu, X., Feng, C.-M., Feng, Y., Zuo, W., & Tang, J. (2025) · npj Health Systems · doi:10.1038/s44401-024-00010-3 · Cited by 21
Xia, T., & Jia, F. (2021) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-021-02382-5 · Cited by 20
A Deep Learning Framework for Recognising Surgical Phases in Laparoscopic Videos
Jalal, N. A., Alshirbaji, T. A., Docherty, P. D., Neumuth, T., & Möeller, K. (2021) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2021.10.278 · Cited by 21
Free Lunch for Surgical Video Understanding by Distilling Self-supervisions
Ding, X., Liu, Z., & Li, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16449-1_35 · Cited by 21
Laparoscopic Video Analysis Using Temporal, Attention, and Multi-Feature Fusion Based-Approaches
Jalal, N. A., Alshirbaji, T. A., Docherty, P. D., Arabian, H., Laufer, B., Krueger‐Ziolek, S., Neumuth, T., & Möeller, K. (2023) · Sensors · doi:10.3390/s23041958 · Cited by 18
Weakly Supervised Temporal Convolutional Networks for Fine-Grained Surgical Activity Recognition
Ramesh, S., Dall’Alba, D., Gonzalez, C. A., Yu, T., Mascagni, P., Mutter, D., Marescaux, J., Fiorini, P., & Padoy, N. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3262847 · Cited by 27
Zhai, Y., Chen, Z., Zheng, Z. G., Wang, X., Yan, X., Liu, X., Yin, J., Wang, J., & Zhang, J. (2023) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-023-03027-5 · Cited by 18
“Based on this, important phase discriminant tools have been developed, including TeCNO and SV-RCNet [21, 22].”
InfluentialLarge-scale surgical workflow segmentation for laparoscopic sacrocolpopexy
Zhang, Y.-T., Bano, S., Page, A., Deprest, J., Stoyanov, D., & Vasconcelos, F. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-021-02544-5 · Cited by 20
Surgical Workflow Anticipation Using Instrument Interaction
Yuan, K., Holden, M., Gao, S., & Lee, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87202-1_59 · Cited by 18
Reducing prediction volatility in the surgical workflow recognition of endoscopic pituitary surgery
Das, A., Bano, S., Vasconcelos, F., Khan, D. Z., Marcus, H. J., & Stoyanov, D. V. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02599-y · Cited by 17
“To improve on the purely spatial recognition methods, stateof-the-art automated workflow recognition spatial-temporal methods [3,5,6,9] were also tested.”
BackgroundMethodLi, Y., Xia, T., Luo, H., He, B., & Jia, F. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3299321 · Cited by 25
Surgical Activity Triplet Recognition via Triplet Disentanglement
Chen, Y., He, S., Jin, Y., & Qin, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43996-4_43 · Cited by 17
Self-citationShi, P., Zhao, Z., Liu, K., & Li, F. (2022) · Journal of Computational Design and Engineering · doi:10.1093/jcde/qwac011 · Cited by 25
Scaling up self-supervised learning for improved surgical foundation models
Jaspers, T. J. M., de Jong, R. L. P. D., Li, Y., Kusters, C. H. J., Bakker, F. H. A., van Jaarsveld, R. C., Kuiper, G. M., van Hillegersberg, R., Ruurda, J. P. H., Brinkman, W. M., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103873 · Cited by 62
“…et al., 2023; Bakker et al., 2024), detecting critical surgical entities such as instruments and their corresponding actions (Twinanda et al., 2017; Jin et al., 2018; Nwoye et al., 2020), and evaluating procedural quality metrics, such as the critical view of safety (Mascagni et al., 2022; Murali…”
Automated assessment of simulated laparoscopic surgical skill performance using deep learning
Power, D., Burke, C., Madden, M. G., & Ullah, I. (2025) · Scientific Reports · doi:10.1038/s41598-025-96336-5 · Cited by 21
“Others reported using a CNN and LSTM architecture to identify instruments and phases of surgery, with a mean average precision of 89.1% and 87.4%, respectively 46 .”
InfluentialRelevance Detection in Cataract Surgery Videos by Spatio- Temporal Action Localization
Ghamsarian, N., Taschwer, M., Putzgruber-Adamitsch, D., Sarny, S., & Schoeffmann, K. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412525 · Cited by 18
TUNeS: A Temporal U-Net With Self-Attention for Video-Based Surgical Phase Recognition
Funke, I., Rivoir, D., Krell, S., & Speidel, S. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2025.3535228 · Cited by 19
“On the other hand, some methods [8], [24] train the CNN with temporal context by combining it with an LSTM network and training both components jointly on truncated video sequences.”
InfluentialSurgical phase recognition by learning phase transitions
Sahu, M., Szengel, A., Mukhopadhyay, A., & Zachow, S. (2020) · Current Directions in Biomedical Engineering · doi:10.1515/cdbme-2020-0037 · Cited by 17
Choi, J., Cho, S., Chung, J. W., & Kim, N. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106251 · Cited by 19
Efficient Global-Local Memory for Real-Time Instrument Segmentation of Robotic Surgical Video
Wang, J., Jin, Y., Wang, L., Cai, S., Heng, P.-A., & Qin, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87202-1_33 · Cited by 19
“Recently, many works in surgical video analysis have shown successes of using long-range temporal information for improving results, mainly focus on recognition tasks for surgical workflow, gesture and tool presence [12,25,3,13,5,27].”
BackgroundSelf-citationLaparoscopic Image-Based Critical Action Recognition and Anticipation With Explainable Features
Zhang, J., Zhou, S., Wang, Y., Shi, S., Wan, C., Zhao, H., Cai, X., & Ding, H. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3306818 · Cited by 16
“[40]createdthepriorknowledge inference(PKI)methodtorefinetheoutputsofphaserecog-nition,whicharesensitivetothephasetransition.”
Surgical Phase Recognition of Short Video Shots based on Temporal Modeling of Deep Features
Loukas, C. (2019) · International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC) · doi:10.5220/0007352000210029 · Cited by 19
Beyersdorffer, P., Kunert, W., Jansen, K. T., Miller, J., Wilhelm, P., Burgert, O., Kirschniak, A., & Rolinger, J. (2021) · Biomedizinische Technik/Biomedical Engineering · doi:10.1515/bmt-2020-0106 · Cited by 15
Retrieval of Surgical Phase Transitions Using Reinforcement Learning
Zhang, Y., Bano, S., Page, A., Deprest, J. A., Stoyanov, D. V., & Vasconcelos, F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16449-1_47 · Cited by 16
ARST: auto-regressive surgical transformer for phase recognition from laparoscopic videos
Zou, X., Liu, W., Wang, J., Tao, R., & Guoyan Zheng, A. (2022) · Computer Methods in Biomechanics and Biomedical Engineering Imaging & Visualization · doi:10.1080/21681163.2022.2145238 · Cited by 17
“2017) include: 1) our CCI strategy is part of our on-line inference procedure and thus is not a post-processing method; and 2) it does not use any surgeryspecific a priori knowledge as in PKI (Jin et al. 2017) and thus can be extended to other types of workflow analysis tasks.”
BackgroundMethodInfluentialMohamadipanah, H., Perrone, K. H., Peterson, K., Nathwani, J. N., Huang, F. C., Garren, A., Garren, M. E., Witt, A. K., & Pugh, C. M. (2020) · ACS Biomaterials Science & Engineering · doi:10.1021/acsbiomaterials.9b01019 · Cited by 15
Park, M., Oh, S., Jeong, T. T., & Yu, S. (2022) · Diagnostics · doi:10.3390/diagnostics13010107 · Cited by 14
Lee, S., Kim, G., Hwang, Y. N., Kwon, J. Y., & Kim, S. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105637 · Cited by 17
SurgPLAN: Surgical Phase Localization Network for Phase Recognition
Luo, X., Pang, Y., Chen, Z., Wu, J., Zhang, Z., Lei, Z., & Liu, H. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635466 · Cited by 20
“4, our SurgPLAN outperforms TeCNO [7], SlowFast [14], and SV-RCNet [8], and is the closest to ground truth.”
BackgroundMethodInfluentialWang, Z., Liu, C., Zhu, L., Wang, T., Zhang, S., & Dou, Q. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3532311 · Cited by 32
“SV-RCNet[20]modelsthetemporaldependencies of different phases in surgical videos, in which a Long-short term memory (LSTM) [21] network is utilized to build the temporal relationships.”
Self-citationLensID: A CNN-RNN-Based Framework Towards Lens Irregularity Detection in Cataract Surgery Videos
Ghamsarian, N., Taschwer, M., Putzgruber-Adamitsch, D., Sarny, S., El‐Shabrawi, Y., & Schoeffmann, K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87237-3_8 · Cited by 16
SF-TMN: SlowFast temporal modeling network for surgical phase recognition
Zhang, B., Sarhan, M. H., Goel, B., Petculescu, S., & Ghanem, A. (2023) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-024-03095-1 · Cited by 17
Enabling Relevance-Based Exploration of Cataract Videos
Ghamsarian, N. (2020) · International Conference on Multimedia Retrieval · doi:10.1145/3372278.3391937 · Cited by 14
“To address transition-sensitivity in predictions, another work [9] proposes the concept of Prior Knowledge Inference (PKI).”
BackgroundMethodState-of-the-art of situation recognition systems for intraoperative procedures
Junger, D., Frommer, S., & Burgert, O. (2022) · Medical and Biological Engineering and Computing · doi:10.1007/s11517-022-02520-4 · Cited by 11
Can Deep Learning Algorithms Help Identify Surgical Workflow and Techniques?
Mohamadipanah, H., Kearse, L. E., Witt, A. K., Wise, B. J., Yang, S., Goll, C. K., & Pugh, C. M. (2021) · Journal of Surgical Research · doi:10.1016/j.jss.2021.07.003 · Cited by 13
AI-Based Video Segmentation: Procedural Steps or Basic Maneuvers?
Perumalla, C. A., Kearse, L. E., Peven, M., Laufer, S., Goll, C. K., Wise, B. J., Yang, S., & Pugh, C. M. (2022) · Journal of Surgical Research · doi:10.1016/j.jss.2022.10.069 · Cited by 13
Fer, D., Zhang, B., Abukhalil, R., Goel, V., Goel, B., Barker, J., Kalesan, B., Barragan, I., Gaddis, M. L., & Kilroy, P. G. (2023) · Surgical Endoscopy · doi:10.1007/s00464-023-09870-6 · Cited by 13
“To capture the spatiotemporality in surgical video segments for surgical workflow recognition, image classification networks and recurrent neural networks together [7, 8] or 3D Convolutional Neural Network alone was used [10].”
BackgroundMethodMulti-level feature aggregation network for instrument identification of endoscopic images
Chu, Y., Yang, X., Li, H., Ai, D., Ding, Y., Fan, J., Song, H., & Yang, J. (2020) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ab8dda · Cited by 13
“…endoscopic camera motion estimation (Grasa et al 2014), surface reconstruction of scenarios (Maier-Hein et al 2013, Palomar et al 2016, Dong et al 2019), and surgical object detection and workflow analysis (Bouget et al 2017, Du et al 2018, Jin et al 2018, Fan et al 2019), as shown in figure 1.”
MethodSurgical Workflow Recognition: From Analysis of Challenges to Architectural Study
Czempiel, T., Sharghi, A., Paschali, M., Navab, N., & Mohareri, O. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25066-8_32 · Cited by 15
Instrument-tissue Interaction Quintuple Detection in Surgery Videos
Lin, W., Hu, Y. P., Hao, L., Zhou, D., Yang, M., Fu, H., Chui, C., & Liu, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16449-1_38 · Cited by 13
Future Frame Prediction for Robot-Assisted Surgery
Gao, X., Jin, Y., Zhao, Z., Dou, Q., & Heng, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78191-0_41 · Cited by 16
Self-citationFeng, X., Zhang, X., Shi, X., Li, L., & Wang, S. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103026 · Cited by 16
MuST: Multi-scale Transformers for Surgical Phase Recognition
Pérez, A., Rodríguez, S., Ayobi, N., Aparicio, N., Dessevres, E., & Arbeláez, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72089-5_40 · Cited by 12
Qin, Y., Allan, M., Burdick, J. W., & Azizian, M. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3091728 · Cited by 12
“introduced the post-processing of predictions using prior knowledge inference [31].”
MethodSpatio-Temporal Features Based Surgical Phase Classification Using CNNs
Pradeep, C. S., & Sinha, N. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/embc46164.2021.9630829 · Cited by 6
“SV-RCNet [10] is a retrospective study, where the authors trained an end to end ResNet-LSTM network which requires prior knowledge about surgery duration for developing the inference system.”
BackgroundMachine Learning in Laparoscopic Surgery
Badgery, H. E., Zhou, Y., Siderellis, A., Read, M. D., & Davey, C. E. (2022) · Springer eBooks · doi:10.1007/978-981-19-1223-8_8 · Cited by 11
Learning With Pedagogical Models: Videos As Adjuncts to Apprenticeship for Surgical Training
Ferrier-Barbut, É., Avellino, I., Canlorbe, G., Vitrani, M., & Luengo, V. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579615 · Cited by 11
“One direction is partial or full automation, as existing powerful tools can identify and segment the steps of a surgical intervention, using laparoscopic videos [41], instruments signals and annotations [21].”
BackgroundA Multimodal Transformer Model for Recognition of Images from Complex Laparoscopic Surgical Videos
Abiyev, R. H., Altobel, M. Z., Darwish, M., & Helwan, A. (2024) · Diagnostics · doi:10.3390/diagnostics14070681 · Cited by 13
“Cholec90 Dataset MTRCNet-CL [25] 89.2 EndoNet [3] 81.7 PhaseNet [26] 78.8 SV-RCNet [27] 85.3 OHFM [28] 87.3 Trans-SVNet [29] 90.3 EndoNet [3 81.7 PhaseNet 78.8 SV-RCNet 85.3 OHFM [ 87.3 Trans-SVNet 90.3 Ours 91.0”
InfluentialLive laparoscopic video retrieval with compressed uncertainty
Yu, T., Mascagni, P., Verde, J. M., Marescaux, J., Mutter, D., & Padoy, N. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102866 · Cited by 11
A Deep Architecture for Surgical Workflow Recognition with Edge Information
Qi, B., Qin, X., Liu, J., Xu, Y., & Chen, Y. (2019) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm47256.2019.8983269 · Cited by 11
EgoSurgery-Phase: A Dataset of Surgical Phase Recognition from Egocentric Open Surgery Videos
Fujii, R., Hatano, M., Saitô, H., & Kajita, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72089-5_18 · Cited by 10
Head-Mounted Displays in Context-Aware Systems for Open Surgery: A State-of-the-Art Review
Tu, M., Jung, H., Kim, J., & Kyme, A. Z. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3485023 · Cited by 12
“For example, the need for constant awareness of the surgical context, coordination among clinical staff, and efficient management of surgical workflows are significant challenges [4], [5], [6] — and inadequate management of these factors can impact surgical outcomes and patient safety [7].”
Surgical Temporal Action-aware Network with Sequence Regularization for Phase Recognition
Chen, Z., Zhai, Y., Zhang, J., & Wang, J. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385308 · Cited by 14
“…used 2D convolutional neural networks (CNNs) to extract the feature vector of each surgical video frame, and then predicted the surgical phase with the inter-frame temporal relationship aggregated by the long short-term memory (LSTM) [17], temporal convolutions [18], [22], or transformers [19].”
BackgroundMethodResultInfluentialAttention-based encoder-decoder networks for workflow recognition
Zhang, M., Hu, H., Li, Z., & Chen, J. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10633-5 · Cited by 10
Reiter, W. (2021) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-021-02411-3 · Cited by 10
“Later work found a way to enable better predictions by training the architecture end to end on short video clips of two seconds [7].”
BackgroundWang, H., Ding, S., Yang, S., Liu, C., Yu, S., & Zheng, X. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3108457 · Cited by 10
Carvalho, T. D., Kader, R., Brandão, P., Puyal, J. G., Lovat, L. B., Mountney, P., & Stoyanov, D. V. (2023) · Biomedical Optics Express · doi:10.1364/boe.485069 · Cited by 9
“Jin et al. [26] use a more advanced architecture, keeping the same base idea of neural networks but including more temporal information using a recurrent neural network (RNN).”
Instrument-Tissue-Guided Surgical Action Triplet Detection via Textual-Temporal Trail Exploration
Pei, J., Zhang, J., Qin, G., Wang, K., Jin, Y., & Heng, P. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3590457 · Cited by 27
“EndoNet [4] and SV-RCNet [33] utilize convolutional neural networks (CNNs) and long short-term memory (LSTM) networks to capture frame features and build memory mechanisms.”
Self-citationWang, C., Liang, H., Chen, H., Khan, R., Shen, D., Liu, H., Shen, D., Wang, W., Liu, J., Panthier, F., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02109-9 · Cited by 10
Self-citationSystem for Central Venous Catheterization Training Using Computer Vision-Based Workflow Feedback
Hisey, R. J., Camiré, D., Erb, J., Howes, D. W., Gabor Fichtinger, A., & Ungi, T. (2021) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2021.3124422 · Cited by 9
Wang, J., Jin, Y., Cai, S., Xu, H., Heng, P., Qin, J., & Wang, L. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102291 · Cited by 10
“Surgery video is actually a form of sequential data, and effectively capturing the sequential dynamics has shown its successes in various surgical tasks, such as workflow recognition and tool presence detection in cholecystectomy procedure (Jin et al., 2018; Czempiel et al., 2020; Jin et al., 2021) and instrument segm…”
Self-citationMohamadipanah, H., Kearse, L. E., Wise, B. J., Backhus, L. M., & Pugh, C. M. (2022) · Journal of Surgical Research · doi:10.1016/j.jss.2022.11.008 · Cited by 9
Deol, E. S., Tollefson, M. K., Antolin, A., Zohar, M., Bar, O., Ben-Ayoun, D., Mynderse, L. A., Lomas, D. J., Avant, R., Miller, A. R., et al. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1375482 · Cited by 11
“A fundamental goal for the application of artificial intelligence (AI) in surgery has been automated surgical step recognition (SSR) from intraoperative video footage (Jin et al., 2018).”
BackgroundSE-OHFM: A surgical phase recognition network with SE attention module
Li, Y., Li, Y.-Q., He, W., Shi, W.-L., Wang, T., & Li, Y. (2021) · 2021 International Conference on Electronic Information Engineering and Computer Science (EIECS) · doi:10.1109/eiecs53707.2021.9587961 · Cited by 6
Tu, P., Ye, H., Shi, H., Young, J., Xie, M., Zhao, P., Zheng, C., Jiang, X., & Chen, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102604 · Cited by 12
Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition
Zhang, B., Meng, J., Cheng, B., Biskup, D., Petculescu, S., & Chapman, A. (2024) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc53108.2024.10782887 · Cited by 8
“…other state-of-the-art approaches, we compare our ResNet MS-ASCT and EffNetV2 MS-ASCT with PhaseLSTM [2], EndoLSTM [2], MTRCNet [4], SV-RCNet [3], TeCNO [5], SlowFast UniGRU, TimeSformer UniGRU, Swin UniGRU [21], Trans-SVNet [12], PATG [26], and Spatio-Temporal Causal Trans-former(denoted as…”
InfluentialGonzález-Cebrián, Á., Bordonaba, S., Pascau, J., Paredes, I., Lagares, A., & de Toledo, P. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110222 · Cited by 7
Surgeons versus computer vision: a comparative analysis on surgical phase recognition capabilities
Mezzina, M., BACKER, P. D., Vercauteren, T. K., Blaschko, M. B., Mottrie, A., & Tuytelaars, T. (2025) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-025-03383-4 · Cited by 8
“We trained a ResNet50 [14] and a ResNet50-Long Short-Term Memory (LSTM) for single frames and video snippets End-To-End classification to automate SPR, as performed by Jin et al. [7].”
InfluentialSurgical phase and instrument recognition: how to identify appropriate dataset splits
Kostiuchik, G., Sharan, L., Mayer, B., Ivo Wolf, A., Preim, B., & Engelhardt, S. (2024) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-024-03063-9 · Cited by 8
Abbas, A. I. A., Namazi, B., Radi, I., Alterio, R. E., Abreu, A. A., Rail, B., Polanco, P. M., Zeh, H. J., Hogg, M. E., Zureikat, A. H., et al. (2024) · Surgical Endoscopy · doi:10.1007/s00464-024-10725-x · Cited by 9
“Jin et al. developed a CNN model using 107 videos to segment the procedure into 7 phases with 92% accuracy [35].”
Yoshida, M., Kitaguchi, D., Takeshita, N., Matsuzaki, H., Ishikawa, Y., Yura, M., Akimoto, T., Kinoshita, T., & Ito, M. (2024) · Langenbeck s Archives of Surgery · doi:10.1007/s00423-024-03411-y · Cited by 8
“AI-based AI-based surgical step recognition is crucial in establishing context-aware computer-aided surgical systems [ 15 , 16].”
Attention-Based Surgical Phase Boundaries Detection in Laparoscopic Videos
Namazi, B., Sankaranarayanan, G., & Devarajan, V. (2019) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci49370.2019.00109 · Cited by 8
“For instance, in [1] the visual features were extracted using a CNN, and [2] used Long Short-Term Memory (LSTM) model to utilize the temporal dependencies between the neighboring frames to assign a surgical phase to each frame.”
MethodReal-time medical phase recognition using long-term video understanding and progress gate method
Zhang, Y., Marsic, I., & Burd, R. S. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102224 · Cited by 11
Shah, N. A., Sikder, S., Vedula, S. S., & Patel, V. M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43996-4_37 · Cited by 7
Organ-DETR: Organ Detection via Transformers
Ghahremani, M., Ernhofer, B. R., Wang, J., Makowski, M. R., & Wachinger, C. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3543581 · Cited by 11
“Accurate detection of organs from 3D Computed Tomography (CT) data is instrumental in modern healthcare for early diagnosis, preventive care, treatment planning, and surgical interventions [6], [7], [8], [9].”
Multiple instance convolutional neural network for gallbladder assessment from laparoscopic images
Loukas, C., Gazis, A., & Schizas, D. (2022) · International Journal of Medical Robotics and Computer Assisted Surgery · doi:10.1002/rcs.2445 · Cited by 7
Global–local multi-stage temporal convolutional network for cataract surgery phase recognition
Fang, L., Mou, L., Gu, Y., Hu, Y., Chen, B., Chen, X., Wang, Y., Liu, J., & Zhao, Y. (2022) · BioMedical Engineering OnLine · doi:10.1186/s12938-022-01048-w · Cited by 8
“[19] proposed an end-to-end recurrent convolutional network for online cholecystectomy video recognition, realizing that visual representations and sequential dynamics can be jointly optimized in the learning process.”
BackgroundAutomatic task recognition in a flexible endoscopy benchtop trainer with semi-supervised learning
Bencteux, V., Saibro, G., Shlomovitz, E., Mascagni, P., Perretta, S., Hostettler, A., Marescaux, J., & Collins, T. (2020) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-020-02208-w · Cited by 7
Know your sensors — a modality study for surgical action classification
Bastian, L., Czempiel, T., Heiliger, C., Karcz, K. W., Eck, U., Busam, B., & Navab, N. (2022) · Computer Methods in Biomechanics and Biomedical Engineering Imaging & Visualization · doi:10.1080/21681163.2022.2152377 · Cited by 12
Namazi, B., Sankaranarayanan, G., & Devarajan, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08983 · Cited by 13
“It uses its internal memory (states) to process a sequence of inputs for time series and videos processing tasks [Jin et al., 2018b].”
MethodCapturing action triplet correlations for accurate surgical activity recognition
Zou, X., Yu, D., & Guoyan Zheng, A. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102604 · Cited by 6
Evaluation of single-stage vision models for pose estimation of surgical instruments
Burton, W. S., Myers, C. A., Rutherford, M. J., & Rullkoetter, P. J. (2023) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-023-02890-6 · Cited by 8
Loukas, C., Seimenis, I., Prevezanou, K., & Schizas, D. (2024) · International Journal of Medical Robotics and Computer Assisted Surgery · doi:10.1002/rcs.2632 · Cited by 8
“The CNN employed was ResNet101 due to its efficiency in similar tasks 17,22 .”
MethodZheng, M., Ye, M., & Rafii-Tari, H. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811982 · Cited by 6
Spatio-Temporal Causal Transformer for Multi-Grained Surgical Phase Recognition
Chen, H., Li, Z., Fu, P., Ni, Z.-L., & Bian, G. (2022) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc48229.2022.9871004 · Cited by 9
“and recurrent neural networks (RNN) [2], learning the spatiotemporal context jointly.”
BackgroundMethodInfluentialSelf-citationPradeep, C. S., & Sinha, N. (2023) · Journal of Applied Biomedicine · doi:10.1016/j.bbe.2023.10.001 · Cited by 6
Surgical Data Science: Emerging Trends and Future Pathways
Levendovics, R., Nagy, T. D., Kronreif, G., & Haidegger, T. P. (2024) · Topics in intelligent engineering and informatics · doi:10.1007/978-3-031-58257-8_5 · Cited by 5
Bai, L., Ma, B., Wang, R., Wang, G., Cui, B., Jiang, Z., Islam, M., Min, Z., Lai, J., Navab, N., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103290 · Cited by 11
Biffi, C., Roffo, G., Salvagnini, P., & Cherubini, A. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108782 · Cited by 6
Real-Time Multi-Label Upper Gastrointestinal Anatomy Recognition from Gastroscope Videos
Yu, T., Hu, H., Zhang, X., Lei, H., Liu, J., Hu, W., Duan, H., & Si, J. (2022) · Applied Sciences · doi:10.3390/app12073306 · Cited by 6
Benchmarking and enhancing surgical phase recognition models for robot-assisted esophagectomy
Li, Y., van Jaarsveld, R. C., de Jong, R., Bongers, J., Kuiper, G. M., van Hillegersberg, R., Ruurda, J. P. H., Breeuwer, M. M., & Khalil, Y. A. (2025) · SPIE Medical Imaging: Image-Guided Procedures, Robotic Interventions, and Modeling · doi:10.1117/12.3040084 · Cited by 7
“We selected four state-of-the-art surgical phase recognition models for benchmarking: SV-RCNet [13], TMRNet [14], TeCNO [7], and Trans-SVNet [15] based on their demonstrated effectiveness in the AutoLaparo [12] and Cholec80 [16] datasets, with both being widely used benchmarks in the field of…”
Guzmán-García, C., González, P. S., García, I. O., & Aguilera, E. J. G. (2022) · Bioengineering · doi:10.3390/bioengineering9120753 · Cited by 5
“9, approximately [16], [18], [20], [21].”
C-ECT: Online Surgical Phase Recognition with Cross-Enhancement Causal Transformer
Zhang, B., Fung, A., Torabi, M., Barker, J., Foley, G., Abukhalil, R., Gaddis, M. L., & Petculescu, S. (2023) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi53787.2023.10230841 · Cited by 5
Simulated outcomes for durotomy repair in minimally invasive spine surgery
Balu, A., Kugener, G., Pangal, D. J., Lee, H. W., Lasky, S., Han, J., Buchanan, I. A., Liu, J., Zada, G., & Donoho, D. A. (2024) · Scientific Data · doi:10.1038/s41597-023-02744-5 · Cited by 5
Nguyen, H., Khairnar, S., Palacios, S. G., Al-abbas, A., Antunes, F., Ribeiro, B., Hogg, M., Zureikat, A., Polanco, P. M., Zeh, H., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3573264 · Cited by 3
“However, due to the complex temporal relations among frames, these techniques exhibit limited representation capacities with predefined dependencies [21].”
Goyal, A., Mendoza, M., Munoz, A. E., Macias, C. A., Abou-Mrad, A. K., Marano, L., & Oviedo, R. J. (2025) · Artificial Intelligence Surgery · doi:10.20517/ais.2024.108 · Cited by 4
SurgLaVi: Large-scale hierarchical dataset for surgical vision-language representation learning
Perez, A., Nwoye, C. I., Kermani, R. R., Mohareri, O., & Jamal, M. A. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.103982 · Cited by 24
Tu, P., Ye, H., Young, J., Xie, M., Zheng, C., & Chen, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43990-2_64 · Cited by 4
Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models
Gudlin, M., Hegedić, M., Golec, M., & Kolar, D. (2024) · Applied Sciences · doi:10.3390/app14031185 · Cited by 4
DACAT: Dual-stream Adaptive Clip-aware Time Modeling for Robust Online Surgical Phase Recognition
Yang, K., Li, Q., & Wang, Z. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890444 · Cited by 8
“In contrast, single-task learning methods [9]–[19] focus solely on surgical phase recognition, requiring only frame-level annotations for training.”
Metadata
- Authors
- Jin, Y., Dou, Q., Chen, H., Yu, L., Qin, J., Fu, C.-W., & Heng, P.-A.
- Affiliations
- Chinese University of Hong Kong (HK)
- Hong Kong Polytechnic University (HK)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2018-05
- Volume
- 37
- Issue
- 5
- Pages
- 1114-1126
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2777273430
- Cited by
- 316
- References
- 50
- Citations by year
- 2026: 32 · 2025: 55 · 2024: 37 · 2023: 45 · 2022: 45 · 2021: 50 · 2020: 30 · 2019: 17 · 2018: 4
- FWCI
- 14.22
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- PolyU Institutional Research Archive (Hong Kong Polytechnic University) (submittedVersion)
- Funders
- Research Grants Council of Hong Kong Special Administrative Region (CUHK 14202514, CUHK 14203115)
- Shenzhen Science and Technology Program (JCYJ20170413162617606)
- Hong Kong Polytechnic University (1-ZE8J)
- Topics
- Health Sciences › Medicine › Surgery › Surgical Simulation and Training
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- surgical workflow recognition, surgical video analysis, recurrent convolutional networks, computer-assisted surgery, spatio-temporal features, Cholec80 dataset, ResNet, LSTM
- MeSH terms
- Algorithms, Humans, Image Processing, Computer-Assisted, Software, Databases, Factual, Neural Networks, Computer, Video-Assisted Surgery, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.