Learning laparoscopic video shot classification for gynecological surgery
Open accessAbstract
Videos of endoscopic surgery are used for education of medical experts, analysis in medical research, and documentation for everyday clinical life. Hand-crafted image descriptors lack the capabilities of a semantic classification of surgical actions and video shots of anatomical structures. In this work, we investigate how well single-frame convolutional neural networks (CNN) for semantic shot classification in gynecologic surgery work. Together with medical experts, we manually annotate hours of raw endoscopic gynecologic surgery videos showing endometriosis treatment and myoma resection of over 100 patients. The cleaned ground truth dataset comprises 9 h of annotated video material (from 111 different recordings). We use the well-known CNN architectures AlexNet and GoogLeNet and train these architectures for both, surgical actions and anatomy, from scratch. Furthermore, we extract high-level features from AlexNet with weights from a pre-trained model from the Caffe model zoo and feed them to an SVM classifier. Our evaluation shows that we reach an average recall of .697 and .515 for classification of anatomical structures and surgical actions respectively using off-the-shelf CNN features. Using GoogLeNet, we achieve a mean recall of .782 and .617 for classification of anatomical structures and surgical actions respectively. With AlexNet the achieved recall is .615 for anatomical structures and .469 for surgical action classification respectively. The main conclusion of our work is that advances in general image classification methods transfer to the domain of endoscopic surgery videos in gynecology. This is relevant as this domain is different from natural images, e.g. it is distinguished by smoke, reflections, or a limited amount of colors.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A group at Alpen-Adria Universität Klagenfurt uses surgery videos as training material to identify a specific intervention made by the surgeon; the algorithm recognizes the likelihood of the intervention and the specific region in the body.
“An example of this is a group at the Institute of Information Technology at the Alpen-Adria Universität Klagenfurt that uses surgery videos as training material in order to identify a specific intervention made by the surgeon. For example, when an act of dissection or cutting is performed on the patient’s tissues or organs, the algorithm recognizes the likelihood of the intervention as well as the”
From The rise of artificial intelligence in healthcare applications (2020) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · see citing works
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759 · see citing works
Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535302 · Cited by 3,277 · see citing works
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299101 · Cited by 2,121 · see citing works
Lung pattern classification for interstitial lung diseases using a deep convolutional neural network
Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., & Mougiakakou, S. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535865 · Cited by 1,343 · see citing works
EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos
Twinanda, A. P., Shehata, S., Mutter, D., Marescaux, J., de Mathelin, M., & Padoy, N. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2593957 · Cited by 1,116 · see citing works
Medical image classification with convolutional neural network
Li, Q., Cai, W., Wang, X., Zhou, Y., Feng, D. D., & Chen, M. (2014) · 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV) · doi:10.1109/icarcv.2014.7064414 · Cited by 880 · see citing works
AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., & Navab, N. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528120 · Cited by 606 · see citing works
OpenCV: Open Source Computer Vision Library
Uranishi, Y. (2018) · The Journal of The Institute of Image Information and Television Engineers · doi:10.3169/itej.72.736 · Cited by 382 · see citing works
An Automatic Learning-Based Framework for Robust Nucleus Segmentation
Xing, F., Xie, Y., & Yang, L. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2481436 · Cited by 353 · see citing works
Multi-Instance Deep Learning: Discover Discriminative Local Anatomies for Bodypart Recognition
Yan, Z., Zhan, Y., Peng, Z., Liao, S., Shinagawa, Y., Zhang, S., Metaxas, D., & Zhou, X. S. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2524985 · Cited by 223 · see citing works
Content-based processing and analysis of endoscopic images and videos: A survey
Münzer, B., Schoeffmann, K., & Böszörményi, L. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-016-4219-z · Cited by 140 · see citing works
Scene classification with semantic Fisher vectors
Dixit, M., Chen, S., Gao, D., Rasiwasia, N., & Vasconcelos, N. M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298916 · Cited by 133 · see citing works
Real-Time Segmentation and Recognition of Surgical Tasks in Cataract Surgery Videos
Quellec, G., Lamard, M., Cochener, B., & Cazuguel, G. (2014) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2340473 · Cited by 72 · see citing works
Samala, R. K., Chan, H., Hadjiiski, L. M., et al. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2217092 · Cited by 71 · see citing works
Qiu, Y., Wang, Y., Yan, S., Tan, M., Cheng, S. S. M., Liu, H., & Zheng, B. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2216275 · Cited by 68 · see citing works
Colonoscopic polyp detection using convolutional neural networks
Park, S. Y., & Sargent, D. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2217148 · Cited by 59 · see citing works
Keyframe extraction in endoscopic video
Schoeffmann, K., Fabro, M. D., Szkaliczki, T., Böszörményi, L., & Keckstein, J. (2014) · Multimedia Tools and Applications · doi:10.1007/s11042-014-2224-7 · Cited by 47 · see citing works
Hasimoto, S. (2000) · Cited by 40 · see citing works
Relevance Segmentation of Laparoscopic Videos
Münzer, B., Schoeffmann, K., & Böszörményi, L. (2013) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2013.22 · Cited by 39 · see citing works
Temporal segmentation of laparoscopic videos into surgical phases
Primus, M. J., Schoeffmann, K., & Böszörményi, L. (2016) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi.2016.7500249 · Cited by 29 · see citing works
Investigation of the Impact of Compression on the Perceptional Quality of Laparoscopic Videos
Münzer, B., Schoeffmann, K., Böszörményi, L., et al. (2014) · IEEE International Symposium on Computer-Based Medical Systems (CBMS) · doi:10.1109/cbms.2014.58 · Cited by 22 · see citing works
Instrument classification in laparoscopic videos
Primus, M. J., Schoeffmann, K., & Böszörményi, L. (2015) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi.2015.7153616 · Cited by 20 · see citing works
Beecks, C., Schoeffmann, K., Lux, M., et al. (2015) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2015.21 · Cited by 19 · see citing works
Deep Learning for Shot Classification in Gynecologic Surgery Videos
Petscharnig, S., & Schoeffmann, K. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-51811-4_57 · Cited by 19 · see citing works
Self-citationShot boundary detection in endoscopic surgery videos using a variational Bayesian framework
Loukas, C., Nikiteas, N. I., Schizas, D., et al. (2016) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-016-1431-2 · Cited by 17 · see citing works
Transfer Learning for Colonic Polyp Classification Using Off-the-Shelf CNN Features
Ribeiro, E. F., Uhl, A., Wimmer, G., & Häfner, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-54057-3_1 · Cited by 16 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Stefan Petscharnig started his studies of computer science in 2010 at AAU Klagenfurt. He received his bachelor's degree in 2014 and his master's degree in 2015 from AAU Klagenfurt. In his master's thesis he investigated the impact of asynchronism on the Quality of Experience in social TV like scenarios using methods from Crowdsourcing, Human Computation, and Gamification. He was occupied as student research assisstant and after graduation as project assistant for the AdvUHD-DASH project in 2015. Since 2016, he works in the KISMET research project with a focus on the analysis of endoscopic video data using machine learning.
- Dr. Klaus Schöffmann is an Associate Professor in the distributed multimedia systems research group at the Institute of Information Technology (ITEC) at Klagenfurt University, Austria. He received his Ph.D. in 2009 and his Habilitation (venia docendi) in 2015, both in Computer Science and from Klagenfurt Uni- versity. His research focuses on human-computer interaction with multimedia data (e.g., video browsing), multimedia content analysis, and multimedia systems (particularly in the mobile and medical domain). He has co-authored more than 80 publications on various topics in multimedia and he has co-organized interna- tional conferences, special sessions and workshops (e.g., MMM 2012, CBMI 2013, VisHMC 2014, MMC 2014, MMC 2015, and MMC 2016). He is co-founder of the Video Browser Showdown (VBS), an editorial board member of the Springer International Journal on Multimedia Tools and Applications (MTAP), Springer International Journal on Multimedia Systems, and a steering committee member of the International Confer- ence on MultiMedia Modelling (MMM). Additionally, he is member of the IEEE and the ACM and a regular reviewer for international conferences and journals in the field of multimedia. Prof. Schoeffmann teaches various courses in computer science, including mobile app development (interactive multimedia applica- tions), video retrieval, media technology, distributed systems, distributed multimedia systems, and operating systems.
Cited by
Sorted by citations
The rise of artificial intelligence in healthcare applications
Bohr, A., & Memarzadeh, K. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00002-2 · Cited by 1,636 · see citing works
- Authors
- Petscharnig, S., & Schöffmann, K.
- Affiliations
- University of Klagenfurt (AT)
- Corresponding author
- Petscharnig, Stefan
- ORCID
- Petscharnig, Stefan
- Venue
- Multimedia Tools and Applications
- Published
- 2017-04-22
- Volume
- 77
- Issue
- 7
- Pages
- 8061-8079
- ISSN
- 1380-7501 (ISSN-L)
- 1573-7721
- OpenAlex
- W2606308270
- Cited by
- 182
- References
- 32
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 3 · 2023: 7 · 2022: 9 · 2021: 28 · 2020: 34 · 2019: 55 · 2018: 31 · 2017: 6
- FWCI
- 35.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Multimedia Tools and Applications (publishedVersion, cc-by)
- Funders
- University of Klagenfurt
- Topics
- Health Sciences › Medicine › Surgery › Surgical Simulation and Training
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- laparoscopic video, gynecologic surgery, endometriosis treatment, medical image annotation, transfer learning, AlexNet, GoogLeNet, support vector machine classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
