A vision transformer for decoding surgeon activity from surgical videos
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
The intraoperative activity of a surgeon has substantial impact on postoperative outcomes. However, for most surgical procedures, the details of intraoperative surgical actions, which can vary widely, are not well understood. Here we report a machine learning system leveraging a vision transformer and supervised contrastive learning for the decoding of elements of intraoperative surgical activity from videos commonly collected during robotic surgeries. The system accurately identified surgical steps, actions performed by the surgeon, the quality of these actions and the relative contribution of individual video frames to the decoding of the actions. Through extensive testing on data from three different hospitals located in two different continents, we show that the system generalizes across videos, surgeons, hospitals and surgical procedures, and that it can provide information on surgical gestures and skills from unannotated videos. Decoding intraoperative activity via accurate machine learning systems could be used to provide surgeons with feedback on their operating skills, and may allow for the identification of optimal surgical behaviour and for the study of relationships between intraoperative factors and postoperative outcomes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1×
“For more details, please refer to the original paper 50 .”
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“This is akin to curriculum learning 20 , whereby an AI system is presented with increasingly difficult tasks during the learning process in order to improve its overall performance.”
Yang (2021) · doi:10.1038/s41598-020-79139-8 · Cited by 5,035
(2020) · doi:10.1007/978-3-030-58536-5_24 · Cited by 2,824
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 1×
“One of the ultimate, yet ambitious, goals of SAIS is to decode surgeon activity from an entire surgical video without annotations and with minimal human oversight.”
Surgical skill and complication rates after bariatric surgery
Birkmeyer, J. D., Finks, J. F., O'Reilly, A., Oerline, M., Carlin, A. M., Nunn, A. R., Dimick, J., Banerjee, M., & Birkmeyer, N. J. O. (2013) · New England Journal of Medicine · doi:10.1056/nejmsa1300625 · Cited by 1,504 · cited 2×
“Each stitch can also be deconstructed into the three recurring subphases of (1) needle handling, where the needle is held in preparation for the stitch, (2) needle driving, where the needle is driven through tissue (such as the urethra), and (3) needle withdrawal, where the needle is withdrawn from tissue to complete…”
Artificial intelligence in surgery: promises and perils
Hashimoto, D. A., Rosman, G., Rus, D., & Meireles, O. R. (2018) · Annals of Surgery · doi:10.1097/sla.0000000000002693 · Cited by 1,468 · cited 1×
“Only in the past decade or so has it been empirically demonstrated that intraoperative surgical activity can have a direct influence on postoperative patient outcomes.”
Guha Roy, A., Ren, J., Azizi, S., Loh, A., Natarajan, V., Mustafa, B., Pawlowski, N., Freyberg, J., Liu, Y., Beaver, Z., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102274 · Cited by 77
Cited by
Bias in AI-based models for medical applications: challenges and mitigation strategies
Mittermaier, M., Raza, M. M., & Kvedar, J. C. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00858-z · Cited by 372
Artificial intelligence in surgery
Varghese, C., Harrison, E. M., O’Grady, G., & Topol, E. J. (2024) · Nature Medicine · doi:10.1038/s41591-024-02970-3 · Cited by 287
Clinical applications of artificial intelligence in robotic surgery
Knudsen, J. E., Ghaffar, U., Ma, R., & Hung, A. J. (2024) · Journal of Robotic Surgery · doi:10.1007/s11701-024-01867-0 · Cited by 285
Self-citationDemystifying AI: Current State and Future Role in Medical Education Assessment
Turner, L., Hashimoto, D. A., Vasisht, S., & Schaye, V. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005598 · Cited by 70
“39 Furthermore, AI analysis of intraoperative patient factors may allow for adjustment of assessments to protect against patient-speci fi c variables that may confound the validity of an automated assessment.”
BackgroundKiyasseh, D., Laca, J., Haque, T. F., Miles, B. J., Wagner, C., Donoho, D. A., Anandkumar, A., & Hung, A. J. (2023) · Communications Medicine · doi:10.1038/s43856-023-00263-3 · Cited by 69
Self-citation
Metadata
- Authors
- Kiyasseh, D., Ma, R., Haque, T. F., Miles, B. J., Wagner, C., Donoho, D. A., Anandkumar, A., & Hung, A. J.
- Venue
- Nature Biomedical Engineering
- Published
- 2023-03-30
- Volume
- 7
- Issue
- 6
- Pages
- 780-796
- ISSN
- 2157-846X (ISSN-L)
- OpenAlex
- W4362458937
- Cited by
- 147
- References
- 12
- FWCI
- 35.06
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A vision transformer for decoding surgeon activity from surgical videos. https://app.cochat.ai/papers/10.1038/s41551-023-01010-8
- The paper
- Kiyasseh, D., Ma, R., Haque, T. F., Miles, B. J., Wagner, C., Donoho, D. A., Anandkumar, A., & Hung, A. J. (2023). A vision transformer for decoding surgeon activity from surgical videos. Nature Biomedical Engineering. https://doi.org/10.1038/s41551-023-01010-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.