Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Importance: Despite advances in the assessment of technical skills in surgery, a clear understanding of the composites of technical expertise is lacking. Surgical simulation allows for the quantitation of psychomotor skills, generating data sets that can be analyzed using machine learning algorithms. Objective: To identify surgical and operative factors selected by a machine learning algorithm to accurately classify participants by level of expertise in a virtual reality surgical procedure. Design, Setting, and Participants: Fifty participants from a single university were recruited between March 1, 2015, and May 31, 2016, to participate in a case series study at McGill University Neurosurgical Simulation and Artificial Intelligence Learning Centre. Data were collected at a single time point and no follow-up data were collected. Individuals were classified a priori as expert (neurosurgery staff), seniors (neurosurgical fellows and senior residents), juniors (neurosurgical junior residents), and medical students, all of whom participated in 250 simulated tumor resections. Exposures: All individuals participated in a virtual reality neurosurgical tumor resection scenario. Each scenario was repeated 5 times. Main Outcomes and Measures: Through an iterative process, performance metrics associated with instrument movement and force, resection of tissues, and bleeding generated from the raw simulator data output were selected by K-nearest neighbor, naive Bayes, discriminant analysis, and support vector machine algorithms to most accurately determine group membership. Results: A total of 50 individuals (9 women and 41 men; mean [SD] age, 33.6 [9.5] years; 14 neurosurgeons, 4 fellows, 10 senior residents, 10 junior residents, and 12 medical students) participated. Neurosurgeons were in practice between 1 and 25 years, with 9 (64%) involving a predominantly cranial practice. The K-nearest neighbor algorithm had an accuracy of 90% (45 of 50), the naive Bayes algorithm had an accuracy of 84% (42 of 50), the discriminant analysis algorithm had an accuracy of 78% (39 of 50), and the support vector machine algorithm had an accuracy of 76% (38 of 50). The K-nearest neighbor algorithm used 6 performance metrics to classify participants, the naive Bayes algorithm used 9 performance metrics, the discriminant analysis algorithm used 8 performance metrics, and the support vector machine algorithm used 8 performance metrics. Two neurosurgeons, 1 fellow or senior resident, 1 junior resident, and 1 medical student were misclassified. Conclusions and Relevance: In a virtual reality neurosurgical tumor resection study, a machine learning algorithm successfully classified participants into 4 levels of expertise with 90% accuracy. These findings suggest that algorithms may be capable of classifying surgical expertise with greater granularity and precision than has been previously demonstrated in surgery.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Barry Issenberg, S., Mcgaghie, W. C., Petrusa, E. R., Lee Gordon, D., & Scalese, R. J. (2005) · Medical Teacher · doi:10.1080/01421590500046924 · Cited by 3,966
Deo, R. C. (2015) · Circulation · doi:10.1161/circulationaha.115.001593 · Cited by 3,466
Assessment in medical education
Epstein, R. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmra054784 · Cited by 1,872
Intelligent dental training simulator with objective skill assessment and feedback
Rhienmora, P., Haddawy, P., Suebnukarn, S., & Dailey, M. N. (2011) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2011.04.003 · Cited by 103
Cited by
Ahmed, Z., Mohamed, K., Zeeshan, S., & Dong, X. (2020) · Database · doi:10.1093/database/baaa010 · Cited by 1,035
Madani, A., Namazi, B., Altieri, M. S., Hashimoto, D. A., Rivera, A. M., Pucher, P. H., Navarrete-Welton, A., Sankaranarayanan, G., Brunt, L. M., Okrainec, A., et al. (2022) · Annals of Surgery · doi:10.1097/sla.0000000000004594 · Cited by 391
Mirchi, N., Bissonnette, V., Yilmaz, R., Ledwos, N., Winkler-Schwartz, A., & Del Maestro, R. F. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229596 · Cited by 296
Self-citationFazlollahi, A. M., Bakhaidar, M., Alsayegh, A., Yilmaz, R., Winkler-Schwartz, A., Mirchi, N., Langleben, I., Ledwos, N., Sabbagh, A. J., Bajunaid, K., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.49008 · Cited by 285
Self-citationLungu, A. J., Swinkels, W., Claesen, L., Tu, P., Egger, J., & Chen, X. (2021) · Expert Review of Medical Devices · doi:10.1080/17434440.2021.1860750 · Cited by 269
Feigerlova, E., Hani, H., & Hothersall-Davies, E. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06719-5 · Cited by 166
“A large amount of information can be used to evaluate the performance and progress of students using AI-based learning analytics tools [4–6].”
Artificial intelligence surgery: how do we get to autonomous actions in surgery?
Gumbs, A. A., Frigerio, I., Spolverato, G., Croner, R., Illanes, A., Chouillard, E., & Elyan, E. (2021) · Sensors · doi:10.3390/s21165526 · Cited by 161
“Either by using surgical video clips or by applying virtual reality scenarios, deep learning models were trained to estimate performance level and specific surgical skills [14,15].”
MethodArtificial Intelligence and Surgical Education: A Systematic Scoping Review of Interventions
Kirubarajan, A., Young, D., Khan, S., Crasto, N., Sobel, M., & Sussman, D. (2022) · Journal of Surgical Education · doi:10.1016/j.jsurg.2021.09.012 · Cited by 101
Fazlollahi, A. M., Yilmaz, R., Winkler-Schwartz, A., Mirchi, N., Ledwos, N., Bakhaidar, M., Alsayegh, A., & Del Maestro, R. F. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.34658 · Cited by 85
Self-citationShaw, K., Henning, M. A., & Webster, C. S. (2025) · Medical Science Educator · doi:10.1007/s40670-025-02373-0 · Cited by 58
“Furthermore, AI readily and accurately assesses performance on VR simulators [19, 42, 47].”
Artificial Intelligence in Health Professions Education
Shankar, P. R. (2022) · Archives of Medicine and Health Sciences · doi:10.4103/amhs.amhs_234_22 · Cited by 40
Ali, M., Harbieh, I., & Haider, K. H. (2026) · Medical Teacher · doi:10.1080/0142159x.2025.2519639 · Cited by 19
Surgical Artificial Intelligence in Urology
Goldenberg, M. G. (2024) · Urologic Clinics of North America · doi:10.1016/j.ucl.2023.06.003 · Cited by 12
Feasibility and acceptability of incorporating artificial intelligence into simulation education
Swan, B. A., Febres-Cordero, S., Steiger, L., Lisenby, A., Getz, T., Hudson, J., Cole, K., Branch, R., McDermott, C., Fugate, K., et al. (2025) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2025.101739 · Cited by 5
Metadata
- Authors
- Winkler-Schwartz, A., Yilmaz, R., Mirchi, N., Bissonnette, V., Ledwos, N., Siyar, S., Azarnoush, H., Karlik, B., & Del Maestro, R.
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2019-08-02
- Volume
- 2
- Issue
- 8
- Pages
- e198363
- ISSN
- 2574-3805 (ISSN-L)
- OpenAlex
- W2966447025
- Cited by
- 168
- References
- 5
- FWCI
- 18.80
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.