CoChat

The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine

Open access

The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

Simulation-based training is increasingly being used for assessment and training of psychomotor skills involved in medicine. The application of artificial intelligence and machine learning technologies has provided new methodologies to utilize large amounts of data for educational purposes. A significant criticism of the use of artificial intelligence in education has been a lack of transparency in the algorithms' decision-making processes. This study aims to 1) introduce a new framework using explainable artificial intelligence for simulation-based training in surgery, and 2) validate the framework by creating the Virtual Operative Assistant, an automated educational feedback platform. Twenty-eight skilled participants (14 staff neurosurgeons, 4 fellows, 10 PGY 4-6 residents) and 22 novice participants (10 PGY 1-3 residents, 12 medical students) took part in this study. Participants performed a virtual reality subpial brain tumor resection task on the NeuroVR simulator using a simulated ultrasonic aspirator and bipolar. Metrics of performance were developed, and leave-one-out cross validation was employed to train and validate a support vector machine in Matlab. The classifier was combined with a unique educational system to build the Virtual Operative Assistant which provides users with automated feedback on their metric performance with regards to expert proficiency performance benchmarks. The Virtual Operative Assistant successfully classified skilled and novice participants using 4 metrics with an accuracy, specificity and sensitivity of 92, 82 and 100%, respectively. A 2-step feedback system was developed to provide participants with an immediate visual representation of their standing related to expert proficiency performance benchmarks. The educational system outlined establishes a basis for the potential role of integrating artificial intelligence and virtual reality simulation into surgical educational teaching. The potential of linking expertise classification, objective feedback based on proficiency benchmarks, and instructor input creates a novel educational tool by integrating these three components into a formative educational paradigm.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0229596

Cites

48 of 49 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 Framework for explainable artificial intelligence for medical simulation training. The top section provides an overview of standard machine learning methodology to obtain a predictive model. The bottom section harvests the power of the machine learning model for education. The simulation task generates raw data, which can be manipulated to create metrics of performance. Using algorithms, statistical methods, or expert opinions, metrics are selected based on their ability to differentiate between two or more groups. The selected metrics are fed to a machine learning algorithm for training and testing. A final predictive model can be selected based on predictive accuracy to build the Virtual Operative Assistant. Upon recruitment of new participants, their metrics of performance can be calculated and normalized. These are then fed to the Virtual Operative Assistant to provide a group classification (e.g. skilled or novice) as well as an individual breakdown of metric performance. The feedback reinforces positive behaviour while providing detailed information on which behaviours to improve. The Virtual Operative Assistant is an iterative program optimized for user learning.
Fig 1 Framework for explainable artificial intelligence for medical simulation training. The top section provides an overview of standard machine learning methodology to obtain a predictive model. The bottom section harvests the power of the machine learning model for education. The simulation task generates raw data, which can be manipulated to create metrics of performance. Using algorithms, statistical methods, or expert opinions, metrics are selected based on their ability to differentiate between two or more groups. The selected metrics are fed to a machine learning algorithm for training and testing. A final predictive model can be selected based on predictive accuracy to build the Virtual Operative Assistant. Upon recruitment of new participants, their metrics of performance can be calculated and normalized. These are then fed to the Virtual Operative Assistant to provide a group classification (e.g. skilled or novice) as well as an individual breakdown of metric performance. The feedback reinforces positive behaviour while providing detailed information on which behaviours to improve. The Virtual Operative Assistant is an iterative program optimized for user learning.CC BY · PubMed Central
Fig 2 Confusion matrix for the classification of skilled and novice participants by the linear support vector machine. The algorithm correctly classified all skilled participants while correctly classifying 18 of the 22 novice participants (82%).
Fig 2 Confusion matrix for the classification of skilled and novice participants by the linear support vector machine. The algorithm correctly classified all skilled participants while correctly classifying 18 of the 22 novice participants (82%).CC BY · PubMed Central
Fig 3 Educational paradigm of the Virtual Operative Assistant. (A) The trainee performs a simulated subpial tumor resection scenario on the NeuroVR (CAE Healthcare, Montreal, Quebec, Canada) platform using a simulated ultrasonic aspirator in the trainee’s dominant hand and a simulated bipolar in the non-dominant hand. (B) The scenario involves removal of a cortical tumor (yellow) with minimal damage to healthy brain regions (white). (C) Upon completion of the simulated task, the data is automatically saved and uploaded to the Virtual Operative Assistant software to provide instant feedback on two monitors.
Fig 3 Educational paradigm of the Virtual Operative Assistant. (A) The trainee performs a simulated subpial tumor resection scenario on the NeuroVR (CAE Healthcare, Montreal, Quebec, Canada) platform using a simulated ultrasonic aspirator in the trainee’s dominant hand and a simulated bipolar in the non-dominant hand. (B) The scenario involves removal of a cortical tumor (yellow) with minimal damage to healthy brain regions (white). (C) Upon completion of the simulated task, the data is automatically saved and uploaded to the Virtual Operative Assistant software to provide instant feedback on two monitors.CC BY · PubMed Central
Fig 4 Performance assessment with the Virtual Operative Assistant. User received feedback on performance on two monitors. (A) The first screen informs the trainee of their classification along with a corresponding percentage for each class. (B) The second screen provides individualized feedback on 2 safety metrics. Metrics are accompanied with a positive statement (if competency achieved) or instructions to improve (if failed to achieve competency). Overall written and auditory feedback is provided on the second monitor. If the trainee has not achieved competency in all safety metrics (top row), the trainee cannot proceed further and must redo the scenario. If the trainee has achieved competency in all safety metrics (bottom row), the trainee can move on to step 2. (C) The third screen provides individualized feedback on 2 instrument motion metrics, accompanied by statements on their performance.
Fig 4 Performance assessment with the Virtual Operative Assistant. User received feedback on performance on two monitors. (A) The first screen informs the trainee of their classification along with a corresponding percentage for each class. (B) The second screen provides individualized feedback on 2 safety metrics. Metrics are accompanied with a positive statement (if competency achieved) or instructions to improve (if failed to achieve competency). Overall written and auditory feedback is provided on the second monitor. If the trainee has not achieved competency in all safety metrics (top row), the trainee cannot proceed further and must redo the scenario. If the trainee has achieved competency in all safety metrics (bottom row), the trainee can move on to step 2. (C) The third screen provides individualized feedback on 2 instrument motion metrics, accompanied by statements on their performance.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 49 references

Research works printed but not linked: 1

  1. Norman D. The design of everyday things: Revised and expanded edition: Constellation; 2013.

Cited by

See all 327

Metadata

Authors
Mirchi, N., Bissonnette, V., Yilmaz, R., Ledwos, N., Winkler-Schwartz, A., & Del Maestro, R. F.
Affiliations
Montreal Neurological Institute and Hospital (CA)
Corresponding author
Mirchi, Nykan
Venue
PLOS ONE
Published
2020-02-27
Volume
15
Issue
2
Pages
e0229596
ISSN
1932-6203 (ISSN-L)
OpenAlex
W3007803772
Cited by
296
References
48
Citations by year
2026: 39 · 2025: 63 · 2024: 67 · 2023: 39 · 2022: 47 · 2021: 29 · 2020: 12
FWCI
29.02
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Funders
Di Giovanni Foundation
Topics
  • Health Sciences › Medicine › Surgery › Surgical Simulation and Training
  • Social Sciences › Education › Simulation-Based Education in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
explainable AI, surgical simulation training, virtual surgery, neurosurgical training, brain tumor resection, support vector machine, formative assessment
MeSH terms
Simulation Training, Virtual Reality, Algorithms, Artificial Intelligence, Brain Neoplasms, Computer Simulation, Education, Medical, Female, Humans, Male, Neurosurgery, Surgery, Computer-Assisted, Support Vector Machine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine. https://app.cochat.ai/papers/10.1371/journal.pone.0229596
The paper
Mirchi, N., Bissonnette, V., Yilmaz, R., Ledwos, N., Winkler-Schwartz, A., & Del Maestro, R. F. (2020). The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine. PLOS ONE. https://doi.org/10.1371/journal.pone.0229596

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.