The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation
Abstract
OBJECTIVE Virtual reality simulators track all movements and forces of simulated instruments, generating enormous datasets which can be further analyzed with machine learning algorithms. These advancements may increase the understanding, assessment and training of psychomotor performance. Consequently, the application of machine learning techniques to evaluate performance on virtual reality simulators has led to an increase in the volume and complexity of publications which bridge the fields of computer science, medicine, and education. Although all disciplines stand to gain from research in this field, important differences in reporting exist, limiting interdisciplinary communication and knowledge transfer. Thus, our objective was to develop a checklist to provide a general framework when reporting or analyzing studies involving virtual reality surgical simulation and machine learning algorithms. By including a total score as well as clear subsections of the checklist, authors and reviewers can both easily assess the overall quality and specific deficiencies of a manuscript. DESIGN The Machine Learning to Assess Surgical Expertise (MLASE) checklist was developed to help computer science, medicine, and education researchers ensure quality when producing and reviewing virtual reality manuscripts involving machine learning to assess surgical expertise. SETTING This study was carried out at the McGill Neurosurgical Simulation and Artificial Intelligence Learning Centre. PARTICIPANTS The authors applied the checklist to 12 articles using machine learning to assess surgical expertise in virtual reality simulation, obtained through a systematic literature review. RESULTS Important differences in reporting were found between medical and computer science journals. The medical journals proved stronger in discussion quality and weaker in areas related to study design. The opposite trends were observed in computer science journals. CONCLUSIONS This checklist will aid in narrowing the knowledge divide between computer science, medicine, and education: helping facilitate the burgeoning field of machine learning assisted surgical education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A survey on feature selection methods
Chandrashekar, G., & Sahin, F. (2014) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2013.11.024 · Cited by 5,189
Proceedings of 1995 IEEE International Conference on Robotics and Automation
(1995) · Proceedings of 1995 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.1995.525251 · Cited by 2,830
Gallagher, A. G., Ritter, E. M., Champion, H., Higgins, G., Fried, M. P., Moses, G., Smith, C. D., & Satava, R. M. (2005) · Annals of Surgery · doi:10.1097/01.sla.0000151982.85062.80 · Cited by 1,051
An Introduction to Machine Learning
Kubat, M. (2015) · Springer eBooks · doi:10.1007/978-3-319-20010-1 · Cited by 1,007
Quality of reporting of experimental studies in medical education: a systematic review
Cook, D. A., Beckman, T. J., & Bordage, G. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02777.x · Cited by 237
Eye Metrics as an Objective Assessment of Surgical Skill
Richstone, L., Schwartz, M. J., Seideman, C., Cadeddu, J., Marshall, S., & Kavoussi, L. R. (2010) · Annals of Surgery · doi:10.1097/sla.0b013e3181e464fb · Cited by 196
Objective Assessment of Surgical Technical Skill and Competency in the Operating Room
Vedula, S., Ishii, M., & Hager, G. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044435 · Cited by 146
A survey on feature extraction for pattern recognition
Ding, S., Zhu, H. G., Jia, W., & Su, C. (2011) · Artificial Intelligence Review · doi:10.1007/s10462-011-9225-y · Cited by 128
Modelling and Evaluation of Surgical Performance Using Hidden Markov Models
Megali, G., Sinigaglia, S., Tonet, O., & Dario, P. (2006) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2006.881784 · Cited by 119
Providing metrics and performance feedback in a surgical simulator
Sewell, C. M., Morris, D., Blevins, N. H., Dutta, S., Agrawal, S. K., Barbagli, F., & Salisbury, J. K. (2008) · Computer Aided Surgery · doi:10.1080/10929080801957712 · Cited by 109
Intelligent dental training simulator with objective skill assessment and feedback
Rhienmora, P., Haddawy, P., Suebnukarn, S., & Dailey, M. N. (2011) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2011.04.003 · Cited by 103
Applying Educational Theory to Simulation-Based Training and Assessment in Surgery
Chauvin, S. W. (2015) · Surgical Clinics of North America · doi:10.1016/j.suc.2015.04.006 · Cited by 65
Loukas, C., & Georgiou, E. (2011) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2011.2167324 · Cited by 47
Watson, R. (2014) · Academic Medicine · doi:10.1097/acm.0000000000000316 · Cited by 43
Automatic scoring of virtual mastoidectomies using expert examples
Kerwin, T. F., Wiet, G. J., Stredney, D., & Shen, H. (2011) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-011-0566-4 · Cited by 40
Meaningful Assessment of Robotic Surgical Style using the Wisdom of Crowds
Ershad, M., Rege, R. V., & Fey, A. M. (2018) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-018-1738-2 · Cited by 30
Jog, A., Itkowitz, B., Liu, M. Q., DiMaio, S., Hager, G., Curet, M. J., & Kumar, R. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5979967 · Cited by 23
Cited by
Artificial intelligence in higher education: the state of the field
Crompton, H., & Burke, D. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00392-8 · Cited by 2,010
“AI is a tool used across subject disciplines, including language education (Liang et al., 2021), engineering education (Shukla et al., 2019), mathematics education (Hwang & Tu, 2021) and medical education (Winkler-Schwartz et al., 2019),”
MethodSapci, A. H., & Sapci, H. A. (2020) · JMIR Medical Education · doi:10.2196/19285 · Cited by 377
“The authors The authors developed a checklist to assess surgical IV 1 Artificial Intelligence in Medical Education: Best Canada WinklerSchwartz et al, 2019 [14] emphasized the need to add further elements.”
MethodWang, C., Wang, H., Li, Y., Dai, J., Gu, X., & Yu, T. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2383033 · Cited by 353
Tang, K.-Y., Chang, C.-Y., & Hwang, G.-J. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1875001 · Cited by 324
Mirchi, N., Bissonnette, V., Yilmaz, R., Ledwos, N., Winkler-Schwartz, A., & Del Maestro, R. F. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229596 · Cited by 296
Self-citation
Metadata
- Authors
- Winkler-Schwartz, A., Bissonnette, V., Mirchi, N., Ponnudurai, N., Yilmaz, R., Ledwos, N., Siyar, S., Azarnoush, H., Karlik, B., & Del Maestro, R. F.
- Venue
- Journal of Surgical Education
- Publisher
- Elsevier BV
- Published
- 2019-11
- Volume
- 76
- Issue
- 6
- Pages
- 1681-1690
- ISSN
- 1878-7452 (ISSN-L)
- 1931-7204
- OpenAlex
- W2951405082
- Cited by
- 295
- References
- 17
- FWCI
- 23.77
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.