The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation
Abstract
OBJECTIVE Virtual reality simulators track all movements and forces of simulated instruments, generating enormous datasets which can be further analyzed with machine learning algorithms. These advancements may increase the understanding, assessment and training of psychomotor performance. Consequently, the application of machine learning techniques to evaluate performance on virtual reality simulators has led to an increase in the volume and complexity of publications which bridge the fields of computer science, medicine, and education. Although all disciplines stand to gain from research in this field, important differences in reporting exist, limiting interdisciplinary communication and knowledge transfer. Thus, our objective was to develop a checklist to provide a general framework when reporting or analyzing studies involving virtual reality surgical simulation and machine learning algorithms. By including a total score as well as clear subsections of the checklist, authors and reviewers can both easily assess the overall quality and specific deficiencies of a manuscript. DESIGN The Machine Learning to Assess Surgical Expertise (MLASE) checklist was developed to help computer science, medicine, and education researchers ensure quality when producing and reviewing virtual reality manuscripts involving machine learning to assess surgical expertise. SETTING This study was carried out at the McGill Neurosurgical Simulation and Artificial Intelligence Learning Centre. PARTICIPANTS The authors applied the checklist to 12 articles using machine learning to assess surgical expertise in virtual reality simulation, obtained through a systematic literature review. RESULTS Important differences in reporting were found between medical and computer science journals. The medical journals proved stronger in discussion quality and weaker in areas related to study design. The opposite trends were observed in computer science journals. CONCLUSIONS This checklist will aid in narrowing the knowledge divide between computer science, medicine, and education: helping facilitate the burgeoning field of machine learning assisted surgical education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A survey on feature selection methods
Chandrashekar, G., & Sahin, F. (2014) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2013.11.024 · Cited by 5,189
Proceedings of 1995 IEEE International Conference on Robotics and Automation
(1995) · Proceedings of 1995 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.1995.525251 · Cited by 2,830
Gallagher, A. G., Ritter, E. M., Champion, H., Higgins, G., Fried, M. P., Moses, G., Smith, C. D., & Satava, R. M. (2005) · Annals of Surgery · doi:10.1097/01.sla.0000151982.85062.80 · Cited by 1,051
An Introduction to Machine Learning
Kubat, M. (2015) · Springer eBooks · doi:10.1007/978-3-319-20010-1 · Cited by 1,007
Quality of reporting of experimental studies in medical education: a systematic review
Cook, D. A., Beckman, T. J., & Bordage, G. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02777.x · Cited by 237
Eye Metrics as an Objective Assessment of Surgical Skill
Richstone, L., Schwartz, M. J., Seideman, C., Cadeddu, J., Marshall, S., & Kavoussi, L. R. (2010) · Annals of Surgery · doi:10.1097/sla.0b013e3181e464fb · Cited by 196
Objective Assessment of Surgical Technical Skill and Competency in the Operating Room
Vedula, S. S., Ishii, M., & Hager, G. D. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044435 · Cited by 164
A survey on feature extraction for pattern recognition
Ding, S., Zhu, H. G., Jia, W., & Su, C. (2011) · Artificial Intelligence Review · doi:10.1007/s10462-011-9225-y · Cited by 128
Modelling and Evaluation of Surgical Performance Using Hidden Markov Models
Megali, G., Sinigaglia, S., Tonet, O., & Dario, P. (2006) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2006.881784 · Cited by 119
Providing metrics and performance feedback in a surgical simulator
Sewell, C. M., Morris, D., Blevins, N. H., Dutta, S., Agrawal, S. K., Barbagli, F., & Salisbury, J. K. (2008) · Computer Aided Surgery · doi:10.1080/10929080801957712 · Cited by 109
Cited by
Artificial intelligence in higher education: the state of the field
Crompton, H., & Burke, D. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00392-8 · Cited by 2,010
“AI is a tool used across subject disciplines, including language education (Liang et al., 2021), engineering education (Shukla et al., 2019), mathematics education (Hwang & Tu, 2021) and medical education (Winkler-Schwartz et al., 2019),”
MethodSapci, A. H., & Sapci, H. A. (2020) · JMIR Medical Education · doi:10.2196/19285 · Cited by 377
“The authors The authors developed a checklist to assess surgical IV 1 Artificial Intelligence in Medical Education: Best Canada WinklerSchwartz et al, 2019 [14] emphasized the need to add further elements.”
MethodWang, C., Wang, H., Li, Y., Dai, J., Gu, X., & Yu, T. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2383033 · Cited by 353
Tang, K.-Y., Chang, C.-Y., & Hwang, G.-J. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1875001 · Cited by 324
Mirchi, N., Bissonnette, V., Yilmaz, R., Ledwos, N., Winkler-Schwartz, A., & Del Maestro, R. F. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229596 · Cited by 296
“practices guidelines for utilizing AI in surgical simulation studies based on systematic literature search.[18] The Virtual Operative Assistant outlined in this study offers objective and automated feedback for the learner based on performance metrics from virtual reality simulators, allowing for an enhanced understan…”
MethodSelf-citationFazlollahi, A. M., Bakhaidar, M., Alsayegh, A., Yilmaz, R., Winkler-Schwartz, A., Mirchi, N., Langleben, I., Ledwos, N., Sabbagh, A. J., Bajunaid, K., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.49008 · Cited by 285
Self-citationDai, C.-P., & Ke, F. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100087 · Cited by 271
Lungu, A. J., Swinkels, W., Claesen, L., Tu, P., Egger, J., & Chen, X. (2021) · Expert Review of Medical Devices · doi:10.1080/17434440.2021.1860750 · Cited by 269
Banerjee, M., Chiew, D., Patel, K. T., Johns, I., Chappell, D., Linton, N., Cole, G. D., Francis, D. P., Szram, J., Ross, J., et al. (2021) · BMC Medical Education · doi:10.1186/s12909-021-02870-x · Cited by 168
“%) of respondents in our survey were yet to regularly encounter AI systems in their training and education, it is an area of active research ranging from assisted radiology teaching [29] to virtual reality for surgical skills development [30] and automated assessment of procedural performance [31].”
BackgroundAssessing AI adoption in developing country academia: A trust and privacy-augmented UTAUT framework
Rana, M. M., Siddiqee, M. S., Sakib, M. N., & Ahamed, M. R. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e37569 · Cited by 143
Fazlollahi, A. M., Yilmaz, R., Winkler-Schwartz, A., Mirchi, N., Ledwos, N., Bakhaidar, M., Alsayegh, A., & Del Maestro, R. F. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.34658 · Cited by 85
Self-citationDemystifying AI: Current State and Future Role in Medical Education Assessment
Turner, L., Hashimoto, D. A., Vasisht, S., & Schaye, V. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005598 · Cited by 70
Shaw, K., Henning, M. A., & Webster, C. S. (2025) · Medical Science Educator · doi:10.1007/s40670-025-02373-0 · Cited by 58
The Evolving Importance of Artificial Intelligence and Radiology in Medical Trainee Education
Fischetti, C., Bhatter, P., Frisch, E., Sidhu, A., Helmy, M., Lungren, M., & Duhaime, E. (2022) · Academic Radiology · doi:10.1016/j.acra.2021.03.023 · Cited by 47
Varma, J. R., Fernando, S., Ting, B. Y., Aamir, S., & Sivaprakasam, R. (2023) · Cureus · doi:10.7759/cureus.39701 · Cited by 41
Guraya, S. Y. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05916-y · Cited by 41
Morjaria, L., Burns, L., Bracken, K., Levinson, A. J., Ngo, Q. N., Lee, M., & Sibbald, M. (2024) · International Medical Education · doi:10.3390/ime3010004 · Cited by 33
“There is an ongoing need for further study into best practices for ethical uses of such tools and concerns for training medical learners to recognize medical misinformation or AI-generated hallucinations [33,34].”
Morjaria, L., Burns, L., Bracken, K., Ngo, Q. N., Lee, M., Levinson, A. J., Smith, J., Thompson, P., & Sibbald, M. (2023) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205231204178 · Cited by 31
“Whether it be facilitating the retrieval and summary of the medical literature, analyzing trends in student responses in real-time, or enhancing simulation exercises with virtual reality, AI technology has already begun to be embedded in education.(1,2) Conversely, the exponential rate of progress of AI-powered tools…”
BackgroundBeketov, V., Lebedeva, M., & Taranova, M. (2024) · Current Psychology · doi:10.1007/s12144-023-05471-7 · Cited by 21
Surgical Artificial Intelligence in Urology
Goldenberg, M. G. (2024) · Urologic Clinics of North America · doi:10.1016/j.ucl.2023.06.003 · Cited by 12
Metadata
- Authors
- Winkler-Schwartz, A., Bissonnette, V., Mirchi, N., Ponnudurai, N., Yilmaz, R., Ledwos, N., Siyar, S., Azarnoush, H., Karlik, B., & Del Maestro, R. F.
- Venue
- Journal of Surgical Education
- Publisher
- Elsevier BV
- Published
- 2019-11
- Volume
- 76
- Issue
- 6
- Pages
- 1681-1690
- ISSN
- 1878-7452 (ISSN-L)
- 1931-7204
- OpenAlex
- W2951405082
- Cited by
- 295
- References
- 17
- FWCI
- 23.77
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.