510 Artificial Intelligence Tutoring Compared with Expert Instruction in Neurosurgical Simulation Training: A Randomized Controlled Trial

Abstract

INTRODUCTION: Resection of subpial tumors adjacent to critical brain areas is one of the most high-stakes surgical procedures that lacks opportunities for safe deliberate practice. Furthermore, performance-based assessment in neurosurgical apprenticeship is inefficient and vulnerable to subjectivity. Utilizing Artificial Intelligence (AI) to classify neurosurgical psychomotor expertise, we developed the first AI-powered tutor in neurosurgical simulation training, known as the Virtual Operative Assistant (VOA), to augment technical skills acquisition training by providing intelligent feedback. METHODS: A multi-institutional randomized controlled trial compared VOA’s automated audiovisual metric-based feedback vs remote verbal debriefing with expert instruction and no-feedback controls. Medical students performed six simulated subpial brain tumor resections: five practice attempts followed with feedback and one complex realistic attempt evaluated skill retention and transfer. A deep learning model, Intelligent Continuous Expertise Monitoring System (ICEMS Expertise Score), and blinded Objective Structured Assessment of Technical Skills (OSATS) evaluated performance. Participants reported emotions before, during and after training and completed a cognitive load questionnaire following training. RESULTS: Seventy medical students from four institutions were randomly assigned to VOA (n=23), Instructor (n=24), and Control (n=23) Groups. 350 practice attempts were assessed by ICEMS, and 70 realistic attempts were evaluated by ICEMS and OSATS. During practice, VOA training resulted in a significant improvement of participants’ Expertise Scores that was on average 0.66 (95% CI 0.55-0.77) and 0.65 (95% CI 0.54-0.77) points higher than Instructor and Control Groups (p<.001). Realistic attempt’s average Expertise Score was significantly higher in the VOA Group compared to Instructor and Control Groups (mean difference 0.53; 0.49, respectively, p<.001). VOA and Instructor Group’s realistic attempt OSATS ratings were not significantly different. OSATS ratings demonstrated that VOA feedback resulted in significantly higher Respect for Tissue and Economy of Movement compared to Control while expert instruction significantly improved Instrument Handling compared to Control. There was no significant between-groups difference in cognitive load, positive-, and negative-activating emotions. CONCLUSION: VOA’s quantitative automated benchmark feedback demonstrated superior performance outcome, improved skill transfer, with equivalent OSATS ratings and similar cognitive and affective responses compared to remote expert instruction.