Exploration of the Role of ChatGPT in Teaching Communication Skills for Medical Students: A Pilot Study

Prior studies exploring LLM applications in medical education-such as Webb (2023) and Chiu et al. (2025)-have primarily reported increased learner confidence following GenAI-assisted simulations but lacked external validation through SP encounters or blinded human scoring.

“Prior studies exploring LLM applications in medical education—such as Webb (2023) and Chiu et al. (2025)—have primarily reported increased learner confidence following GenAI-assisted simulations but lacked external validation through SP encounters or blinded human scoring (Chiu et al., 2025; Webb, 2023).”

Pre-post studies have inherent limitations such as lack of a parallel control group and potential response-shift bias.

“Although pre–post studies have inherent limitations (such as lack of a parallel control group and potential response-shift bias), they are invaluable for pilot assessments of educational innovations, helping to determine whether an intervention shows promise and warrants further development or more rigorous testing (Chiu et al., 2025; Poei et al., 2022).”

Overall evidence suggests AI chatbots can provide meaningful practice that improves learners' confidence and skills, especially when used as part of a blended educational approach.

“Overall, the evidence to date—though still limited in scope—suggests that AI chatbots can provide meaningful practice that improves learners’ confidence and skills, especially when used as part of a blended educational approach (Chiu et al., 2025).”

A target sample size of 30-32 students was considered sufficient to detect large within-subject effect sizes in communication outcomes, assuming 80% statistical power and α = 0.05.

“A target sample size of 30–32 students was selected based on feasibility for a pre–post design with blinded evaluation, consistent with prior AI-based educational studies in medical training (Chiu et al., 2025; Poei et al., 2022). This number was considered sufficient to detect large within-subject effect sizes in communication outcomes, assuming 80% statistical power and α = 0.05, as supported by”

Students acknowledged that a chatbot cannot yet fully convey human emotions or nuanced non-verbal cues.

“students acknowledged the current limitations of AI simulations—for example, a chatbot cannot yet fully convey human emotions or nuanced non-verbal cues.”

In a pilot study, students' self-confidence in communicating with patients, particularly in delivering bad news, increased significantly after a ChatGPT interaction, and their trust in AI as a useful training tool improved.

“The pre–post findings were encouraging: students’ self-confidence in communicating with patients—particularly in delivering bad news—increased significantly after the ChatGPT interaction, and their trust in AI as a useful training tool also improved (Chiu et al., 2025).”