CoChat

Enhancing Medical Interview Skills Through AI-Simulated Patient Interactions: Nonrandomized Controlled Trial

Open access

Published version of a preprint

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; 13 of the 23 sources its claims cite were checked for support.

Abstract

A nonrandomized controlled trial with fourth-year medical students showed that engaging with AI-simulated patients using LLMs and automated feedback resulted in significantly higher scores on the pre-CC OSCE medical interview compared with the control group (mean 28.1 vs 27.1, P=.01).

Overview

CoChat Summary

TL;DR

In a nonrandomized controlled trial at Okayama University, fourth-year medical students who practiced medical interviews with GPT-4 Turbo AI-simulated patients plus feedback scored significantly higher on pre-CC OSCE medical interview scores than historical controls. The authors conclude the AI training was safe and educationally effective but limited for nonverbal communication and should be used as a supplementary tool to traditional simulation education.

Key findings

  • The AI group (n=35) scored higher than the control group (n=110) on pre-CC OSCE medical interviews: total score mean 28.1, SD 1.6 vs 27.1, SD 2.2; P=.01; summary evaluation 4.8, SD 0.7 vs 4.5, SD 0.7; P=.007. The calculated effect size was 0.48.
  • SBT-QA10 scores were relatively high across items (median 4–5) with no significant pre-to-post changes. A trend of inverse correlation appeared between SBT-QA10 and pre-CC OSCE scores; the item “I felt part of the team” was significant pre (β –1.8, SE 0.77; P=.047; R²=0.41) and post (β –3.2, SE 0.54; P<.001; R²=0.81), while total SBT-QA10 scores showed negative but nonsignificant coefficients (–2.0 to –2.1).
  • Postcase questionnaires showed median reproducibility of the AI-simulated patient ranging from 7 to 9 and emotional interaction from 7 to 8. For the first case instance (n=32), 75% (n=24) rated difficulty as “appropriate,” 19% (n=6) “difficult,” 3% (n=2) each “easy” and “very easy,” and 0% “very difficult.” No significant safety concerns were observed.
  • The authors report an MCID reference value of 26.9; the control group’s average was approximately equal to this, while the intervention group significantly exceeded it.

Methods

  • Nonrandomized controlled trial with fourth-year medical students in Japan. The intervention group comprised 35 students in 2023 who consented and completed at least one AI-simulated patient practice; the control group comprised 110 students from 2022 who received only traditional education. Practice period was 1 month, followed by the pre-CC OSCE.
  • The AI platform used GPT-4 Turbo integrated with “miibo” and LINE, with text-based chat and possible voice input. Prompts covered basic structure, case information, and feedback; emotional parameters were set from 1 to 10 for 8 emotions. Cases were based on 8 primary symptoms.
  • Analysis used Prism 9; Mann-Whitney U, Fisher exact, Student t, Wilcoxon, and multiple regression. Primary outcome was pre-CC OSCE medical interview scores (summary and total); secondary outcomes included SBT-QA10 and postcase questionnaires. Cohen d was calculated for effect size.

Limitations

  • Stated in the paper: the study was nonrandomized and voluntary, used different academic years, had a limited sample size, and only some consenting students actually participated. Assessment was indirect and short-term via pre-CC OSCE rather than real clinical settings; the text-based platform restricts nonverbal communication; LLM fabrications and opaque technology require caution and monitoring; SBT-QA10 is based on Western cultures and may need different interpretation; subgroup findings may be influenced by extreme values and lost significance when the comparison group changed.
  • Evident from the text: the historical control group from 2022 may introduce cohort or secular confounding despite similar GPA, age, and sex; only 35 of 87 consenting students participated in the AI group; the SBT-QA10 was partially modified rather than used directly; and the pre-CC OSCE passing score was not disclosed.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/58753

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Research overview diagram. Pre-CC OSCE: Pre-Clinical Clerkship Objective Structured Clinical Examination.
Figure 1 Research overview diagram. Pre-CC OSCE: Pre-Clinical Clerkship Objective Structured Clinical Examination.CC BY · PubMed Central
Figure 2 Pre-CC OSCE and LLMs-educational interventions. (A) Summary evaluation (maximum of 6 points, minimum of 1 point on a scale of 1-6). (B) Total score evaluation (maximum 31 points, minimum 0 points graded). Data were analyzed using the Mann-Whitney U test. LLM: large language model; pre-CC OSCE: Pre-Clinical Clerkship Objective Structured Clinical Examination. *P<.05, **P<.001.
Figure 2 Pre-CC OSCE and LLMs-educational interventions. (A) Summary evaluation (maximum of 6 points, minimum of 1 point on a scale of 1-6). (B) Total score evaluation (maximum 31 points, minimum 0 points graded). Data were analyzed using the Mann-Whitney U test. LLM: large language model; pre-CC OSCE: Pre-Clinical Clerkship Objective Structured Clinical Examination. *P<.05, **P<.001.CC BY · PubMed Central
Figure 3 Questionnaire to be taken at the end of each case. (A) Artificial intelligence–simulated patient reproducibility is rated on a scale of 1–10, with 10 indicating “Very High Accuracy” and 1 indicating “no reproduction.” (B) Interaction through emotions is rated on a scale from 1 to 10, where 10 signifies “Very Effective” and 1 signifies “Not Effective at All.” (C) Anxiety and nervousness are rated on a scale from 1 to 10, with 10 indicating “Not Felt at All” and 1 meaning “Felt Very Strongly.” (D) The difficulty of the case is rated on a scale from 1 to 5, where 1 represents “Very Easy” and 5 represents “Very Difficult.”.
Figure 3 Questionnaire to be taken at the end of each case. (A) Artificial intelligence–simulated patient reproducibility is rated on a scale of 1–10, with 10 indicating “Very High Accuracy” and 1 indicating “no reproduction.” (B) Interaction through emotions is rated on a scale from 1 to 10, where 10 signifies “Very Effective” and 1 signifies “Not Effective at All.” (C) Anxiety and nervousness are rated on a scale from 1 to 10, with 10 indicating “Not Felt at All” and 1 meaning “Felt Very Strongly.” (D) The difficulty of the case is rated on a scale from 1 to 5, where 1 represents “Very Easy” and 5 represents “Very Difficult.”.CC BY · PubMed Central
Figure 4 Multiple regression analysis of pre-CC OSCE and SBT-QA10. Multiple regression analysis of the “I felt part of the team (medical stuff team, including faculty)” item in the SBT-QA10 questionnaire. (A) Pre. (B) Post. Pre-CC OSCE: Pre-Clinical Clerkship Objective Structured Clinical Examination; SBT-QA10: Simulation-Based Training Quality Assurance Tool.
Figure 4 Multiple regression analysis of pre-CC OSCE and SBT-QA10. Multiple regression analysis of the “I felt part of the team (medical stuff team, including faculty)” item in the SBT-QA10 questionnaire. (A) Pre. (B) Post. Pre-CC OSCE: Pre-Clinical Clerkship Objective Structured Clinical Examination; SBT-QA10: Simulation-Based Training Quality Assurance Tool.CC BY · PubMed Central

Verification

Claims the paper makes

10 Valid · 2 Not valid · 8 Not checked · 53 Own results and interpretations

31 more claims · 1 with their cited sources checked

Create a free account to see all 73 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 24
  • Probably not a substantial claim. 13
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 29 references

Research works printed but not linked: 3

  1. Abe, K, 2007, Med Educ (Japan), 38, 301
  2. Hinkle, DE, 2003, Applied Statistics for the Behavioral Sciences. 5th ed
  3. Cohen, J, 1988, Statistical Power Analysis for the Behavioral Sciences. 2nd Edition

Cited by

See all 92

Metadata

Authors
Yamamoto, A., Koda, M., Ogawa, H., Miyoshi, T., Maeda, Y., Otsuka, F., & Ino, H.
Affiliations
  • Okayama University Hospital (JP)
  • Okayama University (JP)
Corresponding author
Yamamoto, Akira
Venue
JMIR Medical Education
Published
2024-09-23
Volume
10
Pages
e58753
ISSN
2369-3762 (ISSN-L)
OpenAlex
W4401610634
Cited by
92
References
24
Citations by year
2026: 61 · 2025: 30 · 2024: 1
FWCI
8.56
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Education (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Okayama University Scientific Achievement Repository (Okayama University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Institutional Repositories DataBase (IRDB) (submittedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Education › Simulation-Based Education in Healthcare
  • Social Sciences › Education › Innovations in Medical Education
Keywords
medical interviewing skills, large language models, simulation-based training, medical students, OSCE, feedback, nonverbal communication, medical education
MeSH terms
Simulation Training, Artificial Intelligence, Clinical Competence, Education, Medical, Undergraduate, Educational Measurement, Female, Humans, Interviews as Topic, Japan, Male, Students, Medical, Patient Simulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.