CoChat

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index; 12 of the 22 sources its claims cite were checked for support.

Development and Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation

Open access

Abstract

Background Objective Structured Clinical Examination (OSCE; Clinical Performance Examination [CPX] in South Korea) is a high-stakes assessment of clinical performance, communication, and reasoning during time-limited patient encounters. As AI-enabled virtual standardized patient (VSP) simulation and automated scoring are introduced for OSCE-like training, prospective evidence is needed on how such systems perform and are perceived when embedded in real educational workflows. Methods We developed CPX with Medical students’ Assistant for Training and Evaluation (CPX-MATE), a web-based platform integrating (1) CPX with Virtual Standardized Patient (CPX-VSP), real-time voice dialogue with a VSP using speech-to-speech (STS) models, and (2) CPX with Real-Time Evaluator (CPX-RTE), automated transcription, checklist-based scoring, and feedback from encounter audio using a Speech-to-Text model and a large language model. During an emergency medicine clerkship (Nov 2025-Jan 2026), 60 senior medical students completed two 12-min CPX encounters (VSP with acute pancreatitis; HSP with ureteral stone) with immediate CPX-RTE feedback. For CPX-VSP, students were assigned to either a full-capacity or a resource-limited STS configuration (n=30 each). Dialogue fidelity was evaluated by turn-by-turn analysis of student–VSP exchanges, classifying responses into clinically meaningful error types (tangential, oversharing, role-breaking, off-script). CPX-RTE performance was assessed by agreement (Gwet’s AC1) with professor real-time and resident video-based ratings using a 45-item checklist. Usability of CPX-VSP and CPX-RTE, with overall system usability scale (SUS), were surveyed, and mean per-session costs for CPX-VSP and CPX-RTE were calculated. Results Across 3,282 dialogue turns, overall error rates were 1.77% versus 9.43% for full-capacity versus resource-limited STS configurations (p<0.001), driven by fewer tangential and oversharing responses; no off-script errors were observed. The mean per-session cost was $0.12 for resource-limited configuration and $0.78 for full-capacity configuration. CPX-RTE showed high agreement with human ratings (AC1=0.916 vs professor; 0.916 vs resident), with slightly different levels of agreement across four sections, and high usability across all domains (mean scores, 4.65–4.92), with a per-session cost of $0.17. CPX-MATE demonstrated good overall usability (median [IQR] of 77.5 [70.0-85.0]). Conclusions Embedded within a prospective clinical clerkship, CPX-MATE demonstrated operational fidelity and human-level checklist agreement as an end-to-end, voice-based AI-assisted OSCE platform. This real-world deployment supports its scalable integration as a complementary assessment tool while highlighting the importance of systematic validation and context-aware implementation in medical education.

Overview

CoChat Summary

TL;DR

CPX-MATE, a web-based platform combining voice-based virtual standardized patient simulation (CPX-VSP) with automated real-time checklist scoring and feedback (CPX-RTE), was deployed prospectively during an emergency medicine clerkship with 60 senior medical students. The system showed low dialogue error rates with a full-capacity speech-to-speech model, human-level checklist agreement (Gwet's AC1 = 0.916), and good usability at low per-session cost.

Key findings

  • Across 3,282 dialogue turns (Minimum Interaction Units), overall error rates were 1.77% (95% CI 0.80–2.84) for the full-capacity STS model versus 9.43% (7.50–11.25) for the resource-limited model (absolute difference 7.65 pp, 95% CI 5.34–9.74; p<0.001), driven by fewer tangential (1.48% vs 5.28%) and oversharing (0.24% vs 4.02%) responses; no off-script errors were observed.
  • CPX-RTE checklist agreement was 0.916 (95% CI 0.902–0.930) versus the attending professor and 0.916 (0.903–0.929) versus the resident, versus 0.976 (0.968–0.983) between the two human raters; agreement was highest for History Taking (0.957/0.958) and lowest for Patient-Physician Interaction (0.860/0.847), with the item "Explore patient's deep concern" at AC1 = 0.250 for both AI–human pairs.
  • CPX-RTE feedback quality was rated 4.82 (95% CI 4.65–4.99) out of 6 overall (items 4.65–4.92), and overall system usability (SUS) had a median of 77.5 (IQR 70.0–85.0). No participants reported perceived bias or stereotyped content.
  • Mean per-session cost was $0.12 for the resource-limited CPX-VSP configuration and $0.78 for the full-capacity configuration; CPX-RTE cost $0.17 per session.

Methods

  • Prospective deployment during an emergency medicine clerkship at Yonsei University College of Medicine (Nov 2025–Jan 2026): 60 year 3–4 medical students each completed a 12-min CPX-VSP encounter (acute pancreatitis; assigned to resource-limited gpt-realtime-mini, n=30, or full-capacity gpt-realtime, n=30) followed by a 12-min HSP encounter (ureteral stone) with immediate CPX-RTE feedback.
  • CPX-VSP performance was assessed by manual turn-by-turn coding of all MIUs into four error types (tangential, oversharing, role-breaking, off-script); CPX-RTE used whisper-1 for transcription and GPT-5 for checklist mapping and narrative feedback, with agreement assessed by Gwet's AC1 against a professor's real-time and a resident's video-based ratings on a 45-item checklist (History Taking 20, Physical Examination 8, Patient Education 6, Patient-Physician Interaction 11).
  • Usability was surveyed on 1–6 Likert scales adapted from Cook et al. plus the SUS; free-text responses were analyzed with inductive thematic analysis; participant-level cluster bootstrap resampling (1,000 iterations) with Benjamini-Hochberg adjustment was used for CPX-VSP comparisons.

Limitations

  • Stated by the authors: single institution and single chief complaint, limiting generalizability; single-session exposure raising possible novelty effects on usability/immersion ratings; no examination of whether CPX-MATE improves clinical performance; and non-random assignment to model exposure conditions, which may introduce selection or order effects.
  • Evident from the text: CPX-RTE outputs for VSP encounters were not validated because no independent human scoring was performed for those encounters; all MIU error coding was done by a single medical student investigator; and physical examination scoring relied on verbalized intent rather than whether the examination was performed correctly, so interpersonal and hands-on competencies remain underdetermined by transcript-based automation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.64898/2026.02.21.26346803

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
From the PDF
Chief Complaint Selection
Chief Complaint SelectionFrom the PDF
Image 3
From the PDF
Image 4
From the PDF

Verification

Claims the paper makes

5 Valid · 1 Not valid · 2 Not checked · 82 Own results and interpretations

35 more claims

Create a free account to see all 90 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 28
  • It may state two separate claims as one. 36
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 17 references

Research works printed but not linked: 2

  1. 2025, AI-Standardized Clinical Examination training on OSCE performance, NEJM AI, 2, AIoa2500066
  2. Geathers, J. et al. in International Conference on Artificial Intelligence in Education. 231-245 (Springer).

Metadata

Authors
Song, J. W., Kim, M., Hong, C., Kim, Y. S., Cho, J., Kim, J. H., Myung, J., Choi, A., Yoon, H., Lee, S. G. W., You, S. C., & Park, C.
Affiliations
  • Yonsei University (KR)
  • Department of Emergency Medicine, Severance Hospital, Yonsei University College of Medicine, Seoul, Republic of Korea
Corresponding author
  • You, Seng Chan
  • Park, Chaeryoung
Venue
medRxiv
Publisher
openRxiv
Published
2026-02-25
OpenAlex
W7131450152
Cited by
0
References
17
Type
posted content
Language
en
Open access
Locations
medRxiv (acceptedVersion, cc-by-nc)
Topics
  • Social Sciences › Education › Innovations in Medical Education
  • Social Sciences › Education › Simulation-Based Education in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
speech-to-text transcription, clinical performance assessment, OSCE, emergency medicine clerkship, usability, medical students, acute pancreatitis, ureteral stones

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Development and Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation. https://app.cochat.ai/papers/10.64898/2026.02.21.26346803
The paper
Song, J. W., Kim, M., Hong, C., Kim, Y. S., Cho, J., Kim, J. H., Myung, J., Choi, A., Yoon, H., Lee, S. G. W., You, S. C., & Park, C. (2026). Development and Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation. medRxiv. https://doi.org/10.64898/2026.02.21.26346803

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.