The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index; 12 of the 22 sources its claims cite were checked for support.
Development and Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation
Open accessAbstract
Background Objective Structured Clinical Examination (OSCE; Clinical Performance Examination [CPX] in South Korea) is a high-stakes assessment of clinical performance, communication, and reasoning during time-limited patient encounters. As AI-enabled virtual standardized patient (VSP) simulation and automated scoring are introduced for OSCE-like training, prospective evidence is needed on how such systems perform and are perceived when embedded in real educational workflows. Methods We developed CPX with Medical students’ Assistant for Training and Evaluation (CPX-MATE), a web-based platform integrating (1) CPX with Virtual Standardized Patient (CPX-VSP), real-time voice dialogue with a VSP using speech-to-speech (STS) models, and (2) CPX with Real-Time Evaluator (CPX-RTE), automated transcription, checklist-based scoring, and feedback from encounter audio using a Speech-to-Text model and a large language model. During an emergency medicine clerkship (Nov 2025-Jan 2026), 60 senior medical students completed two 12-min CPX encounters (VSP with acute pancreatitis; HSP with ureteral stone) with immediate CPX-RTE feedback. For CPX-VSP, students were assigned to either a full-capacity or a resource-limited STS configuration (n=30 each). Dialogue fidelity was evaluated by turn-by-turn analysis of student–VSP exchanges, classifying responses into clinically meaningful error types (tangential, oversharing, role-breaking, off-script). CPX-RTE performance was assessed by agreement (Gwet’s AC1) with professor real-time and resident video-based ratings using a 45-item checklist. Usability of CPX-VSP and CPX-RTE, with overall system usability scale (SUS), were surveyed, and mean per-session costs for CPX-VSP and CPX-RTE were calculated. Results Across 3,282 dialogue turns, overall error rates were 1.77% versus 9.43% for full-capacity versus resource-limited STS configurations (p<0.001), driven by fewer tangential and oversharing responses; no off-script errors were observed. The mean per-session cost was $0.12 for resource-limited configuration and $0.78 for full-capacity configuration. CPX-RTE showed high agreement with human ratings (AC1=0.916 vs professor; 0.916 vs resident), with slightly different levels of agreement across four sections, and high usability across all domains (mean scores, 4.65–4.92), with a per-session cost of $0.17. CPX-MATE demonstrated good overall usability (median [IQR] of 77.5 [70.0-85.0]). Conclusions Embedded within a prospective clinical clerkship, CPX-MATE demonstrated operational fidelity and human-level checklist agreement as an end-to-end, voice-based AI-assisted OSCE platform. This real-world deployment supports its scalable integration as a complementary assessment tool while highlighting the importance of systematic validation and context-aware implementation in medical education.
Overview
CoChat Summary
TL;DR
CPX-MATE, a web-based platform combining voice-based virtual standardized patient simulation (CPX-VSP) with automated real-time checklist scoring and feedback (CPX-RTE), was deployed prospectively during an emergency medicine clerkship with 60 senior medical students. The system showed low dialogue error rates with a full-capacity speech-to-speech model, human-level checklist agreement (Gwet's AC1 = 0.916), and good usability at low per-session cost.
Key findings
- Across 3,282 dialogue turns (Minimum Interaction Units), overall error rates were 1.77% (95% CI 0.80–2.84) for the full-capacity STS model versus 9.43% (7.50–11.25) for the resource-limited model (absolute difference 7.65 pp, 95% CI 5.34–9.74; p<0.001), driven by fewer tangential (1.48% vs 5.28%) and oversharing (0.24% vs 4.02%) responses; no off-script errors were observed.
- CPX-RTE checklist agreement was 0.916 (95% CI 0.902–0.930) versus the attending professor and 0.916 (0.903–0.929) versus the resident, versus 0.976 (0.968–0.983) between the two human raters; agreement was highest for History Taking (0.957/0.958) and lowest for Patient-Physician Interaction (0.860/0.847), with the item "Explore patient's deep concern" at AC1 = 0.250 for both AI–human pairs.
- CPX-RTE feedback quality was rated 4.82 (95% CI 4.65–4.99) out of 6 overall (items 4.65–4.92), and overall system usability (SUS) had a median of 77.5 (IQR 70.0–85.0). No participants reported perceived bias or stereotyped content.
- Mean per-session cost was $0.12 for the resource-limited CPX-VSP configuration and $0.78 for the full-capacity configuration; CPX-RTE cost $0.17 per session.
Methods
- Prospective deployment during an emergency medicine clerkship at Yonsei University College of Medicine (Nov 2025–Jan 2026): 60 year 3–4 medical students each completed a 12-min CPX-VSP encounter (acute pancreatitis; assigned to resource-limited gpt-realtime-mini, n=30, or full-capacity gpt-realtime, n=30) followed by a 12-min HSP encounter (ureteral stone) with immediate CPX-RTE feedback.
- CPX-VSP performance was assessed by manual turn-by-turn coding of all MIUs into four error types (tangential, oversharing, role-breaking, off-script); CPX-RTE used whisper-1 for transcription and GPT-5 for checklist mapping and narrative feedback, with agreement assessed by Gwet's AC1 against a professor's real-time and a resident's video-based ratings on a 45-item checklist (History Taking 20, Physical Examination 8, Patient Education 6, Patient-Physician Interaction 11).
- Usability was surveyed on 1–6 Likert scales adapted from Cook et al. plus the SUS; free-text responses were analyzed with inductive thematic analysis; participant-level cluster bootstrap resampling (1,000 iterations) with Benjamini-Hochberg adjustment was used for CPX-VSP comparisons.
Limitations
- Stated by the authors: single institution and single chief complaint, limiting generalizability; single-session exposure raising possible novelty effects on usability/immersion ratings; no examination of whether CPX-MATE improves clinical performance; and non-random assignment to model exposure conditions, which may introduce selection or order effects.
- Evident from the text: CPX-RTE outputs for VSP encounters were not validated because no independent human scoring was performed for those encounters; all MIU error coding was done by a single medical student investigator; and physical examination scoring relied on verbalized intent rather than whether the examination was performed correctly, so interpersonal and hands-on competencies remain underdetermined by transcript-based automation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 1 Not valid · 2 Not checked · 82 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“However, learners were required to review and correct transcribed text before eliciting a spoken response, introducing an explicit verification step that interrupts conversational flow.”
What the sources say
- Disputedcites [10] 2025, A large language model digital patient system enhances ophthalmology history taking skills, NPJ Digital Medicine sourceExperimental — accuracy not yet measured
The system-development passage describes inquiry audio being converted to text by automatic speech recognition and then matched to the patient knowledge base; it does not place a learner verification step between them. The “User instructions for the LLMDP system” passage describes continuous voice dialogue and immediate text and voice feedback after submission, contrary to the claimed required review-and-correction step. Page numbers are unavailable in our copy. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although automated scoring modules have been developed, most were designed for retrospective evaluation rather than for real-time, formative feedback during ongoing practice.”
What the sources say
- Not matchedcites [17] Geathers, J. et al. in International Conference on Artificial Intelligence in Education. 231-245 (Springer).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [18] Shakur, A. H. , et al. Large Language Models for Medical OSCE Assessment: A Novel Approach to Transcript Analysis. arXiv preprint arXiv:2410.12858 (2024). sourceExperimental — accuracy not yet measured
Prior Work (p. 2) does document that automated OSCE scoring modules exist and that they operate on recorded encounters after the fact — LMM grading of the physical-exam component from extracted frames [11], a deep-learning pipeline scoring ear exams with CLIP/Detectron2 [12], and the single transcript-based communication-skills study [15] on 121 manually transcribed OSCE videos — so the 'automated scoring modules have been developed / retrospective' half of the claim is grounded. What our copy does not state is the design-intent contrast the claim turns on: nowhere does the paper say that MOST of these modules were designed for retrospective evaluation rather than for real-time, formative feedback during ongoing practice, and it never uses the formative-vs-summative framing. Notably, the timeliness critique in the Introduction (p. 2) — 'delays feedback delivery to students by weeks' and 'the time lag between the examination and receiving feedback diminishes the potential for timely improvement' — is directed at the traditional human-expert review of recordings, not at the automated scoring modules; the paper's own conclusion (p. 13) frames prior automated work only as the first LLM transcript-based assessment at scale, without characterising its purpose as retrospective. (LLM, conf=0.68)
- Partly supportedcites [19] Holcomb, M. J. et al. Zero-Shot Multimodal Question Answering for Assessment of Medical Student OSCE Physical Exam Videos. medRxiv, 2024.2006. 2005.24308467 (2024). sourceExperimental — accuracy not yet measured
The Prior Work—Simulation Assessment section (page not provided) says most success in automated OSCE grading has been with post-encounter notes, supporting a retrospective emphasis. It does not establish that most automated scoring modules were designed for retrospective use rather than real-time, formative feedback during practice. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Such challenges are particularly pronounced in low- and middle-income countries (LMICs), where constrained educational budgets, faculty shortages, and limited access to trained SPs hinder the sustainable delivery of OSCE-based education.”
What the sources say
- Partly supportedcites [5] 2024, Assessment approaches in undergraduate health professions education: towards the development of feasible assessment approaches for low-resource settings, BMC Medical Education sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Supportedcites [6] Ataro, G. , Worku, S. & Asaminew, T . Experience and challenges of objective structured clinical examination (OSCE): perspective of students and examiners in a clinical department of Ethiopian University. Ethiopian Journal of Health Sciences 30 (2020). source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“None of the participants reported perceived bias or stereotyped content in CPX-VSP dialogue and/or CPX-RTE feedback.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Emotional expression and 'reactions' (adding small extra remarks) were perceived as human-like behavioral cues.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
35 more claims
Create a free account to see all 90 claims and every source verdict.
- Probably not a substantial claim. 28
- It may state two separate claims as one. 36
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Statistical analysis
“Participant-level cluster bootstrap resampling (1,000 iterations) was used to obtain 95% CIs and p-values.”
§ Supplementary Table S1. Comparison of subjective evaluation scores for CPX-VSP models (resource-limited vs full-capacity)
“P values were calculated using two-sided Welch's t-tests, with multiple comparisons adjusted using the Benjamini-Hochberg false discovery rate procedure.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Participation Information
“All responses are anonymous , and you may discontinue participation at any time.”
§ Section 7. Participant Information
“- Student ID number (e.g., 2019191001)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Section 2. CPX-VSP (Training Module) Experience
“- Five items assessing dialogue authenticity , and”
§ Dialogue Authenticity
“1. The virtual patient's responses felt natural and human-like. (Humanlike) 2. The virtual patient's responses were consistent and contextually appropriate throughout the dialogue. (Coherent) 3. The virtual patient's responses were clinically meaningful and helpful. (Clinically Relevant) 4. The overall dialogue felt similar to a real patient-physician conversation. (Overall)”
Citations
Cites
The System Usability Scale: Past, Present, and Future
Lewis, J. R. (2018) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1455307 · Cited by 1,958
Using thematic analysis in qualitative research
Ahmed, S. K., Mohammed, R. A., Nashwan, A. J., Ibrahim, R. H., Abdalla, A. Q., M. Ameen, B. M., & Khdhir, R. M. (2025) · Journal of Medicine, Surgery, and Public Health · doi:10.1016/j.glmedi.2025.100198 · Cited by 769
Is the OSCE a feasible tool to assess competencies in undergraduate medical education?
Patrício, M. F., Julião, M., Fareleira, F., & Carneiro, A. V. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.774330 · Cited by 309
Holderried, F., Stegemann-Philipps, C., Herrmann-Werner, A., Festl-Wietek, T., Holderried, M., Eickhoff, C., & Mahling, M. (2024) · JMIR Medical Education · doi:10.2196/59213 · Cited by 152
Cook, D. A., Overgaard, J., Pankratz, V. S., Del Fiol, G., & Aakre, C. A. (2025) · Journal of Medical Internet Research · doi:10.2196/68486 · Cited by 96
Yamamoto, A., Koda, M., Ogawa, H., Miyoshi, T., Maeda, Y., Otsuka, F., & Ino, H. (2024) · JMIR Medical Education · doi:10.2196/58753 · Cited by 92
Objective structured clinical examination for teaching and assessment: Evidence-based critique
Dewan, P., Khalil, S., & Gupta, P. (2024) · Clinical Epidemiology and Global Health · doi:10.1016/j.cegh.2023.101477 · Cited by 39
A large language model digital patient system enhances ophthalmology history taking skills
Luo, M.-J., Bi, S., Pang, J., Liu, L., Tsui, C.-K., Lai, Y., Chen, W., Yang, Y., Xu, K., Zhao, L., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01841-6 · Cited by 39
Ataro, G., Worku, S., & Asaminew, T. (2020) · Ethiopian Journal of Health Sciences · doi:10.4314/ejhs.v30i3.13 · Cited by 37
Rethinking artificial intelligence in medicine: from tools to agents
Lee, Y. C. (2025) · Clinical and Experimental Emergency Medicine · doi:10.15441/ceem.25.125 · Cited by 28
Research works printed but not linked: 2
- 2025, AI-Standardized Clinical Examination training on OSCE performance, NEJM AI, 2, AIoa2500066
- Geathers, J. et al. in International Conference on Artificial Intelligence in Education. 231-245 (Springer).
Metadata
- Authors
- Song, J. W., Kim, M., Hong, C., Kim, Y. S., Cho, J., Kim, J. H., Myung, J., Choi, A., Yoon, H., Lee, S. G. W., You, S. C., & Park, C.
- Affiliations
- Yonsei University (KR)
- Department of Emergency Medicine, Severance Hospital, Yonsei University College of Medicine, Seoul, Republic of Korea
- Corresponding author
- You, Seng Chan
- Park, Chaeryoung
- ORCID
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2026-02-25
- OpenAlex
- W7131450152
- Cited by
- 0
- References
- 17
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- medRxiv (acceptedVersion, cc-by-nc)
- Topics
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- speech-to-text transcription, clinical performance assessment, OSCE, emergency medicine clerkship, usability, medical students, acute pancreatitis, ureteral stones
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Development and Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation. https://app.cochat.ai/papers/10.64898/2026.02.21.26346803
- The paper
- Song, J. W., Kim, M., Hong, C., Kim, Y. S., Cho, J., Kim, J. H., Myung, J., Choi, A., Yoon, H., Lee, S. G. W., You, S. C., & Park, C. (2026). Development and Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation. medRxiv. https://doi.org/10.64898/2026.02.21.26346803
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.



