Large Language Models for Medical OSCE Assessment: A Novel Approach to Transcript Analysis
Open accessAbstract
Grading Objective Structured Clinical Examinations (OSCEs) is a time-consuming and expensive process, traditionally requiring extensive manual effort from human experts. In this study, we explore the potential of Large Language Models (LLMs) to assess skills related to medical student communication. We analyzed 2,027 video-recorded OSCE examinations from the University of Texas Southwestern Medical Center (UTSW), spanning four years (2019-2022), and several different medical cases or "stations." Specifically, our focus was on evaluating students' ability to summarize patients' medical history: we targeted the rubric item 'did the student summarize the patients' medical history?' from the communication skills rubric. After transcribing speech audio captured by OSCE videos using Whisper-v3, we studied the performance of various LLM-based approaches for grading students on this summarization task based on their examination transcripts. Using various frontier-level open-source and proprietary LLMs, we evaluated different techniques such as zero-shot chain-of-thought prompting, retrieval augmented generation, and multi-model ensemble methods. Our results show that frontier LLM models like GPT-4 achieved remarkable alignment with human graders, demonstrating a Cohen's kappa agreement of 0.88 and indicating strong potential for LLM-based OSCE grading to augment the current grading process. Open-source models also showed promising results, suggesting potential for widespread, cost-effective deployment. Further, we present a failure analysis identifying conditions where LLM grading may be less reliable in this context and recommend best practices for deploying LLMs in medical education settings.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most automated scoring modules were designed for retrospective evaluation rather than for real-time, formative feedback during ongoing practice.
“Although automated scoring modules have been developed, most were designed for retrospective evaluation rather than for real-time, formative feedback during ongoing practice.”
Cites
Still being checked.
Cited by
Sorted by citations
Artificial Intelligence in Health Professions Education assessment: AMEE Guide No. 178
Masters, K., MacNeil, H., Benjamin, J., Carver, T., Nemethy, K., Valanci-Aroesty, S., Taylor, D. C. M., Thoma, B., & Thesen, T. (2025) · Medical Teacher · doi:10.1080/0142159x.2024.2445037 · Cited by 80 · see citing works
Geathers, J., Hicke, Y., Chan, C., Rajashekar, N., Young, S., Sewell, J. L., Cornes, S. B., Kizilcec, R. F., & Shung, D. L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98420-4_17 · Cited by 28 · see citing works
Kubota, K., Nishimura, A., Seto, R., & Liu, L. (2026) · JMIR Formative Research · doi:10.2196/87830 · Cited by 5 · see citing works
Domingo, J. L. (2026) · Toxicology Letters · doi:10.1016/j.toxlet.2026.111871 · Cited by 4 · see citing works
Automated Assessment of OSCE Physical Exams using Multimodal AIPreprint
Kang, S., Holcomb, M. J., Shakur, A. H., Hein, D., Ngo, H.-T., Schuler, H., Jarrett, P. C., Dalton, T. O., & Jamieson, A. R. (2026) · medRxiv · doi:10.64898/2026.01.09.26343786 · Cited by 3 · see citing works
Self-citationMultimodal LLM for Anomaly Detection
Anba, A., Lethbridge, N., Millhouse, P., & Abid, M. R. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-08977-9_21 · Cited by 1 · see citing works
CliniSense AI for automated clinical skills assessment with real-time feedback
Khan, A. N., Tupayachi, J., Dabbs, W. S., Chamberlin, S. M., Yao, B., & Li, X. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03100-8 · Cited by 1 · see citing works
Cook, D. A., Laack, T. A., & Pankratz, V. S. (2026) · medRxiv · doi:10.64898/2026.03.20.26348918 · Cited by 1 · see citing works
Song, J. W., Kim, M., Hong, C., Kim, Y. S., Cho, J., Kim, J. H., Myung, J., Choi, A., Yoon, H., Lee, S. G. W., et al. (2026) · medRxiv · doi:10.64898/2026.02.21.26346803
Automatic Physical Examination Segmentation within Objective Structured Clinical Examination VideosPreprint
Kang, S., Holcomb, M. J., Hein, D., Shakur, A. H., Dalton, T. O., & Jamieson, A. R. (2025) · medRxiv · doi:10.1101/2025.04.03.25325195
Self-citationMcLaughlin, J., & Newton, P. (2026) · MedEdPublish · doi:10.12688/mep.21127.1
AI in Medical Education and Healthcare Management
Tayebi, L., Shetty, S. P., & Imani, S. (2026) · Springer eBooks · doi:10.1007/978-3-032-15977-9_4
Song, J. W., Kim, M., Hong, C., Kim, Y. S., Cho, J., Kim, J. H., Myung, J., Choi, A., Yoon, H., Lee, S. J., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-8972105/v1
Kropmans, T., Neverauskaité, G., Fort, K., & Melvin, E. (2026) · European Journal of Education and Pedagogy · doi:10.24018/ejedu.2026.7.2.70159
Feng, J., Wang, S., Gao, X., Wang, L., Huang, X., & Sun, X. (2026) · doi:10.2196/preprints.110683
Briley, R., Grace¹, K. R., Adcock, B., King, E. C., Ripple¹, A. D., McGoff¹, E. K., Self¹, P. D., & Tezanos, A. G. V. (2026) · Research Square · doi:10.21203/rs.3.rs-10767584/v1
Prospective deployment of multimodal AI grading for medical student OSCEs
Ngo, H.-T., Shakur, A. H., Holcomb, M. J., Kang, S., Hein, D., Gruen, J. K., Schuler, H., Jarrett, P. C., Dalton, T. O., Campbell, K. K., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03352-4
Self-citationDanneels, P., Gardic, R., Bourgeais, A., Bourreau, L., Brochard, A., Marechal-Girault, S., Annweiler, C., Lebdai, S., Chabrun, F., & Démoulins, E. (2026) · doi:10.2196/preprints.114581
Danneels, P., Gardic, R., Lebdai, S., Bourgeais, A., Bourreau, L., Brochard, A., Despré, M., Demarquette, A., Dhersin, R., Declerck, C., et al. (2026) · doi:10.2196/preprints.114564
- Authors
- Shakur, A. H., Holcomb, M. J., Hein, D., Kang, S., Dalton, T. O., Campbell, K. K., Scott, D. J., & Jamieson, A. R.
- Affiliations
- The University of Texas Southwestern Medical Center (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2410.12858
- OpenAlex
- W4403578746
- Cited by
- 9
- References
- 0
- Citations by year
- 2026: 8 · 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- large language models, OSCE, automated assessment, speech-to-text transcription, Whisper, retrieval-augmented generation, multi-model ensemble, GPT-4, open-source LLMs, failure analysis, medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
