Zero-Shot Multimodal Question Answering for Assessment of Medical Student OSCE Physical Exam Videos
Open accessAbstract
The Objective Structured Clinical Examination (OSCE) is a critical component of medical education whereby the data gathering, clinical reasoning, physical examination, diagnostic and planning capabilities of medical students are assessed in a simulated outpatient clinical setting with standardized patient actors (SPs) playing the role of patients with a predetermined diagnosis, or case. This study is the first to explore the zero-shot automation of physical exam grading in OSCEs by applying multimodal question answering techniques to the analysis of audiovisual recordings of simulated medical student encounters. Employing a combination of large multimodal models (LLaVA-1.6 7B,13B,34B, GPT-4V, and GPT-4o), automatic speech recognition (Whisper v3), and large language models (LLMs), we assess the feasibility of applying these component systems to the domain of student evaluation without any retraining. Our approach converts video content into textual representations, encompassing the transcripts of the audio component and structured descriptions of selected video frames generated by the multimodal model. These representations, referred to as “exam stories,” are then used as context for an abstractive question-answering problem via an LLM. A collection of 191 audiovisual recordings of medical student encounters with an SP for a single OSCE case was used as a test bed for exploring relevant features of successful exams. During this case, the students should have performed three physical exams: 1) mouth exam, 2) ear exam, and 3) nose exam. These examinations were each scored by two trained, non-faculty standardized patient evaluators (SPE) using the audiovisual recordings—an experienced, non-faculty SPE adjudicated disagreements. The percentage agreement between the described methods and the SPEs’ determination of exam occurrence as measured by percentage agreement varied from 26% to 83%. The audio-only methods, which relied exclusively on the transcript for exam recognition, performed uniformly higher by this measure compared to both the image-only methods and the combined methods across differing model sizes. The outperformance of the transcript-only model was strongly linked to the presence of key phrases where the student-physician would “signpost” the progression of the physical exam for the standardized patient, either alerting when they were about to begin an examination or giving the patient instructions. Multimodal models offer tremendous opportunity for improving the workflow of the physical examinations’ evaluation, for example by saving time and guiding focus for better assessment. While these models offer the promise of unlocking audiovisual data for downstream analysis with natural language processing methods, our findings reveal a gap between the off-the-shelf AI capabilities of many available models and the nuanced requirements of clinical practice, highlighting a need for further development and enhanced evaluation protocols in this area. We are actively pursuing a variety of approaches to realize this vision.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most automated scoring modules were designed for retrospective evaluation rather than for real-time, formative feedback during ongoing practice.
“Although automated scoring modules have been developed, most were designed for retrospective evaluation rather than for real-time, formative feedback during ongoing practice.”
From Development and Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation (2026) · cites this paper as [19]
The Prior Work—Simulation Assessment section (page not provided) says most success in automated OSCE grading has been with post-encounter notes, supporting a retrospective emphasis. It does not establish that most automated scoring modules were designed for retrospective use rather than real-time, formative feedback during practice. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 1× · see citing works
“We used the vLLM (Kwon et al., 2023) inference engine to serve the model with float16 precision and a default context window of 8,192 tokens.”
MethodEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802 · see citing works
An interpretable natural language processing system for written medical examination assessment
Sarker, A., Klein, A. Z., Mee, J., Harik, P., & Gonzalez‐Hernandez, G. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103268 · Cited by 44 · see citing works
A dataset of simulated patient-physician medical interviews with a focus on respiratory cases
Fareez, F., Parikh, T., Wavell, C., Shahab, S., Chevalier, M., Good, S., De Blasi, I., Rhouma, R., McMahon, C., Lam, J.-P., et al. (2022) · Scientific Data · doi:10.1038/s41597-022-01423-1 · Cited by 39 · see citing works
Salt, J., Harik, P., & Barone, M. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002558 · Cited by 23 · see citing works
Automated Patient Note Grading: Examining Scoring Reliability and Feasibility
Bond, W. F., Zhou, J., Bhat, S., Park, Y. S., Ebert-Allen, R. A., Ruger, R. L., & Yudkowsky, R. (2023) · Academic Medicine · doi:10.1097/acm.0000000000005357 · Cited by 18 · see citing works
Object Detection During Newborn Resuscitation Activities
Meinich-Bache, Ø., Engan, K., Austvoll, I., Eftestøl, T. C., Myklebust, H., Blacy, L., Kidanto, H. L., & Ersdal, H. L. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2924808 · Cited by 15 · see citing works
Automatic rubric-based content grading for clinical notes
Yim, W., Mills, A., Chun, H., Hashiguchi, T., Yew, J., & Lu, B. (2019) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/d19-6216 · Cited by 5 · see citing works
Vedovato, S., Kang, S.-Y., Holcomb, M. J., Campbell, K. K., Scott, D. J., Dalton, T. O., Danuser, G. M., & Jamieson, A. R. (2024) · IEEE International Conference on Artificial Intelligence for Medicine, Health and Care (AIMHC) · doi:10.1109/aimhc59811.2024.00036 · Cited by 4 · see citing works
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 4
- Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO- Awareness. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16344-16359.
- Meladel Mistica, Timothy Baldwin, Marisa Cordella, and Simon Musgrave. 2008. Applying discourse analysis and data mining methods to spoken OSCE assessments. In Proceedings of the 22nd International Conference on Computational Linguistics -COLING '08, volume 1, pages 577- 584, Manchester, United Kingdom. Association for Computational Linguistics.
- Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492-28518.
- PMLR.
Cited by
Sorted by citations
Rubrics to Prompts: Assessing Medical Student Post-Encounter Notes with AI
Jamieson, A. R., Holcomb, M. J., Dalton, T. O., Campbell, K. K., Vedovato, S., Shakur, A. H., Kang, S.-Y., Hein, D., Lawson, J., Danuser, G. M., et al. (2024) · NEJM AI · doi:10.1056/aics2400631 · Cited by 37 · see citing works
Self-citationBasil, M., Ahmed, W., Hajeomar, R., Strawbridge, J., Lynch, M., & Mukhalalati, B. (2026) · BMC Medical Education · doi:10.1186/s12909-025-08527-3 · Cited by 9 · see citing works
Automated Assessment of OSCE Physical Exams using Multimodal AIPreprint
Kang, S., Holcomb, M. J., Shakur, A. H., Hein, D., Ngo, H.-T., Schuler, H., Jarrett, P. C., Dalton, T. O., & Jamieson, A. R. (2026) · medRxiv · doi:10.64898/2026.01.09.26343786 · Cited by 3 · see citing works
Self-citationLeón-Ariza, S. A., Orobio, M. C., Ibáñez-Gutiérrez, H. M., García-Serna, J., & Vivas, D. A. (2026) · Medical Teacher · doi:10.1080/0142159x.2026.2685794 · Cited by 3 · see citing works
Song, J. W., Kim, M., Hong, C., Kim, Y. S., Cho, J., Kim, J. H., Myung, J., Choi, A., Yoon, H., Lee, S. G. W., et al. (2026) · medRxiv · doi:10.64898/2026.02.21.26346803
Prospective deployment of multimodal AI grading for medical student OSCEs
Ngo, H.-T., Shakur, A. H., Holcomb, M. J., Kang, S., Hein, D., Gruen, J. K., Schuler, H., Jarrett, P. C., Dalton, T. O., Campbell, K. K., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03352-4
Self-citation
- Authors
- Holcomb, M. J., Kang, S., Shakur, A., Vedovato, S., Hein, D., Dalton, T. O., Campbell, K. K., Scott, D. J., Danuser, G., & Jamieson, A. R.
- Affiliations
- The University of Texas Southwestern Medical Center (US)
- Southwestern Medical Center (US)
- Corresponding author
- Holcomb, Michael J.
- Jamieson, Andrew R.
- ORCID
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2024-06-09
- OpenAlex
- W4399508082
- Cited by
- 6
- References
- 8
- Citations by year
- 2026: 5 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- dx.doi.org
- Locations
- medRxiv (acceptedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- OSCE, zero-shot question answering, multimodal question answering, medical student assessment, standardized patients, LLaVA, GPT-4V, GPT-4o, large language models, interobserver agreement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
