CoChat
Sign Up
Cited by 5Artificial Intelligence in Healthcare and EducationInnovations in Medical EducationArtificial Intelligence in Education

Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study

Open access
Leong, Yun Hao, Nambiar, Lathiga, Tay, Victoria Y J, Lie, Sui An, Yuhe, Ke·2025·JMIR Formative Research
DOI 10.2196/77580PMC PMC12712563

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Large language models (LLMs) are increasingly used in medical education for feedback and grading; yet their role in postgraduate examination preparation remains uncertain due to inconsistent grading, hallucinations, and user acceptance. Objective: This study evaluates the Personalized Anesthesia Study Support (PASS), a specialized GPT-4 model developed to assist candidates preparing for Singapore’s postgraduate specialist anesthesiology examination. We assessed user acceptance, grading interrater reliability, and hallucination detection rates to determine the feasibility of integrating specialized LLMs into high-stakes examination preparation. Methods: PASS was built on OpenAI’s GPT-4 and adapted with domain-specific prompts and references. Twenty-one senior anesthesiology residents completed a mock short answer question examination, which was independently graded by 3 human examiners and 3 PASS iterations. Participants reviewed feedback from both PASS and standard GPT-4 and completed a technology acceptance model (TAM) survey. Grading reliability was evaluated using Cohen and Fleiss κ. Hallucination rates were assessed by participants and examiners. Results: Of the 21 participants, 17 (81%) completed the TAM survey, generating 136 responses. PASS scored significantly higher than standard GPT-4 in usefulness (mean 4.25, SD 0.50 vs mean 3.44, SD 0.82; P<.001), efficiency (mean 4.12, SD 0.61 vs mean 3.41, SD 0.74; P<.001), and likelihood of future use (mean 4.13, SD 0.75 vs mean 3.59, SD 0.90; P<.001), with no significant difference in ease of use (mean 4.56, SD 0.63 vs mean 4.50, SD 0.61; P=.35). Internal grading reliability was moderate for PASS (κ=0.522) and fair for human examiners (κ=0.275). Across 316 PASS-generated responses, 67 hallucinations and 189 deviations were labeled. Hallucination labeling rates were comparable between candidates (10/67, 15%) and examiners (57/249, 22.9%; P=.21), while examiners labeled significantly more deviations (168/249, 67.5% vs 21/67, 31%; P<.001). Conclusions: PASS demonstrated strong user acceptance and grading reliability, suggesting feasibility in high-stakes examination preparation. Experienced learners could identify major hallucinations at comparable rates to examiners, suggesting potential in self-directed learning but with continued need for caution. Further research should refine grading accuracy and explore multicenter evaluation of specialized LLMs for postgraduate medical education.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/77580

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Well-structured AI learning tools have been shown to improve examination performance.

    “well-structured AI learning tools have been shown to improve examination performance 10”

    From AI-induced never-skilling in medical education (2026) · cites this paper as [10]

    The Discussion’s Limitations and Future Work section states that the study did not examine whether PASS affected actual examination performance or learning outcomes. It reports user acceptance and grading agreement instead, so it does not show improved examination performance. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Leong et al. demonstrated that AI-assisted learning improved postgraduate examination performance.

    “Leong et al. demonstrated that AI-assisted learning improved postgraduate examination performance 10 .”

    From AI-induced never-skilling in medical education (2026) · cites this paper as [10]

    The Limitations and Future Work section explicitly says the study did not examine whether using PASS affected actual examination performance or learning outcomes. Its Conclusions report feasibility and user acceptance, not demonstrated improvement in examination performance; page numbers are not provided in our copy. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. [1]Abd-Alrazaq A AlSaad R Alhuwail D Ahmed A Healy PM Latifi S Aziz S Damseh R Alabed Alrazak S Sheikh J Large language models in medical education: opportunities, challenges, and future directions JMIR Med Educ 2023 06 01 9 e48291 10.2196/48291 37261894 v9i1e48291 37261894 PMC10273039 · DOI · Paper page
  2. [3]Lucas HC Upperman JS Robinson JR A systematic review of large language models and their implications in medical education Med Educ 2024 11 58 11 1276 1285 10.1111/medu.15402 38639098 38639098 · DOI · Paper page
  3. [4]Ye H Xu J Huang D Xie M Guo J Yang J Bao H Zhang M Zheng C Assessment of large language models’ performances and hallucinations for Chinese postgraduate medical entrance examination Discov Educ 2025 03 13 4 1 59 10.1007/s44217-025-00446-1 · DOI
  4. [5]Flodén J Grading exams using large language models: a comparison between human and AI grading of exams in higher education using ChatGPT British Educational Res J 2024 09 16 51 1 201 224 10.1002/berj.4069 · DOI
  5. [6]Quah B Zheng L Sng TJH Yong CW Islam I Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations BMC Med Educ 2024 09 03 24 1 962 10.1186/s12909-024-05881-6 39227811 10.1186/s12909-024-05881-6 39227811 PMC11373238 · DOI
  6. [7]Law AK So J Lui CT Choi YF Cheung KH Kei-Ching Hung K Graham CA AI versus human-generated multiple-choice questions for medical education: a cohort study in a high-stakes examination BMC Med Educ 2025 02 08 25 1 208 10.1186/s12909-025-06796-6 39923067 10.1186/s12909-025-06796-6 39923067 PMC11806894 · DOI
  7. [8]Emirtekin E Large language model-powered automated assessment: a systematic review Applied Sciences 2025 05 20 15 10 5683 10.3390/app15105683 · DOI
  8. [9]Fernández AA López-Torres M Fernández JJ Vázquez-García D ChatGPT as an instructor’s assistant for generating and scoring exams J Chem Educ 2024 08 14 101 9 3780 3788 10.1021/acs.jchemed.4c00231 · DOI
  9. [11]Jia Q Cui J Xi R Liu C Rashid P Li R Gehringer E On assessing the faithfulness of LLM-generated feedback on student assignments 2024 Proceedings of the 17th International Conference on Educational Data Mining July 14-17, 2024 Atlanta, GA 10.5281/zenodo.12729867 · DOI
  10. [12]Zhou J Zhang J Wan R Cui X Liu Q Guo H Shi X Fu B Meng J Yue B Zhang Y Zhang Z Integrating AI into clinical education: evaluating general practice trainees' proficiency in distinguishing AI-generated hallucinations and impacting factors BMC Med Educ 2025 03 19 25 1 406 10.1186/s12909-025-06916-2 40108629 10.1186/s12909-025-06916-2 40108629 PMC11924592 · DOI
  11. [13]Mondal H Karri JKK Ramasubramanian S Mondal S Juhi A Gupta P A qualitative survey on perception of medical students on the use of large language models for educational purposes Adv Physiol Educ 2025 03 01 49 1 27 36 10.1152/advan.00088.2024 39447120 39447120 · DOI · Paper page
  12. [14]Liu F Chang X Zhu Q Huang Y Li Y Wang H Assessing clinical medicine students' acceptance of large language model: based on technology acceptance model BMC Med Educ 2024 11 03 24 1 1251 10.1186/s12909-024-06232-1 39490999 10.1186/s12909-024-06232-1 39490999 PMC11533422 · DOI · Paper page
  13. [15]Meskó Bertalan Prompt engineering as an important emerging skill for medical professionals: tutorial J Med Internet Res 2023 10 04 25 e50638 10.2196/50638 37792434 v25i1e50638 37792434 PMC10585440 · DOI · Paper page
  14. [16]Davis FD Granić A The Technology Acceptance Model: 30 Years of TAM 2024 03 04 Cham Springer International Publishing
  15. [17]Granić A Marangunić N Technology acceptance model in educational context: a systematic literature review Brit J Educational Tech 2019 07 09 50 5 2572 2593 10.1111/bjet.12864 · DOI · Paper page
  16. [19]von Elm Erik Altman D Egger M Pocock S Gøtzsche Peter C Vandenbroucke J STROBE Initiative The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies Ann Intern Med 2007 10 16 147 8 573 7 10.7326/0003-4819-147-8-200710160-00010 17938396 147/8/573 17938396 · DOI · Paper page
  17. [20]Landis JR Koch GG The measurement of observer agreement for categorical data Biometrics 1977 03 33 1 159 174 843571 843571
  18. [21]Cruz ROD Pedagogical practice preferences among generational groups of learners: towards effective twenty-first century higher education JUTLP 2020 05 02 17(5) 10.53761/1.17.5.6 · DOI
  19. [22]Shorey S Chan V Rajendran P Ang E Learning styles, preferences and needs of generation Z healthcare students: scoping review Nurse Educ Pract 2021 11 57 103247 10.1016/j.nepr.2021.103247 34768214 S1471-5953(21)00283-3 34768214 · DOI
  20. [23]Neumann AT Yin Y Sowe S Decker S Jarke M An LLM-driven chatbot in higher education for databases and information systems IEEE Trans Educ 2025 2 68 1 103 116 10.1109/te.2024.3467912 · DOI
  21. [24]Bernabei M Colabianchi S Falegnami A Costantino F Students’ use of large language models in engineering education: a case study on technology acceptance, perceptions, efficacy, and detection chances Computers and Education: Artificial Intelligence 2023 5 100172 10.1016/j.caeai.2023.100172 · DOI
  22. [25]French S Dickerson A Mulder RA A review of the benefits and drawbacks of high-stakes final examinations in higher education High Educ 2023 12 01 88 3 893 918 10.1007/s10734-023-01148-z · DOI · Paper page
  23. [26]Davis FD User acceptance of information technology: system characteristics, user perceptions and behavioral impacts International Journal of Man-Machine Studies 1993 3 38 3 475 487 10.1006/imms.1993.1022 · DOI
  24. [28]Naujoks N Harder B Händel M Testing pays off twice: potentials of practice tests and feedback regarding exam performance and judgment accuracy Metacognition Learning 2022 03 18 17 2 479 498 10.1007/s11409-022-09295-x · DOI
  25. [29]Faherty A Counihan T Kropmans T Finn Y Inter-rater reliability in clinical assessments: do examiner pairings influence candidate ratings? BMC Med Educ 2020 05 11 20 1 147 10.1186/s12909-020-02009-4 32393228 10.1186/s12909-020-02009-4 32393228 PMC7212618 · DOI
  26. [30]Zhang D Boey M Tan YY Jia AHS Evaluating large language models for criterion-based grading from agreement to consistency NPJ Sci Learn 2024 12 30 9 1 79 10.1038/s41539-024-00291-1 39738131 10.1038/s41539-024-00291-1 39738131 PMC11683144 · DOI
  27. [31]Zhui L Fenghe L Xuehu W Qining F Wei R Ethical considerations and fundamental principles of large language models in medical education: viewpoint J Med Internet Res 2024 08 01 26 e60083 10.2196/60083 38971715 v26i1e60083 38971715 PMC11327620 · DOI
  28. [32]Ji Z Yu T Xu Y Lee N Ishii E Fung P Towards mitigating LLM hallucination via self reflection Findings of the Association for Computational Linguistics 2023 12 EMNLP 2023 Dec 6-10, 2023 Singapore 1827 1843 10.18653/v1/2023.findings-emnlp.123 · DOI

Cited by

Sorted by citations

Authors
Leong, Y. H., Nambiar, L., Tay, V. Y. J., Lie, S. A., & Yuhe, K.
Affiliations
Singapore General Hospital (SG)
Corresponding author
Leong, Yun Hao
ORCID
Venue
JMIR Formative Research
Publisher
JMIR Publications Inc.
Published
2025-12-03
Volume
9
Pages
e77580
ISSN
2561-326X (ISSN-L)
DOI
10.2196/77580
PMC
PMC12712563
OpenAlex
W4416965115
Cited by
5
References
3
Citations by year
2026: 5
FWCI
1.20
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Formative Research (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
large language models, anesthesiology residency, GPT-4, automated grading, inter-rater reliability, hallucination detection, technology acceptance, personalized feedback, medical education, high-stakes examinations
MeSH terms
Proof of Concept Study, Large Language Models, Adult, Anesthesiology, Clinical Competence, Education, Medical, Graduate, Educational Measurement, Feasibility Studies, Female, Humans, Internship and Residency, Language, Male, Surveys and Questionnaires, Singapore, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review