Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments
Open accessAbstract
The United States Medical Licensing Examination (USMLE) has been a subject of performance study for artificial intelligence (AI) models. However, their performance on questions involving USMLE soft skills remains unexplored. This study aimed to evaluate ChatGPT and GPT-4 on USMLE questions involving communication skills, ethics, empathy, and professionalism. We used 80 USMLE-style questions involving soft skills, taken from the USMLE website and the AMBOSS question bank. A follow-up query was used to assess the models’ consistency. The performance of the AI models was compared to that of previous AMBOSS users. GPT-4 outperformed ChatGPT, correctly answering 90% compared to ChatGPT’s 62.5%. GPT-4 showed more confidence, not revising any responses, while ChatGPT modified its original answers 82.5% of the time. The performance of GPT-4 was higher than that of AMBOSS's past users. Both AI models, notably GPT-4, showed capacity for empathy, indicating AI's potential to meet the complex interpersonal, ethical, and professional demands intrinsic to the practice of medicine.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several studies have evaluated large language model performance on high-stakes examinations, including the USMLE, demonstrating passing-level performance without specific fine-tuning.
“Several studies have evaluated LLM performance on high-stakes examinations, including the United States Medical Licensing Examination (USMLE), demonstrating passing-level performance without specific fine-tuning (Brin et al. 2023;Bicknell et al. 2024;Kung et al. 2023).”
From Evidence on Demand: OpenEvidence versus Established LLMs on the Orthopaedic In-Training Examination (2026) · cites this paper as Brin et al. 2023
The Discussion (page unavailable) says previous studies found that LLMs could pass the USMLE, and the Results report high accuracy on this paper’s USMLE-type questions. But this study tested only 80 selected soft-skills questions, not the full examination, and our copy does not establish the claim’s ‘without specific fine-tuning’ qualifier. (LLM, conf=0.86)
Experimental — accuracy not yet measuredBrin et al. (2023) found that ChatGPT and GPT-4 showed decent accuracy in soft skill questions, with increased accuracy with GPT-4.
“While the findings showed decent accuracy in soft skill questions with ChatGPT and increased accuracy with GPT-4, the researchers point out that although the findings show the empathetic nature of AI, further research is needed using more ethical scenarios.”
From Fostering AI Literacy in Undergraduates: A ChatGPT Workshop Case Study (2025) · cites this paper as Brin et al. (2023)
The development of LLMs has pushed boundaries in tasks previously limited to humans.
“The development of LLMs has pushed boundaries in tasks previously limited to humans [6-10], leading to new policies emphasizing responsible integration, ethical considerations, and equitable access to AI in medical education [4].”
From Challenges in the Rapid and Responsible Integration of Generative Artificial Intelligence (AI) Into a New Medical School Curriculum (2025) · cites this paper as [8]
ChatGPT has been shown to pass the USMLE, the German State Examination in Medicine, and a radiology board-style examination without images.
“However, even when trained for general purposes, ChatGPT has previously been shown to pass the United States Medical Licensing Examination (USMLE), the German State Examination in Medicine, or even a radiology board-style examination without images117–120.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [118]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 14 · Cited by 319
- Authors
- Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B. S., Charney, A. W., Nadkarni, G., & Klang, E.
- Affiliations
- Tel Aviv University (IL)
- Icahn School of Medicine at Mount Sinai (US)
- Corresponding author
- Brin, Dana
- ORCID
- Venue
- Scientific Reports
- Published
- 2023-10-01
- Volume
- 13
- Issue
- 1
- Pages
- 16492-16492
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC10543445
- OpenAlex
- W4387232979
- Cited by
- 319
- References
- 14
- Citations by year
- 2026: 51 · 2025: 132 · 2024: 128 · 2023: 7 · 2022: 1
- FWCI
- 12.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (1s10od026880-01, 1s10od030463-01)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- ChatGPT, GPT-4, USMLE, soft skills, communication skills, medical ethics, empathy, professionalism, artificial intelligence in medical education, AMBOSS, AI model comparison
- MeSH terms
- Artificial Intelligence, Empathy, Medicine, Mental Processes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
