Assessing the performance of ChatGPT in bioethics: a large language model’s moral compass in medicine
Abstract
Chat Generative Pre-Trained Transformer (ChatGPT) has been a growing point of interest in medical education yet has not been assessed in the field of bioethics. This study evaluated the accuracy of ChatGPT-3.5 (April 2023 version) in answering text-based, multiple choice bioethics questions at the level of US third-year and fourth-year medical students. A total of 114 bioethical questions were identified from the widely utilised question banks UWorld and AMBOSS. Accuracy, bioethical categories, difficulty levels, specialty data, error analysis and character count were analysed. We found that ChatGPT had an accuracy of 59.6%, with greater accuracy in topics surrounding death and patient–physician relationships and performed poorly on questions pertaining to informed consent. Of all the specialties, it performed best in paediatrics. Yet, certain specialties and bioethical categories were under-represented. Among the errors made, it tended towards content errors and application errors. There were no significant associations between character count and accuracy. Nevertheless, this investigation contributes to the ongoing dialogue on artificial intelligence’s (AI) role in healthcare and medical education, advocating for further research to fully understand AI systems’ capabilities and constraints in the nuanced field of medical bioethics.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several studies examined the performance of LLMs on various medical exams; comparisons between older and newer versions indicated that newer versions exhibited better accuracy; one study comparing ChatGPT and Google's Bard in answering multiple-choice questionnaires related to endocrinology and diabetes found ChatGPT to be slightly superior, but both models lacked up-to-date information; another
“Several studies examined the performance of LLMs on various medical exams [97–99]. Comparisons between older and newer versions indicated that newer versions exhibited better accuracy, highlighting the need for ongoing assessment of AI tools in medical education [100–102]. One study comparing ChatGPT and Google’s Bard in answering multiple-choice questionnaires (MCQs) related to endocrinology and”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [97]
Chen and colleagues reported that the accuracy of ChatGPT in answering bioethical questions and assessing ethical scenarios targeting medical students was suboptimal, with an overall accuracy of 60%.
“Chen and colleagues reported that the accuracy of ChatGPT in answering bioethical questions and assessing ethical scenarios targeting medical students was suboptimal, with an overall accuracy of 60% [97].”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [97]
Fifteen publications focused explicitly on ethical considerations concerning the use of AI in medical education.
“15 publications focused explicitly on this topic [7, 38, 84, 97, 186, 190–199].”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [97]
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
ChatGPT: the future of discharge summaries?
Patel, S. B., & Lam, K. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00021-3 · Cited by 728 · see citing works
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P., Trivedi, H. D., Vipani, A., Ayoub, W. S., Yang, J. D., Liran, O., Spiegel, B. M. R., et al. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0089 · Cited by 655 · see citing works
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563 · see citing works
Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment
Mihalache, A., Popovic, M. M., & Muni, R. H. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.1144 · Cited by 305 · see citing works
Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06603-5 · Cited by 282 · see citing works
Suchman, K., Garg, S., & Trindade, A. J. (2023) · American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002320 · Cited by 139 · see citing works
Kairys, S. W. (2020) · Pediatric Clinics of North America · doi:10.1016/j.pcl.2019.11.001 · Cited by 36 · see citing works
Exploring Preparation for the USMLE Step 2 Exams to Inform Best Practices
Drake, E., Phillips, J. P., & Kovar-Gough, I. (2021) · PRiMER · doi:10.22454/primer.2021.693105 · Cited by 18 · see citing works
Fang, C., Ling, J., Zhou, J., et al. (2023) · medRxiv · doi:10.1101/2023.05.03.23289443 · Cited by 16 · see citing works
Disproof of Concept: Resolving Ethical Dilemmas Using Algorithms
Pilkington, B., & Binkley, C. E. (2022) · The American Journal of Bioethics · doi:10.1080/15265161.2022.2087789 · Cited by 14 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Artificial intelligence in undergraduate medical education: an updated scoping review
Simoni, J., Urtubia-Fernandez, J., Mengual, E., Simoni, D. A., Royo, M., Egaña-Yin, D., Hertog, O. L. A., López-Ortiz, L., Muñoz-Tomás, A., Santiago-Martínez, P., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08188-2 · Cited by 62 · see citing works
“While many of the overviews and perspectives referred to ethical considerations concerning the use of AI in medical education [77], 15 publications focused explicitly on this topic [7, 38, 84, 97, 186, 190–199].”
- Authors
- Chen, J., Cadiente, A., Kasselman, L. J., & Pilkington, B.
- Affiliations
- Hackensack Meridian School of Medicine (US)
- Hackensack Meridian Health (US)
- Corresponding author
- Chen, Jamie
- Venue
- Journal of Medical Ethics
- Publisher
- BMJ
- Published
- 2023-11-16
- Volume
- 50
- Issue
- 2
- Pages
- 97-101
- ISSN
- 0306-6800 (ISSN-L)
- 1473-4257
- OpenAlex
- W4388728551
- Cited by
- 28
- References
- 13
- Citations by year
- 2026: 10 · 2025: 8 · 2024: 10
- FWCI
- 1.13
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Medical Ethics (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- ChatGPT, large language models, bioethics education, medical students, artificial intelligence in medical education, multiple-choice question accuracy, informed consent, death and dying, physician-patient relationship, AMBOSS, error analysis
- MeSH terms
- Artificial Intelligence, Child, Education, Medical, Humans, Language, Medicine, Morals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
