Investigating the impact of innovative AI chatbot on post‐pandemic medical education and clinical assistance: a comprehensive analysis
Open accessAbstract
Background: The COVID‐19 pandemic has significantly disrupted clinical experience and exposure of medical students and junior doctors. Artificial Intelligence (AI) integration in medical education has the potential to enhance learning and improve patient care. This study aimed to evaluate the effectiveness of three popular large language models (LLMs) in serving as clinical decision‐making support tools for junior doctors. Methods: A series of increasingly complex clinical scenarios were presented to ChatGPT, Google's Bard and Bing's AI. Their responses were evaluated against standard guidelines, and for reliability by the Flesch Reading Ease Score, Flesch–Kincaid Grade Level, the Coleman‐Liau Index, and the modified DISCERN score for assessing suitability. Lastly, the LLMs outputs were assessed by using the Likert scale for accuracy, informativeness, and accessibility by three experienced specialists. Results: In terms of readability and reliability, ChatGPT stood out among the three LLMs, recording the highest scores in Flesch Reading Ease (31.2 ± 3.5), Flesch–Kincaid Grade Level (13.5 ± 0.7), Coleman–Lau Index (13) and DISCERN (62 ± 4.4). These results suggest statistically significant superior comprehensibility and alignment with clinical guidelines in the medical advice given by ChatGPT. Bard followed closely behind, with BingAI trailing in all categories. The only non‐significant statistical differences ( P > 0.05) were found between ChatGPT and Bard's readability indices, and between the Flesch Reading Ease scores of ChatGPT/Bard and BingAI. Conclusion: This study demonstrates the potential utility of LLMs in fostering self‐directed and personalized learning, as well as bolstering clinical decision‐making support for junior doctors. However further development is needed for its integration into education.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Use of LLMs or ChatGPT in social studies significantly improved students' comprehension of intricate political and economic concepts.
“Furthermore, the use of LLMs in specific courses, such as social studies, has been found to significantly improve students' comprehension of intricate political and economic concepts (Xie, 2023).”
From Investigating the Use of chatGPT as a Tool for Enhancing Critical Thinking and Argumentation Skills in International Relations Debates among Undergraduate Students (2024) · cites this paper as Xie, 2023
The improvements observed in students' comprehension of intricate political and economic concepts in the current study corroborate the findings of Xie (2023) and Gupta and Singh (2019) that LLMs show clear explanations and real-world examples, facilitating students' understanding of complex ideas.
“The improvements observed in students' comprehension of intricate political and economic concepts in the current study corroborate the findings of Xie (2023) and Gupta and Singh (2019), who emphasized the ability of LLMs to show clear explanations and real-world examples, facilitating students' understanding of complex ideas.”
From Investigating the Use of chatGPT as a Tool for Enhancing Critical Thinking and Argumentation Skills in International Relations Debates among Undergraduate Students (2024) · cites this paper as Xie (2023)
Cites
Baidoo-Anu, D., & Owusu Ansah, L. (2023) · doi:10.2139/ssrn.4337484 · Cited by 3,004 · see citing works
Chatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355 · see citing works
Exposure to COVID-19 patients increases physician trainee stress and burnout
Kannampallil, T. G., Goss, C. W., Evanoff, B. A., Strickland, J. R., McAlister, R. P., & Duncan, J. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237301 · Cited by 389 · see citing works
Burnout syndrome in Romanian medical residents in time of the COVID-19 pandemic
Dimitriu, M. C. T., Pantea-Stoian, A., Smaranda, A. C., Nica, A. A., Carap, A. C., Constantin, V. D., Davitoiu, A. M., Cirstoveanu, C., Bacalbasa, N., Bratu, O. G., et al. (2020) · Medical Hypotheses · doi:10.1016/j.mehy.2020.109972 · Cited by 319 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 543 · see citing works
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424 · see citing works
Mapping the use of artificial intelligence in medical education: a scoping review
Rincón, E. H. H., Jimenez, D., Aguilar, L. A. C., Flórez, J. M. P., Tapia, Á. E. R., & Peñuela, C. L. J. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07089-8 · Cited by 145 · see citing works
de la Puente, M., Torres, J., Troncoso, A. L. B., Meza, Y. Y. H., & Carrascal, J. X. M. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00347-0 · Cited by 46 · see citing works
“Furthermore, the use of LLMs in specific courses, such as social studies, has been found to significantly improve students’ comprehension of intricate political and economic concepts (Xie, 2023).”
Morjaria, L., Burns, L., Bracken, K., Levinson, A. J., Ngo, Q. N., Lee, M., & Sibbald, M. (2024) · International Medical Education · doi:10.3390/ime3010004 · Cited by 33 · see citing works
- Authors
- Xie, Y., Seth, I., Hunter‐Smith, D. J., Rozen, W. M., & Seifman, M. A.
- Affiliations
- Peninsula Health (AU)
- Venue
- ANZ Journal of Surgery
- Publisher
- Wiley
- Published
- 2023-08-21
- Volume
- 94
- Issue
- 1-2
- Pages
- 68-77
- ISSN
- 1445-1433 (ISSN-L)
- 1445-2197
- OpenAlex
- W4386019728
- Cited by
- 55
- References
- 4
- Citations by year
- 2026: 8 · 2025: 21 · 2024: 25 · 2023: 1
- FWCI
- 2.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ANZ Journal of Surgery (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, AI chatbots, ChatGPT, Google Bard, Bing Chat, clinical decision support, medical residents, medical education, readability, DISCERN, clinical guideline adherence, AI-generated medical advice
- MeSH terms
- Artificial Intelligence, Education, Medical, Humans, Reproducibility of Results, Comprehension, Health Literacy, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
