Exploring the Possible Use of AI Chatbots in Public Health Education: Feasibility Study
Open accessPublished version of a preprint
Abstract
Background: Artificial intelligence (AI) is a rapidly developing field with the potential to transform various aspects of health care and public health, including medical training. During the “Hygiene and Public Health” course for fifth-year medical students, a practical training session was conducted on vaccination using AI chatbots as an educational supportive tool. Before receiving specific training on vaccination, the students were given a web-based test extracted from the Italian National Medical Residency Test. After completing the test, a critical correction of each question was performed assisted by AI chatbots. Objective: The main aim of this study was to identify whether AI chatbots can be considered educational support tools for training in public health. The secondary objective was to assess the performance of different AI chatbots on complex multiple-choice medical questions in the Italian language. Methods: A test composed of 15 multiple-choice questions on vaccination was extracted from the Italian National Medical Residency Test using targeted keywords and administered to medical students via Google Forms and to different AI chatbot models (Bing Chat, ChatGPT, Chatsonic, Google Bard, and YouChat). The correction of the test was conducted in the classroom, focusing on the critical evaluation of the explanations provided by the chatbot. A Mann-Whitney U test was conducted to compare the performances of medical students and AI chatbots. Student feedback was collected anonymously at the end of the training experience. Results: In total, 36 medical students and 5 AI chatbot models completed the test. The students achieved an average score of 8.22 (SD 2.65) out of 15, while the AI chatbots scored an average of 12.22 (SD 2.77). The results indicated a statistically significant difference in performance between the 2 groups (U=49.5, P<.001), with a large effect size (r=0.69). When divided by question type (direct, scenario-based, and negative), significant differences were observed in direct (P<.001) and scenario-based (P<.001) questions, but not in negative questions (P=.48). The students reported a high level of satisfaction (7.9/10) with the educational experience, expressing a strong desire to repeat the experience (7.6/10). Conclusions: This study demonstrated the efficacy of AI chatbots in answering complex medical questions related to vaccination and providing valuable educational support. Their performance significantly surpassed that of medical students in direct and scenario-based questions. The responsible and critical use of AI chatbots can enhance medical education, making it an essential aspect to integrate into the educational system.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The use of chatbots has been increasingly reported; Gorashi et al. analyzed their strengths, weaknesses, opportunities, and threats in this context; chatbots have assisted students in history-taking practice and preparing for objective structured clinical examinations (OSCE); in one study comparing the use of ChatGPT with a traditional textbook for learning medical terminology, ChatGPT provided
“The use of chatbots has been increasingly reported [40, 112–115]. Gorashi et al. analyzed their strengths, weaknesses, opportunities, and threats in this context [107]. Chatbots have assisted students in history-taking practice [116] and preparing for objective structured clinical examinations (OSCE) [117]. In one study comparing the use of ChatGPT with a traditional textbook for learning medical”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [113]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1× · see citing works
“Medical education is no exception, with papers exploring examples of AI chatbot applications, including the generation of accurate and versatile clinical vignettes, improving personalized learning experiences, and being an adjunct in group learning [5,6].”
BackgroundWhat Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024 · see citing works
Eysenbach, G. (2023) · JMIR Medical Education · doi:10.2196/46885 · Cited by 840 · see citing works
Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions
Abd-alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P. M., Latifi, S., Aziz, S., Damseh, R., Alabed Alrazak, S., & Sheikh, J. (2023) · JMIR Medical Education · doi:10.2196/48291 · Cited by 722 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health
De Angelis, L., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1166120 · Cited by 708 · see citing works
Self-citationChatGPT - Reshaping medical education and clinical management
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.39.2.7653 · Cited by 628 · see citing works
Leveraging media and health communication strategies to overcome the COVID-19 infodemic
Mheidly, N., & Fares, J. (2020) · Journal of Public Health Policy · doi:10.1057/s41271-020-00247-w · Cited by 526 · see citing works
Yu, H. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1181712 · Cited by 409 · cited 1× · see citing works
“Especially in the medical domain, it is better to promote awareness of the benefits and limits of LLMs rather than prohibiting students from using them [24].”
BackgroundChaiyo, Y., & Nokham, R. (2017) · 2017 International Conference on Digital Arts, Media and Technology (ICDAMT) · doi:10.1109/icdamt.2017.7904957 · Cited by 274 · see citing works
Schachner, T., Keller, R., & v Wangenheim, F. (2020) · Journal of Medical Internet Research · doi:10.2196/20701 · Cited by 267 · see citing works
Moldt, J.-A., Festl-Wietek, T., Madany Mamlouk, A., Nieselt, K., Fuhl, W., & Herrmann-Werner, A. (2023) · Medical Education Online · doi:10.1080/10872981.2023.2182659 · Cited by 186 · see citing works
Technical Aspects of Developing Chatbots for Medical Applications: Scoping Review
Safi, Z., Abd-Alrazaq, A. A., Khalifa, M., & Househ, M. S. (2020) · Journal of Medical Internet Research · doi:10.2196/19127 · Cited by 129 · see citing works
AI and Medical Education — A 21st-Century Pandora’s Box
Cooper, A. Z., & Rodman, A. M. (2023) · New England Journal of Medicine · doi:10.1056/nejmp2304993 · Cited by 128 · see citing works
Publisher Correction: Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06455-0 · Cited by 101 · see citing works
Hallucinations in ChatGPT: A Cautionary Tale for Biomedical Researchers
Goddard, J. (2023) · The American Journal of Medicine · doi:10.1016/j.amjmed.2023.06.012 · Cited by 89 · see citing works
Lee, K. Y., Dabak, S. V., Kong, V. H., Park, M., Kwok, S. L. L., Silzle, M., Rachatan, C., Cook, A. R., Passanante, A., Pertwee, E., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00843-6 · Cited by 35 · see citing works
Which of the Following Is True: We Can Write Better Multiple Choice Questions
Rauschert, E. S. J., Yang, S., & Pigg, R. M. (2018) · Bulletin of the Ecological Society of America · doi:10.1002/bes2.1468 · Cited by 11 · see citing works
Tarchi, L., Damiani, S., Marini, S., et al. (2021) · Italian Journal of Medicine · doi:10.4081/itjm.2021.1470 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [10]OpenAI GPT-4 technical report arXiv. Preprint posted online on March 15, 2023
- [23]Zhang Z Zhang A Li M Smola A Shanghai ? Tong J Automatic chain of thought prompting in large language models arXiv. Preprint posted online on October 7, 2022
- [32]Chen L Zaharia M Zou J How is ChatGPT's behavior changing over time? arXiv. Posted online on July 18, 2023 2021
Cited by
Sorted by citations
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424 · see citing works
Aljamaan, F., Temsah, M.-H., Altamimi, I., Al-Eyadhy, A., Jamal, A., Alhasan, K., Mesallam, T. A., Farahat, M., & Malki, K. H. (2024) · JMIR Medical Informatics · doi:10.2196/54345 · Cited by 166 · see citing works
“The implementation of AI chatbots in health care is optimistically viewed as a means to improve health care systems, medical education, and patient outcomes [4-6].”
BackgroundArtificial intelligence in undergraduate medical education: an updated scoping review
Simoni, J., Urtubia-Fernandez, J., Mengual, E., Simoni, D. A., Royo, M., Egaña-Yin, D., Hertog, O. L. A., López-Ortiz, L., Muñoz-Tomás, A., Santiago-Martínez, P., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08188-2 · Cited by 62 · see citing works
“As medical students become physicians, independent problem-solving and quick decision-making are crucial for managing life-threatening situations [113].”
Mondal, H., Tiu, D. N., Mondal, S., Dutta, R., Naskar, A., & Podder, I. (2025) · Journal of Mid-life Health · doi:10.4103/jmh.jmh_182_24 · Cited by 6 · see citing works
- Authors
- Baglivo, F., De Angelis, L., Casigliani, V., Arzilli, G., Privitera, G. P., & Rizzo, C.
- Affiliations
- University of Pisa (IT)
- Corresponding author
- Baglivo, Francesco
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-11-01
- Volume
- 9
- Pages
- e51421
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC10652189
- OpenAlex
- W4387108520
- Cited by
- 69
- References
- 20
- Citations by year
- 2026: 15 · 2025: 35 · 2024: 18 · 2023: 1
- FWCI
- 2.76
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, cc-by)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Università di Pisa
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- AI chatbots, public health education, medical students, vaccine education, multiple-choice questions, large language models, ChatGPT, Bing Chat, Google Bard, scenario-based questions, student satisfaction, Italian language, feasibility study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
