Comparative accuracy of ChatGPT-4, Microsoft Copilot and Google Gemini in the Italian entrance test for healthcare sciences degrees: a cross-sectional study
Open accessAbstract
Background: Artificial intelligence (AI) chatbots are emerging educational tools for students in healthcare science. However, assessing their accuracy is essential prior to adoption in educational settings. This study aimed to assess the accuracy of predicting the correct answers from three AI chatbots (ChatGPT-4, Microsoft Copilot and Google Gemini) in the Italian entrance standardized examination test of healthcare science degrees (CINECA test). Secondarily, we assessed the narrative coherence of the AI chatbots’ responses (i.e., text output) based on three qualitative metrics: the logical rationale behind the chosen answer, the presence of information internal to the question, and presence of information external to the question. Methods: An observational cross-sectional design was performed in September of 2023. Accuracy of the three chatbots was evaluated for the CINECA test, where questions were formatted using a multiple-choice structure with a single best answer. The outcome is binary (correct or incorrect). Chi-squared test and a post hoc analysis with Bonferroni correction assessed differences among chatbots performance in accuracy. A p -value of < 0.05 was considered statistically significant. A sensitivity analysis was performed, excluding answers that were not applicable (e.g., images). Narrative coherence was analyzed by absolute and relative frequencies of correct answers and errors. Results: Overall, of the 820 CINECA multiple-choice questions inputted into all chatbots, 20 questions were not imported in ChatGPT-4 ( n = 808) and Google Gemini ( n = 808) due to technical limitations. We found statistically significant differences in the ChatGPT-4 vs Google Gemini and Microsoft Copilot vs Google Gemini comparisons ( p -value < 0.001). The narrative coherence of AI chatbots revealed “Logical reasoning” as the prevalent correct answer ( n = 622, 81.5%) and “Logical error” as the prevalent incorrect answer ( n = 40, 88.9%). Conclusions: Our main findings reveal that: (A) AI chatbots performed well; (B) ChatGPT-4 and Microsoft Copilot performed better than Google Gemini; and (C) their narrative coherence is primarily logical. Although AI chatbots showed promising accuracy in predicting the correct answer in the Italian entrance university standardized examination test, we encourage candidates to cautiously incorporate this new technology to supplement their learning rather than a primary resource. Trial registration: Not required.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Research on the application of AI Chatbots in physiotherapy education remains obscure.
“research on their application in physiotherapy education remains obscure [4].”
From Knowledge and use, Perceptions of Benefits and Limitations of Artificial Intelligence Chatbots among Italian Physiotherapy students: a cross-sectional National Study (2025) · cites this paper as [4]
The Background (page not supplied) says AI-chatbot exam studies have focused on medical specialties rather than healthcare sciences such as rehabilitation, and reports a gap in research on healthcare-science admissions tests. That supports a narrower research gap, but not the broader claim that research on AI chatbots throughout physiotherapy education remains obscure. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995 · see citing works
Prompt Engineering with ChatGPT: A Guide for Academic Writers
Giray, L. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03272-4 · Cited by 776 · see citing works
Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions
Abd-alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P. M., Latifi, S., Aziz, S., Damseh, R., Alabed Alrazak, S., & Sheikh, J. (2023) · JMIR Medical Education · doi:10.2196/48291 · Cited by 722 · see citing works
Bhayana, R., Krishna, S., & Bleakney, R. R. (2023) · Radiology · doi:10.1148/radiol.230582 · Cited by 460 · see citing works
Ali, R., Tang, O. Y., Connolly, I. D., Fridley, J., Shin, J. H., Sullivan, P. L. Z., Cielo, D., Oyelese, A. A., Doberstein, C. E., Telfeian, A. E., et al. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002551 · Cited by 354 · see citing works
Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments
Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B. S., Charney, A. W., Nadkarni, G., & Klang, E. (2023) · Scientific Reports · doi:10.1038/s41598-023-43436-9 · Cited by 319 · see citing works
Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment
Mihalache, A., Popovic, M. M., & Muni, R. H. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.1144 · Cited by 305 · see citing works
ChatGPT in education: global reactions to AI innovations
Fütterer, T., Fischer, C., Alekseeva, A., Chen, X., Tate, T., Warschauer, M., & Gerjets, P. (2023) · Scientific Reports · doi:10.1038/s41598-023-42227-6 · Cited by 242 · cited 1× · see citing works
“Since their introduction, AI chatbots have gained considerable attention and sparked discussions in medical and health science education and clinical practice [4–7].”
BackgroundGiannakopoulos, K., Kavadella, A., Aaqel Salim, A., Stamatopoulos, V., & Kaklamanos, E. G. (2023) · Journal of Medical Internet Research · doi:10.2196/51580 · Cited by 227 · see citing works
The Advent of Generative Language Models in Medical Education
Karabacak, M., Ozkara, B. B., Margetis, K., Wintermark, M., & Bisdas, S. (2023) · JMIR Medical Education · doi:10.2196/48163 · Cited by 208 · see citing works
Massey, P. A., Montgomery, C., & Zhang, A. S. (2023) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-23-00396 · Cited by 194 · see citing works
Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination
Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K., & Młyńczak, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-46995-z · Cited by 156 · see citing works
Toyama, Y., Harigai, A., Abe, M., Nagano, M., Kawabata, M., Seki, Y., & Takase, K. (2023) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01491-2 · Cited by 126 · see citing works
Guerra, G. A., Hofmann, H., Sobhani, S., Hofmann, G., Gomez, D., Soroudi, D., Hopkins, B. S., Dallas, J., Pangal, D. J., Cheok, S., et al. (2023) · World Neurosurgery · doi:10.1016/j.wneu.2023.08.042 · Cited by 123 · see citing works
ChatGPT in medical school: how successful is AI in progress testing?
Friederichs, H., Friederichs, W. J., & März, M. (2023) · Medical Education Online · doi:10.1080/10872981.2023.2220920 · Cited by 122 · see citing works
The Pros and Cons of Using ChatGPT in Medical Education: A Scoping Review
Mohammad, B., Supti, T., Alzubaidi, M., Shah, H., Alam, T., Shah, Z., & Househ, M. (2023) · Studies in Health Technology and Informatics · doi:10.3233/shti230580 · Cited by 114 · cited 2× · see citing works
“AI chatbots can provide simulations with digital patients, personalized feedback, and help eliminate language barriers; they also present biases, ethical and legal concerns, and content quality issues [8, 9].”
BackgroundChatGPT: not all languages are equal
Seghier, M. L. (2023) · Nature · doi:10.1038/d41586-023-00680-3 · Cited by 108 · see citing works
Comparative Performance of ChatGPT and Bard in a Text-Based Radiology Knowledge Assessment
Patil, N. S., Huang, R. S., van der Pol, C. B., & Larocque, N. (2023) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231193716 · Cited by 101 · see citing works
Giannos, P., & Delardas, O. (2023) · JMIR Medical Education · doi:10.2196/47737 · Cited by 100 · see citing works
ChatGPT failed Taiwan’s Family Medicine Board Exam
Weng, T.-L., Wang, Y.-M., Chang, S., Chen, T.-J., & Hwang, S.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000946 · Cited by 100 · see citing works
Performance of ChatGPT on Specialty Certificate Examination in Dermatology multiple-choice questions
Passby, L., Jenko, N., & Wernham, A. G. H. (2023) · Clinical and Experimental Dermatology · doi:10.1093/ced/llad197 · Cited by 98 · see citing works
Lewandowski, M., Łukowicz, P., Świetlik, D., & Barańska‐Rybak, W. (2023) · Clinical and Experimental Dermatology · doi:10.1093/ced/llad255 · Cited by 98 · see citing works
Roos, J., Kasapovic, A., Jansen, T., & Kaczmarczyk, R. (2023) · JMIR Medical Education · doi:10.2196/46482 · Cited by 98 · see citing works
Performance of ChatGPT in medical examinations: A systematic review and a meta‐analysis
Levin, G., Horesh, N., Brezinov, Y., & Meyer, R. (2023) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.17641 · Cited by 92 · see citing works
Yanagita, Y., Yokokawa, D., Uchida, S., Tawara, J., & Ikusaka, M. (2023) · JMIR Formative Research · doi:10.2196/48023 · Cited by 92 · see citing works
Improved Performance of ChatGPT-4 on the OKAP Examination: A Comparative Study with ChatGPT-3.5
Teebagy, S., Colwell, L., Wood, E., Yaghy, A., & Faustina, M. (2023) · Journal of Academic Ophthalmology · doi:10.1055/s-0043-1774399 · Cited by 86 · see citing works
Giannos, P. (2023) · BMJ Neurology Open · doi:10.1136/bmjno-2023-000451 · Cited by 84 · see citing works
Madrid-García, A., Rosales, Z., Freites, D., Pérez-Sancristóbal, I., Pato-Cour, E., Plasencia‐Rodríguez, C., Osorio, L. C., Abásolo, L., León, L., Fernández‐Gutiérrez, B., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-49483-6 · Cited by 77 · see citing works
Kumari, A., Kumari, A., Singh, A., Singh, S. K., Juhi, A., Dhanvijay, A. K. D., Pinjar, M. J., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.43861 · Cited by 69 · see citing works
ChatGPT can pass the AHA exams: Open-ended questions outperform multiple-choice format
Zhu, L., Mou, W., Yang, T., & Chen, R. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2023.109783 · Cited by 65 · see citing works
Rossettini, G., Cook, C. E., Palese, A., Pillastrini, P., & Turolla, A. (2023) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2023.12000 · Cited by 62 · see citing works
Self-citationBonetti, M. A., Giorgino, R., Afflitto, G. G., Lorenzi, F. D., & Egro, F. M. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03318-7 · Cited by 59 · see citing works
Dhanvijay, A. K. D., Pinjar, M. J., Dhokane, N., Sorte, S. R., Kumari, A., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.42972 · Cited by 59 · cited 2× · see citing works
“Secondly, we studied three AI Chatbots without considering others presented in the market (e.g., Cloude, Perplexity) [31].”
BackgroundMethodInfluence on the accuracy in ChatGPT: Differences in the amount of information per medical field
Haze, T., Kawano, R., Takase, H., Suzuki, S., Hirawa, N., & Tamura, K. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105283 · Cited by 58 · see citing works
Torres-Zegarra, B. C., Rios-García, W., Ñaña-Cordova, A. M., Arteaga-Cisneros, K. F., Benavente-Chalco, X. C., Bustamante-Ordoñez, M. A., Rios, C. J. G., Godoy, C. A. R., Quezada, K. L. T. P., Gutierrez-Arratia, J. D., et al. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.30 · Cited by 58 · see citing works
Cuthbert, R., & Simpson, A. I. (2023) · Postgraduate Medical Journal · doi:10.1093/postmj/qgad053 · Cited by 57 · see citing works
The challenges imposed by artificial intelligence: are we ready in medical education?
Azer, S. A., & Guerrero, A. P. S. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04660-z · Cited by 48 · see citing works
Ebrahimian, M., Behnam, B., Ghayebi, N., & Sobhrakhshankhah, E. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100815 · Cited by 47 · see citing works
Technology in the Era of COVID-19: A Systematic Review of Current Evidence
Mehraeen, E., Mehrtak, M., SeyedAlinaghi, S., Nazeri, Z., Afsahi, A. M., Behnezhad, F., Vahedi, F., Barzegary, A., Karimi, A., Mehrabi, N., et al. (2022) · Infectious Disorders - Drug Targets · doi:10.2174/1871526522666220324090245 · Cited by 46 · see citing works
Performance of ChatGPT and Bard on the official part 1 FRCOphth practice questions
Fowler, T., Pullen, S., & Birkett, L. (2023) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324091 · Cited by 46 · see citing works
Assessing the Performance of GPT-3.5 and GPT-4 on the 2023 Japanese Nursing Examination
Kaneda, Y., Takahashi, R., Kaneda, U., Akashima, S., Okita, H., Misaki, S., Yamashiro, A., Ozaki, A., & Tanimoto, T. (2023) · Cureus · doi:10.7759/cureus.42924 · Cited by 45 · see citing works
Meo, S. A., Al-Khlaiwi, T., AbuKhalaf, A. A., Meo, A. S., & Klonoff, D. C. (2025) · Journal of Diabetes Science and Technology · doi:10.1177/19322968231203987 · Cited by 40 · cited 1× · see citing works
“Others have reported mixed results when comparing the accuracy of multiple AI chatbots (e.g., ChatGPT-4 vs Microsoft Bing, ChatGPT-4 vs Google Bard) in several medical examinations tests [23–29].”
ResultThe Development of Coherence in Narratives by Understanding Intentional Action
Trabasso, T. (1991) · Advances in psychology · doi:10.1016/s0166-4115(08)61559-9 · Cited by 33 · see citing works
Kleinig, O., Gao, C., & Bacchi, S. (2023) · The Medical Journal of Australia · doi:10.5694/mja2.52061 · Cited by 29 · see citing works
Artificial Intelligence in COVID-19 Management: A Systematic Review
Mohammadi, S., Mohammadi, S., SeyedAlinaghi, S., et al. (2023) · Journal of Computer Science · doi:10.3844/jcssp.2023.554.568 · Cited by 27 · see citing works
Performance of ChatGPT in French language Parcours d'Accès Spécifique Santé test and in OBGYN
Guigue, P., Meyer, R., Lioux, G. T., et al. (2023) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.15083 · Cited by 20 · see citing works
Nowell, J. L. (2009) · BMJ · doi:10.1136/bmj.b450 · Cited by 19 · see citing works
Scaioli, G., Moro, G. L., Conrado, F., Rosset, L., Bert, F., & Siliquini, R. (2023) · Annali dell Istituto Superiore di Sanità · doi:10.4415/ann_23_04_05 · Cited by 19 · see citing works
The Challenges of ChatGPT in Healthcare Scientific Writing
SeyedAlinaghi, S., Abbaspour, F., & Mehraeen, E. (2023) · Shiraz E-Medical Journal · doi:10.5812/semj-141861 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Salman, I. M., Ameer, O. Z., Khanfar, M. A., & Hsieh, Y.-H. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1495378 · Cited by 36 · see citing works
Tortella, F., Palese, A., Turolla, A., Castellini, G., Pillastrini, P., Landuzzi, M. G., Cook, C., Galeoto, G., Giovannico, G., Rodeghiero, L., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07176-w · Cited by 28 · see citing works
Self-citation
- Authors
- Rossettini, G., Rodeghiero, L., Corradi, F., Cook, C., Pillastrini, P., Turolla, A., Castellini, G., Chiappinotto, S., Gianola, S., & Palese, A.
- Affiliations
- University of Verona (IT)
- Krankenhaus Meran (IT)
- Duke University (US)
- University of Bologna (IT)
- IRCCS Ospedale Galeazzi - Sant'Ambrogio (IT)
- University of Udine (IT)
- Corresponding author
- Rossettini, Giacomo
- Rodeghiero, Lia
- Chiappinotto, Stefania
- Gianola, Silvia
- Palese, Alvisa
- ORCID
- Venue
- BMC Medical Education
- Published
- 2024-06-26
- Volume
- 24
- Issue
- 1
- Pages
- 694-694
- ISSN
- 1472-6920 (ISSN-L)
- PMC
- PMC11210096
- OpenAlex
- W4400041715
- Cited by
- 103
- References
- 54
- Citations by year
- 2026: 39 · 2025: 52 · 2024: 12
- FWCI
- 9.63
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Provincia autonoma di Bolzano - Alto Adige
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- AI chatbots, ChatGPT-4, Microsoft Copilot, Google Gemini, multiple-choice question accuracy, narrative coherence, artificial intelligence in medical education
- MeSH terms
- Artificial Intelligence, Cross-Sectional Studies, Educational Measurement, Female, Humans, Italy, Male
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
