ChatGPT’s Response Consistency: A Study on Repeated Queries of Medical Examination Questions
Open accessAbstract
(1) Background: As the field of artificial intelligence (AI) evolves, tools like ChatGPT are increasingly integrated into various domains of medicine, including medical education and research. Given the critical nature of medicine, it is of paramount importance that AI tools offer a high degree of reliability in the information they provide. (2) Methods: A total of n = 450 medical examination questions were manually entered into ChatGPT thrice, each for ChatGPT 3.5 and ChatGPT 4. The responses were collected, and their accuracy and consistency were statistically analyzed throughout the series of entries. (3) Results: ChatGPT 4 displayed a statistically significantly improved accuracy with 85.7% compared to that of 57.7% of ChatGPT 3.5 (p < 0.001). Furthermore, ChatGPT 4 was more consistent, correctly answering 77.8% across all rounds, a significant increase from the 44.9% observed from ChatGPT 3.5 (p < 0.001). (4) Conclusions: The findings underscore the increased accuracy and dependability of ChatGPT 4 in the context of medical education and potential clinical decision making. Nonetheless, the research emphasizes the indispensable nature of human-delivered healthcare and the vital role of continuous assessment in leveraging AI in medicine.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several studies examined the performance of LLMs on various medical exams; comparisons between older and newer versions indicated that newer versions exhibited better accuracy; one study comparing ChatGPT and Google's Bard in answering multiple-choice questionnaires related to endocrinology and diabetes found ChatGPT to be slightly superior, but both models lacked up-to-date information; another
“Several studies examined the performance of LLMs on various medical exams [97–99]. Comparisons between older and newer versions indicated that newer versions exhibited better accuracy, highlighting the need for ongoing assessment of AI tools in medical education [100–102]. One study comparing ChatGPT and Google’s Bard in answering multiple-choice questionnaires (MCQs) related to endocrinology and”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [102]
Split into 3 atoms — 2 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806 · cited 1× · see citing works
“These include increased accuracy, a reduced tendency to generate implausible or irrelevant information, and a boost in the confidence levels of its responses, as our study effectively illustrates [26–29].”
ResultDave, T., Athaluri, S. A., & Singh, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1169595 · Cited by 1,204 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
Assessment methods in medical education
Norcini, J. J., & McKinley, D. W. (2007) · Teaching and Teacher Education · doi:10.1016/j.tate.2006.12.021 · Cited by 366 · see citing works
ChatGPT outperforms humans in emotional awareness evaluations
Elyoseph, Z., Hadar-Shoval, D., Asraf, K., & Lvovsky, M. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1199058 · Cited by 324 · see citing works
Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments
Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B. S., Charney, A. W., Nadkarni, G., & Klang, E. (2023) · Scientific Reports · doi:10.1038/s41598-023-43436-9 · Cited by 319 · see citing works
Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023) · JMIR Medical Education · doi:10.2196/48002 · Cited by 271 · see citing works
Massey, P. A., Montgomery, C., & Zhang, A. S. (2023) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-23-00396 · Cited by 194 · see citing works
Chatbot vs medical student performance on free-response clinical reasoning examinations.
Strong, E., DiGiammarino, A., Weng, Y., Kumar, A., Hosamani, P., Hom, J., & Chen, J. H. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.2909 · Cited by 169 · see citing works
Hoch, C. C., Wollenberg, B., Lüers, J.-C., Knoedler, S., Knoedler, L., Frank, K., Cotofana, S., & Alfertshofer, M. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08051-4 · Cited by 154 · see citing works
Self-citationMoshirfar, M., Altaf, A. W., Stoakes, I. M., Tuttle, J. J., & Hoopes, P. C. (2023) · Cureus · doi:10.7759/cureus.40822 · Cited by 124 · see citing works
Yokoi, R., Eguchi, Y., Fujita, T., & Nakayachi, K. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1861763 · Cited by 122 · see citing works
Trust me, i'm a chatbot: how artificial intelligence in health care fails the Turing Test
Powell, J. (2019) · Journal of Medical Internet Research · doi:10.2196/16222 · Cited by 115 · see citing works
Tangadulrat, P., Sono, S., & Tangtrakulwanich, B. (2023) · JMIR Medical Education · doi:10.2196/50658 · Cited by 99 · see citing works
Assessment Methods in Undergraduate Medical Education
Al-Wardy, N. (2010) · Sultan Qaboos University medical journal · doi:10.18295/2075-0528.1187 · Cited by 84 · see citing works
Performance of ChatGPT on the Peruvian National Licensing Medical Examination: Cross-Sectional Study
Flores-Cohaila, J. A., García-Vicente, A., Vizcarra-Jiménez, S. F., la Cruz-Galán, J. P. D., Gutierrez-Arratia, J. D., Torres, B. G. Q., & Taype-Rondán, Á. (2023) · JMIR Medical Education · doi:10.2196/48039 · Cited by 83 · see citing works
Koga, S., Martin, N. B., & Dickson, D. W. (2024) · Brain Pathology · doi:10.1111/bpa.13207 · Cited by 78 · see citing works
ChatGPT performance in the medical specialty exam: An observational study
Oztermeli, A. D., & Öztermeli, A. (2023) · Medicine · doi:10.1097/md.0000000000034673 · Cited by 71 · see citing works
Kumari, A., Kumari, A., Singh, A., Singh, S. K., Juhi, A., Dhanvijay, A. K. D., Pinjar, M. J., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.43861 · Cited by 69 · see citing works
Assessing the accuracy of ChatGPT references in head and neck and ENT disciplines
Frosolini, A., Franz, L., Benedetti, S., Vaira, L. A., de Filippis, C., Gennaro, P., Marioni, G., & Gabriele, G. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08205-4 · Cited by 63 · see citing works
Zaitsu, W., & Jin, M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0288453 · Cited by 61 · see citing works
Alfertshofer, M. G., Hoch, C. C., Funk, P. F., Hollmann, K., Wollenberg, B., Knoedler, S. A., & Knoedler, L. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03338-3 · Cited by 60 · see citing works
Self-citationDhanvijay, A. K. D., Pinjar, M. J., Dhokane, N., Sorte, S. R., Kumari, A., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.42972 · Cited by 59 · see citing works
Knoedler, L., Alfertshofer, M. G., Knoedler, S. A., Hoch, C. C., Funk, P. F., Cotofana, S. C., Maheta, B. J., Frank, K., Brébant, V., Prantl, L., et al. (2024) · JMIR Medical Education · doi:10.2196/51148 · Cited by 50 · see citing works
Self-citationStonier, T. (1992) · Springer eBooks · doi:10.1007/978-1-4471-1835-0 · Cited by 48 · see citing works
Assessment methods in medical education.
(2008) · Europe PMC (PubMed Central) · Cited by 47 · see citing works
ChatGPT’s advice is perceived as better than that of professional advice columnists
Howe, P. D. L., Fay, N., Saletta, M., & Hovy, E. H. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1281255 · Cited by 42 · see citing works
Norcini, J. J., Grabovsky, I., Barone, M., et al. (2023) · Academic Medicine · doi:10.1097/acm.0000000000005480 · Cited by 38 · see citing works
Is Artificial Intelligence Better Than Human Clinicians in Predicting Patient Outcomes?
Lee, J. (2020) · Journal of Medical Internet Research · doi:10.2196/19918 · Cited by 36 · see citing works
Ireland, S. (2023) · Acta Informatica Medica · doi:10.5455/aim.2023.31.300-305 · Cited by 22 · see citing works
Automatic Question Answering for Medical MCQs: Can It Go Further than Information Retrieval?
Ha, L. A., & Yaneva, V. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_049 · Cited by 19 · see citing works
The Evolution of Machine Intelligence
Stonier, T. (1992) · Springer eBooks · doi:10.1007/978-1-4471-1835-0_6 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [8]Singhal K. Tu T. Gottweis J. Sayres R. Wulczyn E. Hou L. Clark K. Pfohl S. Cole-Lewis H. Neal D. Towards Expert-Level Medical Question Answering with Large Language Models arXiv 2023 2305.09617 PMC11922739 39779926
Cited by
Sorted by citations
Performance of multimodal artificial intelligence chatbots evaluated on clinical oncology cases
Chen, D., Huang, R. S., Jomy, J., Wong, P., Yan, M., Croke, J., Tong, D., Hope, A., Eng, L., & Raman, S. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.37711 · Cited by 49 · see citing works
Artificial intelligence in undergraduate medical education: an updated scoping review
Simoni, J., Urtubia-Fernandez, J., Mengual, E., Simoni, D. A., Royo, M., Egaña-Yin, D., Hertog, O. L. A., López-Ortiz, L., Muñoz-Tomás, A., Santiago-Martínez, P., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08188-2 · Cited by 62 · see citing works
“Comparisons between older and newer versions indicated that newer versions exhibited better accuracy, highlighting the need for ongoing assessment of AI tools in medical education [100– 102].”
Higher education assessment practice in the era of generative AI tools
(2024) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2024.7.1.28 · Cited by 37 · see citing works
- Authors
- Funk, P. F., Hoch, C. C., Knoedler, S., Knoedler, L., Cotofana, S., Sofo, G., Bashiri Dezfouli, A., Wollenberg, B., Guntinas-Lichius, O., & Alfertshofer, M.
- Affiliations
- Jena University Hospital (DE)
- TUM Klinikum (DE)
- Harvard University (US)
- Queen Mary University of London (GB)
- Pontifícia Universidade Católica do Rio de Janeiro (BR)
- Department of Otolaryngology, Head and Neck Surgery, School of Medicine and Health, Technical University of Munich (TUM), Ismaningerstrasse 22, 81675 Munich, Germany
- Department of Otorhinolaryngology, Head and Neck Surgery, University Hospital Jena, Friedrich Schiller University Jena, Am Klinikum 1, 07747 Jena, Germany
- Corresponding author
- Funk, Paul F.
- ORCID
- Venue
- European Journal of Investigation in Health, Psychology and Education
- Publisher
- MDPI AG
- Published
- 2024-03-08
- Volume
- 14
- Issue
- 3
- Pages
- 657-668
- ISSN
- 2174-8144 (ISSN-L)
- 2254-9625
- PMC
- PMC10969490
- OpenAlex
- W4392599487
- Cited by
- 69
- References
- 35
- Citations by year
- 2026: 27 · 2025: 31 · 2024: 11
- FWCI
- 6.45
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- European Journal of Investigation in Health Psychology and Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Dialnet (Universidad de la Rioja) (submittedVersion, other-oa)
- Common Library Network (Der Gemeinsame Bibliotheksverbund) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, cc-by)
- Funders
- German Research Foundation Projekt-Nr. (512648189)
- Open Access Publication Fund of the Thueringer Universitaets—und Landesbibliothek Jena (512648189)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- ChatGPT, response consistency, AI accuracy, ChatGPT-4, ChatGPT-3.5, large language models, medical education, clinical decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
