Analysing the Applicability of ChatGPT, Bard, and Bing to Generate Reasoning-Based Multiple-Choice Questions in Medical Physiology
Open accessAbstract
Background Artificial intelligence (AI) is evolving in the medical education system. ChatGPT, Google Bard, and Microsoft Bing are AI-based models that can solve problems in medical education. However, the applicability of AI to create reasoning-based multiple-choice questions (MCQs) in the field of medical physiology is yet to be explored. Objective We aimed to assess and compare the applicability of ChatGPT, Bard, and Bing in generating reasoning-based MCQs for MBBS (Bachelor of Medicine, Bachelor of Surgery) undergraduate students on the subject of physiology. Methods The National Medical Commission of India has developed an 11-module physiology curriculum with various competencies. Two physiologists independently chose a competency from each module. The third physiologist prompted all three AIs to generate five MCQs for each chosen competency. The two physiologists who provided the competencies rated the MCQs generated by the AIs on a scale of 0-3 for validity, difficulty, and reasoning ability required to answer them. We analyzed the average of the two scores using the Kruskal-Wallis test to compare the distribution across the total and module-wise responses, followed by a post-hoc test for pairwise comparisons. We used Cohen's Kappa (Κ) to assess the agreement in scores between the two raters. We expressed the data as a median with an interquartile range. We determined their statistical significance by a p-value <0.05. Results ChatGPT and Bard generated 110 MCQs for the chosen competencies. However, Bing provided only 100 MCQs as it failed to generate them for two competencies. The validity of the MCQs was rated as 3 (3-3) for ChatGPT, 3 (1.5-3) for Bard, and 3 (1.5-3) for Bing, showing a significant difference (p<0.001) among the models. The difficulty of the MCQs was rated as 1 (0-1) for ChatGPT, 1 (1-2) for Bard, and 1 (1-2) for Bing, with a significant difference (p=0.006). The required reasoning ability to answer the MCQs was rated as 1 (1-2) for ChatGPT, 1 (1-2) for Bard, and 1 (1-2) for Bing, with no significant difference (p=0.235). K was ≥ 0.8 for all three parameters across all three AI models. Conclusion AI still needs to evolve to generate reasoning-based MCQs in medical physiology. ChatGPT, Bard, and Bing showed certain limitations. Bing generated significantly least valid MCQs, while ChatGPT generated significantly least difficult MCQs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Although LLMs can rapidly produce MCQs, the resulting questions often fall short in quality and difficulty.
“Overall, while LLMs can quickly generate MCQs, their quality and difficulty often do not meet expectations.”
From MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback (2025) · cites this paper as Agarwal et al. (2023)
The Discussion reports that ChatGPT produced the least difficult MCQs, none of the three models produced many questions requiring high-level reasoning, and Bard and Bing produced questions with quality shortcomings. The study examined MCQs in medical physiology; it did not measure how quickly the models generated them, so the speed claim is unsupported. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.01 · Cited by 199 · see citing works
Artificial Intelligence Applications in Health Care Practice: Scoping Review
Sharma, M., Savage, C., Nair, M., Larsson, I., Svedberg, P., & Nygren, J. M. (2022) · Journal of Medical Internet Research · doi:10.2196/40238 · Cited by 198 · see citing works
Das, D., Kumar, N., Longjam, L. A., Sinha, R., Deb Roy, A., Mondal, H., & Gupta, P. (2023) · Cureus · doi:10.7759/cureus.36034 · Cited by 116 · see citing works
Item analysis of multiple choice questions: A quality assurance test for an assessment tool
Kumar, D., Jaipurkar, R., Shekhar, A., Sikri, G., & Srinivas, V. (2021) · Medical Journal Armed Forces India · doi:10.1016/j.mjafi.2020.11.007 · Cited by 109 · see citing works
Flagged first, then by citations
Research works printed but not linked: 15
- [2]ChatGPT for future medical and dental research Cureus Fatani B 0 15 2023 PMC10165936 37168166
- [4]Introducing artificial intelligence training in medical education JMIR Med Educ Paranjape K Schinkel M Nannan Panday R Car J Nanayakkara P 0 5 2019 PMC6918207 31793895
- [5]Cognitive psychology-based artificial intelligence review Front Neurosci Zhao J Wu M Zhou L Wang X Jia J 1024316 16 2022 36278021 PMC9582153
- [6]Why physiology matters in medicine Physiology (Bethesda) Joyner MJ 72 75 26 2011 21487024
- [7]Pushing critical thinking skills with multiple-choice questions: Does Bloom's taxonomy work? Acad Med Zaidi NL Grob KL Monrad SM 856 859 93 2018 29215375
- [9]Applicability of ChatGPT in assisting to solve higher order problems in pathology Cureus Sinha RK Deb Roy A Kumar N Mondal H 0 15 2023 PMC10033699 36968864
- [10]Evaluating ChatGPT’s ability to solve higher-order questions on the competency-based medical education curriculum in medical biochemistry Cureus Ghosh A Bir A 0 15 2023 PMC10152308 37143631
- [11]Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models PLOS Digit Health Kung TH Cheatham M Medenilla A 0 2 2023 PMC9931230 36812645
- [12]How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment JMIR Med Educ Gilson A Safranek CW Huang T Socrates V Chi L Taylor RA Chartash D 0 9 2023 PMC9947764 36753318
- [16]Evaluating the performance of ChatGPT in medical physiology university examination of phase I MBBS Adv Physiol Educ Subramani M Jaleel I Krishna Mohan S 270 271 47 2023 36971685
- [17]ChatGPT in medical school: How successful is AI in progress testing? Med Educ Online Friederichs H Friederichs WJ März M 2220920 28 2023 37307503 PMC10262795
- [18]Assessing the accuracy and reliability of AI-generated medical responses: An evaluation of the Chat-GPT model Res Sq Johnson D Goodman R Patrinely J PREPRINT (Version 1) 2023
- [20]Finding the place of ChatGPT in medical education Acad Med van de Ridder JM Shoja MM Rajput V (Online Ahead of Print) 2023 37162206
- [21]Performance of ChatGPT, GPT-4, and Google Bard on a neurosurgery oral boards preparation question bank [PREPRINT] Neurosurgery Ali R Tang OY Connolly ID 2023 37306460
- [22]How AI responds to common lung cancer questions: ChatGPT vs Google Bard Radiology Rahsepar AA Tavakoli N Kim GH Hassani C Abtin F Bedayat A 0 307 2023 37310252
Cited by
Sorted by citations
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 543 · see citing works
Aljamaan, F., Temsah, M.-H., Altamimi, I., Al-Eyadhy, A., Jamal, A., Alhasan, K., Mesallam, T. A., Farahat, M., & Malki, K. H. (2024) · JMIR Medical Informatics · doi:10.2196/54345 · Cited by 166 · see citing works
“Such performance by Bing has been observed in another study that prompted ChatGPT, Bard, and Bing for multiple choice question generation, where Bing had a significant rate of generation failure compared to the other two [41].”
ResultLarge language models for generating medical examinations: systematic review
Artsi, Y., Sorin, V., Konen, E., Glicksberg, B. S., Nadkarni, G., & Klang, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05239-y · Cited by 128 · see citing works
Dhanvijay, A. K. D., Pinjar, M. J., Dhokane, N., Sorte, S. R., Kumari, A., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.42972 · Cited by 59 · see citing works
“There are various domains of medical education like solving higherorder problems, generating questions, and preparing content for PowerPoint slides where LLMs can help teachers and students [3].”
BackgroundMorjaria, L., Burns, L., Bracken, K., Levinson, A. J., Ngo, Q. N., Lee, M., & Sibbald, M. (2024) · International Medical Education · doi:10.3390/ime3010004 · Cited by 33 · see citing works
Lin, Y., Luo, Z., Ye, Z., Zhong, N., Zhao, L., Zhang, L., Li, X., Chen, Z., & Chen, Y. (2025) · JMIR Medical Education · doi:10.2196/71125 · Cited by 31 · see citing works
“In addition, the cutoff date for the training data means that GAI may lack knowledge of the latest research, leading to outdated or inaccurate recommendations [26,32,41,66,67,74,80,89,92,94,109, 129,136,141,144].”
Yao, Z., Parashar, A., Zhou, H., Jang, W. S., Ouyang, F., Yang, Z., & Yu, H. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.538 · Cited by 12 · see citing works
- Authors
- Agarwal, M., Sharma, P., & Goswami, A.
- Affiliations
- All India Institute of Medical Sciences (IN)
- Corresponding author
- Agarwal, Mayank
- Venue
- Cureus
- Published
- 2023-06-26
- Volume
- 15
- Issue
- 6
- Pages
- e40977-e40977
- ISSN
- 2168-8184 (ISSN-L)
- PMC
- PMC10372539
- OpenAlex
- W4382020836
- Cited by
- 72
- References
- 4
- Citations by year
- 2026: 6 · 2025: 26 · 2024: 27 · 2023: 13
- FWCI
- 2.88
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Cureus (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- ChatGPT, Google Bard, Bing, large language models, multiple-choice question generation, medical physiology, medical education, AI-generated questions, question validity, question difficulty, competency-based curriculum, undergraduate medical students, model comparison, AI-assisted learning, AI ethics in education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
