CoChat
Sign Up
Cited by 40Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareArtificial Intelligence in Education

The Scientific Knowledge of Bard and ChatGPT in Endocrinology, Diabetes, and Diabetes Technology: Multiple-Choice Questions Examination-Based Performance

Open access
Meo, Sultan Ayoub, Al-Khlaiwi, Thamir, AbuKhalaf, Abdulelah Adnan, Meo, Anusha Sultan, Klonoff, David C.·2025·Journal of Diabetes Science and Technology
DOI 10.1177/19322968231203987PMC PMC12035228
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: The present study aimed to investigate the knowledge level of Bard and ChatGPT in the areas of endocrinology, diabetes, and diabetes technology through a multiple-choice question (MCQ) examination format. Methods: Initially, a 100-MCQ bank was established based on MCQs in endocrinology, diabetes, and diabetes technology. The MCQs were created from physiology, medical textbooks, and academic examination pools in the areas of endocrinology, diabetes, and diabetes technology and academic examination pools. The study team members analyzed the MCQ contents to ensure that they were related to the endocrinology, diabetes, and diabetes technology. The number of MCQs from endocrinology was 50, and that from diabetes and science technology was also 50. The knowledge level of Google’s Bard and ChatGPT was assessed with an MCQ-based examination. Results: In the endocrinology examination section, ChatGPT obtained 29 marks (correct responses) of 50 (58%), and Bard obtained a similar score of 29 of 50 (58%). However, in the diabetes technology examination section, ChatGPT obtained 23 marks of 50 (46%), and Bard obtained 20 marks of 50 (40%). Overall, in the entire three-part examination, ChatGPT obtained 52 marks of 100 (52%), and Bard obtained 49 marks of 100 (49%). ChatGPT obtained slightly more marks than Bard. However, both ChatGPT and Bard did not achieve satisfactory scores in endocrinology or diabetes/technology of at least 60%. Conclusions: The overall MCQ-based performance of ChatGPT was slightly better than that of Google’s Bard. However, both ChatGPT and Bard did not achieve appropriate scores in endocrinology and diabetes/diabetes technology. The study indicates that Bard and ChatGPT have the potential to facilitate medical students and faculty in academic medical education settings, but both artificial intelligence tools need more updated information in the fields of endocrinology, diabetes, and diabetes technology.

AI snapshot

Written from the paper’s full text.

TL;DR

In a 100-item MCQ examination, ChatGPT scored 52% overall and Bard scored 49%, with both falling below the 60% satisfactory/passing threshold used by the authors. ChatGPT slightly outperformed Bard overall, but neither reached satisfactory performance in endocrinology or diabetes/diabetes technology.

Key findings

  • Endocrinology: ChatGPT scored 29/50 (58%) and Bard scored 29/50 (58%).
  • Diabetes and diabetes technology: ChatGPT scored 23/50 (46%) and Bard scored 20/50 (40%).
  • Overall: ChatGPT scored 52/100 (52%) and Bard scored 49/100 (49%); both were below the 60% satisfactory score threshold.

Methods

  • Cross-sectional study conducted in July 2023 in the Department of Physiology, College of Medicine, King Saud University, and the Diabetes Research Institute, Mills-Peninsula Medical Center; no human or animal subjects were involved, so ethical approval was not required.
  • A 100-MCQ bank was created: 50 endocrinology MCQs and 50 diabetes/diabetes technology MCQs, drawn from textbooks and university examination pools; each question was scenario-based with four substems and one correct answer, and image-based questions were excluded.
  • Questions were manually entered one by one into Bard and ChatGPT, with a fresh session for each entry; the initial response was scored 0 or 1 against a predetermined answer key, and results were reported as numbers and percentages.

Limitations

  • Stated: the study used a small set of MCQs, did not assess knowledge in real-world student settings, and used scenario-based MCQs without images.
  • Evident: the study used only the initial response per question and did not use “regenerate response,” so response variability across repeated attempts was not assessed.
  • Evident: no human comparison group was included, so the chatbots’ performance was not directly compared with that of students or clinicians.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1177/19322968231203987

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 10 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    There is a notable scarcity of literature that evaluates the proficiency of Google's Bard and ChatGPT specifically in the field of endocrinology, diabetes, and diabetes technology.

    “Nevertheless, there is a notable scarcity of literature that evaluates the proficiency of Google's Bard and ChatGPT specifically in the field of endocrinology, diabetes, and diabetes technology.7,8”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [7] source

      The Author note discusses ChatGPT and its use to write a perspective on AI in medicine. Searches for Bard, endocrinology, diabetes technology, and studies evaluating chatbot proficiency found no passage addressing the claimed scarcity of literature. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [8] source

      The Introduction’s healthcare section discusses general medical uses of ChatGPT, including answering questions and generating clinical vignettes. Our copy does not assess the literature on Bard and ChatGPT proficiency in endocrinology, diabetes, or diabetes technology; searches for those topics found no relevant passage. No page numbers were provided. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Findingown data

    In the diabetes and diabetes technology section, Bard obtained 20 correct responses out of 50 (40%).

    “However, in the diabetes and diabetes technology sections, ChatGPT obtained 23 marks of 50 (46%) and Bard obtained 20 marks of 50 (40%).”
    Quote in the paperSubstantial claim
  • Findingown data

    In the diabetes and diabetes technology section, ChatGPT obtained 23 correct responses out of 50 (46%).

    “However, in the diabetes and diabetes technology sections, ChatGPT obtained 23 marks of 50 (46%) and Bard obtained 20 marks of 50 (40%).”
    Quote in the paperSubstantial claim
  • Findingown data

    In the endocrinology section, ChatGPT obtained 29 correct responses out of 50 (58%).

    “In the endocrinology examination section, ChatGPT obtained 29 marks (correct responses) of 50 (58%) and Bard obtained a similar mark of 29 marks of 50 (58%).”
    Quote in the paperSubstantial claim
  • Findingown data

    Overall, in the entire three-part examination, ChatGPT obtained 52 correct responses out of 100 (52%).

    “However, with the whole score in the entire set of MCQs, ChatGPT obtained 52 marks of 100 (52%) and Bard obtained 49 marks of 100 (49%).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 17
  • May combine two claims. 7

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The most recent literature highlights the role of ChatGPT in basic and clinical medical sciences and its use in diabetes technology.

    “The most recent literature highlights the role of ChatGPT in basic and clinical medical sciences, 9 and its use in diabetes technology.10”
    Quote in the paperNot a substantial claim2 of 2 sources checked

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    Google's Bard and ChatGPT provided clear and consistent explanations for their answers, demonstrating a satisfactory degree of understanding.

    “Finally, Google's Bard and ChatGPT provided clear and consistent explanations for their answers, demonstrating a satisfactory degree of understanding.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    AI still needs more updates in knowledge to achieve a passing score and reach the level of human knowledge.

    “It shows that AI still needs more updates in knowledge to achieve a passing score and reach the level of human knowledge.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The performance of Bard and ChatGPT on the MCQ examination reflects their scientific knowledge in endocrinology, diabetes, and diabetes technology.
    • Rests on an assumption: The passing scores for human licensing examinations (USMLE, MCCQE) are appropriate benchmarks for evaluating AI chatbots' performance.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    A fresh Google's Bard and ChatGPT session was started for each MCQ entry to avoid memory retention bias.

    “The questions were entered manually one by one, and a fresh Google's Bard and ChatGPT session was started for each entry to avoid memory retention bias.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The knowledge level was not assessed in real-world students' settings.

    “The study is based on a small size set of MCQs, and the knowledge level was not in real-world students' settings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The MCQs were based on only scenarios without images.

    “The MCQs were based on only scenarios without images in the MCQs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The initial response obtained was used as the final response, and the option "regenerate response" was not used.

    “The initial response obtained was used as the final response, and we did not use the option "regenerate response."”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    To the authors' knowledge, this is the first study of its kind to compare these two chatbots head-to-head in the field of knowledge about facts in endocrinology, diabetes, and diabetes technology.

    “To our knowledge, this is the first study of its kind to compare these two chatbots head-to-head in the field of knowledge about facts in the areas of endocrinology, diabetes, and diabetes technology to evaluate and compare the knowledge of two different AI tools, ChatGPT and Bard.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Test questions with visual components, such as clinical images, graphs, and illustrations, were removed from the study.

    “In addition, we removed any test questions that had visual components, such as clinical images, graphs, and illustrations from the study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The USMLE program provides a recommended pass or fail outcome on all step examinations.

    “The USMLE program provides a recommended pass or fail outcome on all step examinations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Ethical permission was not required because the study did not involve any animal or human subjects.

    “The MCQs were gathered and constructed using databases and textbooks, and this study did not involve any animal or human subjects, hence, ethical permission was not required.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study is based on a small size set of MCQs.

    “The study is based on a small size set of MCQs, and the knowledge level was not in real-world students' settings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    These chatbots will need to analyze more data from various sources, such as medical literature, textbooks, websites, databases, and social media, to be able to provide sufficient performance as necessary to pass a licensing examination.

    “however, these chatbots will need to analyze more data from various sources, such as the medical literature, medical textbooks, websites, databases, and social media to be able to provide sufficient performance as necessary to pass a licensing examination.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The performance of Bard and ChatGPT on the MCQ examination reflects their scientific knowledge in endocrinology, diabetes, and diabetes technology.
    • Rests on an assumption: The passing scores for human licensing examinations (USMLE, MCCQE) are appropriate benchmarks for evaluating AI chatbots' performance.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Study design and conductnone given

    The knowledge evaluation at the same time on two different AI tools provides more validity and reliability of the examination contents as well as AI tools.

    “The knowledge evaluation at the same time on two different AI tools provides more validity and reliability of the examination contents as well as AI tools.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    MCQs provide a robust framework for assessing higher cognitive functions by promoting critical thinking and the application of knowledge.

    “Moreover, MCQs provide a robust framework for assessing higher cognitive functions, by promoting critical thinking and the application of knowledge.12,13”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [12] source
    • cites [13] source

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The study team analyzed the MCQ contents to ensure they were related to endocrinology, diabetes, and diabetes technology, and the MCQs and answer key were carefully checked and assured relevant to the subject contents.

    “The study team members analyzed the MCQ contents to ensure that they were related to the endocrinology, diabetes, and diabetes technology.”
    • Also stated in: § Guyton and Hall's Text of Medical Physiology; First Aid USMLE Step 1; First Aid USMLE 2; AMBOSS Step 1; Kumar and Clark's Clinical Medicine; and university examination pools. · plain
      “The MCQs and answer key were carefully checked, and assured that all the questions were relevant to the subject contents.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    MCQs were carefully checked to ensure that no answers, related content, or explanations could be indexed on search engines.

    “MCQs were carefully checked to ensure that no answers, related content, or explanations could be indexed on search engines.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    Multiple-choice questions provide a broad range of assessments of critical thinking and provide an ample evaluation of higher cognitive functions.

    “Multiple-choice questions provide a broad range of assessments of critical thinking and provide an ample evaluation of higher cognitive functions.12,13”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [12] source
    • cites [13] source
18 of 18

  • The paper relies on this without support.

    The performance of Bard and ChatGPT on the MCQ examination reflects their scientific knowledge in endocrinology, diabetes, and diabetes technology.

    “The present study aimed to investigate the knowledge level of Bard and ChatGPT in the areas of endocrinology, diabetes, and diabetes technology through a multiple-choice question (MCQ) examination format.”
    “The knowledge level of Google's Bard and ChatGPT was assessed with an MCQ-based examination.”

  • The paper relies on this without support.

    The passing scores for human licensing examinations (USMLE, MCCQE) are appropriate benchmarks for evaluating AI chatbots' performance.

    “For a better understanding of the scientific knowledge of Bard and ChatGPT, it is vital to know the passing scores in international examinations, such as the United States Medical Licensing Examination (USMLE) conducted for worldwide physicians.”
    “It shows that AI still needs more updates in knowledge to achieve a passing score and reach the level of human knowledge.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Several studies examined the performance of LLMs on various medical exams; comparisons between older and newer versions indicated that newer versions exhibited better accuracy; one study comparing ChatGPT and Google's Bard in answering multiple-choice questionnaires related to endocrinology and diabetes found ChatGPT to be slightly superior, but both models lacked up-to-date information; another

    “Several studies examined the performance of LLMs on various medical exams [97–99]. Comparisons between older and newer versions indicated that newer versions exhibited better accuracy, highlighting the need for ongoing assessment of AI tools in medical education [100–102]. One study comparing ChatGPT and Google’s Bard in answering multiple-choice questionnaires (MCQs) related to endocrinology and”

    From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [103]

    Split into 3 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conclusion calls the scores satisfactory despite the stated threshold

    Major

    The conclusion says both chatbots scored satisfactorily, but the abstract says neither reached the study's 60% threshold for a satisfactory score.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. USMLE. Scoring & score reporting. https://www.usmle.org/bulletin-information/scoring-and-score-reporting. Accessed July 27 2023.

Cited by

Sorted by citations

Related

Authors
Meo, S. A., Al-Khlaiwi, T., AbuKhalaf, A. A., Meo, A. S., & Klonoff, D. C.
Affiliations
  • University of Aberdeen (GB)
  • King Saud University (SA)
  • San Mateo Medical Center (US)
Corresponding author
  • Meo, Sultan Ayoub
  • Meo, Anusha Sultan
ORCID
Venue
Journal of Diabetes Science and Technology
Publisher
SAGE Publications
Published
2023-10-05
Volume
19
Issue
3
Pages
705-710
ISSN
1932-2968 (ISSN-L)
DOI
10.1177/19322968231203987
PMC
PMC12035228
OpenAlex
W4387393234
Semantic Scholar
458f8dc23e634d1be16947d3f6a864bd5a5c3af4
Cited by
40
References
19
Citations by year
2026: 4 · 2025: 15 · 2024: 20 · 2023: 1
FWCI
1.38
Type
journal article
Language
en
Open access
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
  • Journal of Diabetes Science and Technology (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion)
Funders
Deputyship for Research and Innovation, Ministry of Education, Saudi Arabia ((IFKSUOR3-51-2))
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
ChatGPT, Google Bard, large language models, artificial intelligence in medical education, endocrinology, diabetes mellitus, diabetes technology, multiple-choice questions, medical education
MeSH terms
Generative Artificial Intelligence, Diabetes Mellitus, Educational Measurement, Endocrinology, Humans

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-13.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-13.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review