Cited by 4Artificial Intelligence in Healthcare and EducationHealth Literacy and Information AccessibilityArtificial Intelligence in Healthcare

ChatGPT for diabetes education: potential, accuracy, and accessibility in patient support

Open access
Abuzar, Mohammad, Rai, Sandeep, Pathan, Tabreskhan, Fakki, Foorkan, Syed, Shariq·2026·Frontiers in Digital Health

Abstract

Background: Diabetes mellitus is a chronic metabolic disease with rising global prevalence. Adequate patient education is essential to encourage self-management and reduce complications. Artificial intelligence applications such as ChatGPT have emerged as potential supplementary resources for patient education alongside the broader integration of technology in healthcare. Methods: A cross-sectional evaluation was conducted using ten frequently asked questions (FAQs) on diabetes, selected from the Diabetic Association of India and the International Diabetes Federation. ChatGPT-4o (accessed via the web interface in March 2025) generated responses to each question in separate, stand-alone chat sessions to simulate typical patient interactions. Five board-certified endocrinologists (diabetologists) with a mean clinical experience of ≥10 years independently evaluated the responses using a 4-point Likert scale across five domains: overall quality, content accuracy, clarity, relevance, and trustworthiness. Final domain scores were computed as the mean of all five raters’ scores. Readability was assessed using the Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL). All readability analyses apply exclusively to the English-language outputs generated in this study. Results: The mean FRES was 38.19 and the mean FKGL was 16.87, indicating a reading level appropriate for college-educated individuals and substantially above the recommended sixth-grade benchmark for patient health materials. Mean response length was 300 ± 100 words across the ten prompts. Expert ratings were generally high: aggregated mean scores (±SD) were 4.0 (±0.0) for content accuracy and overall quality, 3.98 (±0.10) for relevance, and 3.9 (±0.20) for clarity and trustworthiness. No clinically inaccurate statements were identified by the raters; however, the high scores and narrow score range indicate a potential ceiling effect that limits discrimination between responses. Raters expressed concern about linguistic complexity, which may impede comprehension among patients with limited health literacy. Conclusions: ChatGPT-4o generated generally accurate and relevant diabetes education content, suggesting potential as a supplementary tool in diabetes care. However, the high reading-level complexity, small evaluation scope (ten prompts, one model, one session), and English-only assessment limit the generalisability of these findings. AI-generated content should supplement, not replace, clinician-led education. Future work should address language simplification, multilingual evaluation, and longitudinal assessment of patient outcomes.

AI snapshot

Written from the paper’s full text.

TL;DR

ChatGPT-4o produced generally accurate and relevant answers to ten diabetes FAQs, with high expert ratings but readability far above recommended patient-education levels. The authors conclude it may supplement, not replace, clinician-led diabetes education, while noting small scope, single-run generation, and English-only assessment.

Key findings

  • Five board-certified endocrinologists rated ChatGPT-4o responses highly across five domains: aggregated mean ± SD was 4.0 ± 0.0 for content accuracy and overall quality, 3.98 ± 0.06 for relevance, and 3.9 ± 0.1 for clarity and trustworthiness; mean across all domains was 3.96 ± 0.04.
  • No clinically inaccurate statements meeting the pre-specified error definition were identified by raters, but the narrow score range and proximity to the scale ceiling indicated a potential ceiling effect limiting discrimination between responses.
  • Readability was consistently college-level and above the recommended sixth-grade benchmark: mean FRES 38.19 ± 8.42, FKGL 16.87 ± 2.31, SMOG 17.24 ± 1.98, and Gunning Fog 19.63 ± 2.54.
  • Mean response length was 312 ± 91 words (range 198–487 words) in Table 4; the abstract reports 300 ± 100 words across the ten prompts.

Methods

  • Cross-sectional evaluation of ten diabetes FAQs selected from the Diabetic Association of India and the International Diabetes Federation, covering symptoms, management, diet, complications, monitoring, prevention, medications, reversal, and insulin therapy.
  • ChatGPT-4o responses were generated via the web interface in March 2025 in separate, stand-alone chat sessions with web browsing and plugins disabled; no prompt engineering or response regeneration was used, and one response was generated per prompt.
  • Five board-certified endocrinologists with ≥10 years’ clinical experience independently rated all responses on a 4-point Likert scale across overall quality, content accuracy, clarity, relevance, and trustworthiness; readability was assessed with FRES, FKGL, SMOG, and Gunning Fog using textstat Python library v0.7.0.

Limitations

  • Stated: the evaluation was restricted to ten prompts, one model (ChatGPT-4o), and a single session; a single response was generated per prompt, so LLM output variability was not assessed.
  • Stated: only five raters were used, formal inter-rater reliability statistics were not computed, raters were not blinded to the AI-generated nature of the content, and no formal adjudication framework or external guideline reference-check was applied to error assessment.
  • Stated: all readability analyses were English-only and cannot be extrapolated to other languages; outputs may vary across sessions, model updates, or access configurations.
  • Evident from the text: the study assessed no patient comprehension or clinical outcomes, did not compare multiple LLMs, and its hypothesis-generating design limits generalisability beyond the evaluated prompt set.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fdgth.2026.1768843

Cites

24 of 25 references verified1 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 6 cited sources checked · 3 supported · 1 flagged · accuracy not yet measured

  • Findingown data

    Mean response length was 312 words (range 198-487) across the ten responses.

    “Mean response length was 312 words (range: 198–487 words).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Mean SMOG Index across the ten ChatGPT-4o responses was 17.24 (SD 1.98).

    “Mean (±SD) values were: FRES 38.19 (±8.42), FKGL 16.87 (±2.31), SMOG Index 17.24 (±1.98), and Gunning Fog Index 19.63 (±2.54).”
    Quote in the paperSubstantial claim
  • Findingown data

    Mean FKGL across the ten ChatGPT-4o responses was 16.87 (SD 2.31).

    “Mean (±SD) values were: FRES 38.19 (±8.42), FKGL 16.87 (±2.31), SMOG Index 17.24 (±1.98), and Gunning Fog Index 19.63 (±2.54).”
    Quote in the paperSubstantial claim
  • Findingown data

    Mean Gunning Fog Index across the ten ChatGPT-4o responses was 19.63 (SD 2.54).

    “Mean (±SD) values were: FRES 38.19 (±8.42), FKGL 16.87 (±2.31), SMOG Index 17.24 (±1.98), and Gunning Fog Index 19.63 (±2.54).”
    Quote in the paperSubstantial claim
  • Findingown data

    No clinically inaccurate statements meeting the pre-specified error definition were identified by the raters in any of the ten responses.

    “Raters did not identify any clinically inaccurate statements meeting the pre-specified error definition (factual inaccuracy, clinically unsafe recommendation, or significant omission of safety-critical information) in any of the ten responses”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The operational definition of an 'error' adequately captures clinically significant inaccuracies.
  • Classified as probably not a substantial claim. 21
  • May combine two claims. 8
  • The quote only partly states this claim. 2
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In musculoskeletal care, ChatGPT-generated guidance for patients with knee osteoarthritis was found to be more comprehensive than clinician-written advice, though more difficult to read.

    “In musculoskeletal care, ChatGPT-generated guidance for patients with knee osteoarthritis was found to be more comprehensive than clinician-written advice, though more difficult to read (17).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [17] Singh, 2025, Comparing artificial intelligence–generated and clinician-created personalized self-management guidance for patients with knee osteoarthritis: prospective observational study, J Med Internet Res source

      The Discussion’s Principal Findings support the comprehensiveness comparison, but contradict the readability claim: GPT-4 guidance had generally lower reading difficulty than clinician guidance. The Results’ Readability Evaluation likewise report better GPT-4 scores on three of four metrics; clinician-written material had a slight advantage only on the Coleman-Liau Index. Page numbers are not provided in our copy. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In dentistry, AI-generated materials were associated with improved patient understanding and reduced anxiety.

    “In dentistry, AI-generated materials were associated with improved patient understanding and reduced anxiety (20).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [20] Ozan, 2025, Evaluating the impact of AI-generated educational content on patient understanding and anxiety in endodontics and restorative dentistry, BMC Oral Health source
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A recent systematic review concluded that AI tools consistently improved the readability of patient education materials, while emphasising the continued need for accuracy oversight, particularly for complex topics.

    “A recent systematic review concluded that AI tools consistently improved the readability of patient education materials, while emphasising the continued need for accuracy oversight, particularly for complex topics (21).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [21] Raj, 2024, Can artificial intelligence improve patient educational material readability? A systematic review and narrative synthesis, Intern Med J source

      The passages returned for this DOI are headed “Functional outcomes of patients with ischaemic stroke with known atrial fibrillation not on therapeutic anticoagulation” (Discussion and Conclusions; pages unavailable), rather than the named AI-readability review. Searches for patient-education readability and accuracy oversight returned only stroke-related text, so our available copy cannot verify the claim. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A comparative study of ChatGPT, Gemini, and Copilot for cardiovascular imaging patient education found ChatGPT to perform best on accuracy and completeness.

    “A comparative study of ChatGPT, Gemini, and Copilot for cardiovascular imaging patient education found ChatGPT to perform best on accuracy and completeness (18).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [18] Gad, 2025, Evaluating the accuracy and reliability of AI chatbots in patient education on cardiovascular imaging: a comparative study of ChatGPT, Gemini, and Copilot, Egyptian J Radiol Nucl Med source
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In oncology, GPT-4-generated prostate cancer education materials scored favourably on understandability and reliability metrics.

    “In oncology, GPT-4-generated prostate cancer education materials scored favourably on understandability and reliability metrics (19).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [19] Gibson, 2024, Evaluating the efficacy of ChatGPT as a patient education tool in prostate cancer: multimetric assessment, J Med Internet Res source
  • The quote only partly states this claim.

    Findingown data

    ChatGPT-4o generated generally accurate and relevant diabetes education content across the ten FAQs, as assessed by five endocrinologists.

    “ChatGPT-4o generated generally accurate and relevant diabetes education content”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 4-point Likert scale across the five domains is a valid and sensitive instrument for assessing the quality of AI-generated patient education content.
    • Rests on an assumption: The five board-certified endocrinologists are representative of expert opinion on diabetes patient education.
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Mean clarity score across the ten responses was 3.9 ± 0.1.

    “Mean ± SD | 4.0 ± 0.0 | 4.0 ± 0.0 | 3.9 ± 0.1 | 3.98 ± 0.06 | 3.9 ± 0.1 | 3.96 ± 0.04”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Mean relevance score across the ten responses was 3.98 ± 0.06.

    “Mean ± SD | 4.0 ± 0.0 | 4.0 ± 0.0 | 3.9 ± 0.1 | 3.98 ± 0.06 | 3.9 ± 0.1 | 3.96 ± 0.04”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Findingown data

    Raters expressed concern about linguistic complexity of the responses.

    “Raters expressed concern about linguistic complexity, which may impede comprehension among patients with limited health literacy.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The five board-certified endocrinologists are representative of expert opinion on diabetes patient education.
  • Classified as probably not a substantial claim.

    Findingown data

    Mean overall quality score across the ten responses was 4.0 ± 0.0.

    “Mean ± SD | 4.0 ± 0.0 | 4.0 ± 0.0 | 3.9 ± 0.1 | 3.98 ± 0.06 | 3.9 ± 0.1 | 3.96 ± 0.04”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Collectively, the prior findings suggest promise for LLMs in health communication, while underscoring the necessity of clinician oversight and validation.

    “Collectively, these findings suggest promise for LLMs in health communication, while underscoring the necessity of clinician oversight and validation.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Study design and conductown data

    Five board-certified endocrinologists independently evaluated all ten responses.

    “Five board-certified endocrinologists (diabetologists) with a minimum of ten years of clinical experience in diabetes management were recruited to evaluate the ChatGPT-4o responses.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Prompt engineering and prompt customisation were deliberately excluded to simulate a non-expert user interaction.

    “Prompt engineering and prompt customisation were deliberately excluded so that outputs would reflect the experience of a non-expert user interacting with the model in a naturalistic manner”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Each prompt was entered as a new, stand-alone chat session with no prior conversation history to avoid contextual carryover.

    “each prompt was entered as a new, stand-alone chat session with no prior conversation history”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Raters were not blinded to the AI-generated nature of the content, which may have introduced bias.

    “Raters were informed that the content was AI-generated; blinding to the AI-generated nature of the content was not implemented, and this is acknowledged as a potential source of bias.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    All readability analyses apply exclusively to the English-language outputs generated in this study.

    “All readability analyses in this study therefore apply exclusively to the English-language outputs generated by ChatGPT-4o.”
    Quote in the paperSubstantial claim
  • May combine two claims. The quote only partly states this claim.

    Interpretationown data

    The high reading-level complexity, small evaluation scope, single-run generation, absence of inter-rater reliability analysis, and English-only assessment limit the generalisability of the findings.

    “The high reading-level complexity, small evaluation scope (ten prompts, one model, one session), and English-only assessment limit the generalisability of these findings.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Web-browsing and plugin tools were disabled during the ChatGPT session to ensure responses reflected the model's parametric knowledge only.

    “Web-browsing and plugin tools were disabled during the session to ensure that responses reflected the model's parametric knowledge only, without real-time internet retrieval.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Ratings were conducted independently, without discussion between raters.

    “Ratings were conducted independently, without discussion between raters, to ensure independence of assessments.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    No formal adjudication framework was pre-specified; the absence of flagged errors reflects unanimous agreement across all five raters.

    “No formal adjudication framework (e.g., majority vote or consensus panel) was pre-specified; raters recorded independent assessments, and the absence of flagged errors reflects unanimous agreement across all five raters.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    No outputs were discarded or regenerated.

    “No outputs were discarded or regenerated.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Formal inter-rater reliability statistics were not computed.

    “formal inter-rater reliability statistics (e.g., intraclass correlation coefficient or Kendall's W) were not computed”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The study's findings are hypothesis-generating and should not be generalised beyond the evaluated prompt set.

    “These findings are hypothesis-generating and should not be generalised beyond the evaluated prompt set.”
    Quote in the paperSubstantial claim
23 of 23
  • The paper relies on this without support.

    The ten FAQs selected from the Diabetic Association of India and the International Diabetes Federation represent the range of educational needs most relevant to diabetic patients and caregivers.

    “The questions were selected to represent the range of educational needs most relevant to diabetic patients and caregivers”
  • The paper relies on this without support.

    The 4-point Likert scale across the five domains is a valid and sensitive instrument for assessing the quality of AI-generated patient education content.

    “Each rater independently assessed all ten responses using a 4-point Likert scale (1 = lowest; 4 = highest) across five domains: overall quality, content accuracy, clarity, relevance, and trustworthiness”
  • The paper relies on this without support.

    The operational definition of an 'error' adequately captures clinically significant inaccuracies.

    “An “error” was operationally defined as: (a) a factual inaccuracy, (b) a clinically unsafe recommendation, or (c) a significant omission of information critical to patient safety.”
  • The paper relies on this without support.

    A reading level of approximately the sixth grade (FKGL ≤ 6; FRES ≥ 60–70) is the appropriate benchmark for patient education materials.

    “Patient education materials are generally recommended to target a reading level of approximately the sixth grade (FKGL ≤ 6; FRES ≥ 60–70).”
  • The paper relies on this without support.

    The five board-certified endocrinologists are representative of expert opinion on diabetes patient education.

    “Five board-certified endocrinologists (diabetologists) with a minimum of ten years of clinical experience in diabetes management were recruited to evaluate the ChatGPT-4o responses.”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Abstract and Table 3 give different clarity and trustworthiness SDs

    Minor

    The abstract reports an SD of 0.20 for both domains, whereas Table 3 reports 0.1 for both.

1 of 1

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Abuzar, M., Rai, S., Pathan, T., Fakki, F., & Syed, S.
Affiliations
  • Anjuman-I-Islam’s Kalsekar Technical Campus School of Pharmacy
  • Department of Internal Medicine, Incharge- Diabetes Speciality Centre, MGM Institute of Health Sciences
Corresponding author
Abuzar, Mohammad
Venue
Frontiers in Digital Health
Published
2026-04-13
Volume
8
Pages
1768843-1768843
ISSN
2673-253X (ISSN-L)
OpenAlex
W7154101056
Cited by
4
References
23
Citations by year
2026: 4
FWCI
7.57
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Digital Health (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
ChatGPT, diabetes mellitus, patient education, artificial intelligence in healthcare, GPT-4o, content accuracy, readability, health literacy, self-management, frequently asked questions, patient support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-13.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-13.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review