Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index; 11 of the 11 sources its claims cite were checked for support.

When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease

Open access

Abstract

Background: Artificial intelligence (AI) is gaining traction in healthcare, especially for patients’ education. Inflammatory bowel diseases (IBD) require continuous engagement, yet the quality of online information accessed by patients is inconsistent. ChatGPT, a generative AI model, has shown promise in medical scenarios, but its role in IBD communication needs further evaluation. The objective of this study was to assess the quality of ChatGPT-4’s responses to common patient questions about IBD, compared to those provided by experienced IBD specialists. Methods: Twenty-five frequently asked questions were collected during routine IBD outpatient visits and categorized into five themes: pregnancy/breastfeeding, diet, vaccinations, lifestyle, and medical therapy/surgery. Each question was answered by ChatGPT-4 and by two expert gastroenterologists. Responses were anonymized and evaluated by 12 physicians (six IBD experts and six non-experts) using a 5-point Likert scale across four dimensions: accuracy, reliability, comprehensibility, and actionability. Evaluators also attempted to identify whether responses were AI- or human-generated. Results: ChatGPT-4 responses received significantly higher overall scores than those from human experts (mean 4.28 vs. 4.05; p < 0.001). The best-rated scenarios were medical therapy and surgery; the diet scenario consistently received lower scores. Only 33% of AI-generated responses were correctly identified as such, indicating strong similarity to human-written answers. Both expert and non-expert evaluators rated AI responses highly, though IBD specialists gave higher ratings overall. Conclusion: ChatGPT-4 generated high-quality, clear, and actionable responses to IBD-related patient questions, often outperforming human experts. Its outputs were frequently indistinguishable from those written by physicians, suggesting potential as a supportive tool for patient education. Nonetheless, further studies are needed to assess real-world application and ensure appropriate use in personalized clinical care.

Overview

CoChat Summary

TL;DR

ChatGPT-4 produced high-quality answers to common IBD patient questions that were often rated higher than answers from experienced IBD specialists and were frequently indistinguishable from human-written responses. The authors conclude it may be a useful supplementary patient-education tool, but real-world and patient-centered evaluation is still needed.

Key findings

  • ChatGPT-4 responses received significantly higher overall ratings than human expert responses (mean 4.28 ± 0.82 vs. 4.05 ± 0.93; p < 0.001).
  • Only 98 of 300 ChatGPT-generated answers (33%) were correctly identified as AI-generated; experts identified 34% and non-experts 31%.
  • IBD experts gave significantly higher ratings than non-experts across all four dimensions (mean 4.32 ± 0.88 vs. 3.95 ± 0.88; p < 0.001).
  • Medical therapy and surgery scenarios were best rated, while diet consistently received lower scores.

Methods

  • Prospective study: 25 frequently asked IBD patient questions were collected during a one-month observational period in an IBD outpatient clinic (approximately 500 patients seen), then categorized into pregnancy/breastfeeding, diet, vaccinations, lifestyle, and medical therapy/surgery.
  • Each question was answered by ChatGPT-4 and by two expert gastroenterologists; responses were anonymized and evaluated by 12 physicians (six IBD experts, six non-experts) using a 5-point Likert scale for accuracy, reliability, comprehensibility, and actionability. Evaluators also tried to identify whether responses were AI- or human-generated.
  • Statistical analysis used SPSS version 29; normality was assessed with Shapiro–Wilk, and non-parametric tests (Friedman and Mann–Whitney U) were applied, with p < 0.05 considered significant.

Limitations

  • The paper states: small sample size of physician evaluators; fixed prompt structure for AI responses; evaluation exclusively by physicians without patient input; only two experts generated the human responses; and the method does not capture the empathetic, interactive nature of doctor–patient communication.
  • Evident from the text: Table 2 reports identical GPT-4 and human scores for several domains (e.g., accuracy 4.0 ± 1.14 for both; comprehensibility 4.5 ± 0.71 for both), which conflicts with the reported overall superiority of ChatGPT-4 and complicates interpretation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/frai.2025.1678320

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Box plots illustrating the overall ratings of answers generated by humans and ChatGPT across five clinical scenarios: (A) pregnancy and breastfeeding, (B) diet, (C) vaccinations, (D) lifestyle, and (E) medical therapy and surgery. The complete set of questions and answers for each scenario is provided in Supplementary Table 1.
Figure 1 Box plots illustrating the overall ratings of answers generated by humans and ChatGPT across five clinical scenarios: (A) pregnancy and breastfeeding, (B) diet, (C) vaccinations, (D) lifestyle, and (E) medical therapy and surgery. The complete set of questions and answers for each scenario is provided in Supplementary Table 1.CC BY · PubMed Central
Figure 2 Expert versus non-experts. Mean score per question according to evaluator group (IBD experts vs. non-experts) across the four assessed domains: (A) accuracy, (B) actionability, (C) comprehensibility, and (D) reliability. Each bar represents the average rating for one of the 25 questions (Q1–Q25) on a 5-point Likert scale.
Figure 2 Expert versus non-experts. Mean score per question according to evaluator group (IBD experts vs. non-experts) across the four assessed domains: (A) accuracy, (B) actionability, (C) comprehensibility, and (D) reliability. Each bar represents the average rating for one of the 25 questions (Q1–Q25) on a 5-point Likert scale.CC BY · PubMed Central

Verification

Claims the paper makes

8 Valid · 0 Not valid · 1 Not checked · 48 Own results and interpretations

26 more claims

Create a free account to see all 57 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 12
  • It may state two separate claims as one. 19

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 6 · the rest aren’t loaded

1 more finding

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 10 references

Cited by

See all 1

Metadata

Authors
De Cristofaro, E., Zorzi, F., Abreu, M., Colella, A., Blanco, G. D. V., Fiorino, G., Lolli, E., Noor, N., Lopetuso, L. R., Pioche, M., Grimaldi, J., Paoluzi, O. A., Roseira, J., Sena, G., Troncone, E., Calabrese, E., Monteleone, G., & Marafini, I.
Affiliations
  • Policlinico Tor Vergata (IT)
  • University of Miami (US)
  • University of Rome Tor Vergata (IT)
  • Azienda Ospedaliera San Camillo-Forlanini (IT)
  • University of Cambridge (GB)
  • Agostino Gemelli University Polyclinic (IT)
  • Hospices Civils de Lyon (FR)
  • Algarve Biomedical Center (PT)
Corresponding author
Marafini, Irene
Venue
Frontiers in Artificial Intelligence
Published
2025-10-10
Volume
8
Pages
1678320-1678320
ISSN
2624-8212 (ISSN-L)
OpenAlex
W4415017401
Cited by
1
References
10
Citations by year
2026: 1
FWCI
0.24
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Artificial Intelligence (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • Cineca Institutional Research Information System (Tor Vergata University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
ChatGPT-4, inflammatory bowel disease, patient education, generative AI, patient questions, AI-generated text detection, pregnancy and lactation, vaccination, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease. https://app.cochat.ai/papers/10.3389/frai.2025.1678320
The paper
De Cristofaro, E., Zorzi, F., Abreu, M., Colella, A., Blanco, G. D. V., Fiorino, G., Lolli, E., Noor, N., Lopetuso, L. R., Pioche, M., Grimaldi, J., Paoluzi, O. A., Roseira, J., Sena, G., Troncone, E., Calabrese, E., Monteleone, G., & Marafini, I. (2025). When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease. Frontiers in Artificial Intelligence. https://doi.org/10.3389/frai.2025.1678320

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.