The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index; 11 of the 11 sources its claims cite were checked for support.
When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease
Open accessAbstract
Background: Artificial intelligence (AI) is gaining traction in healthcare, especially for patients’ education. Inflammatory bowel diseases (IBD) require continuous engagement, yet the quality of online information accessed by patients is inconsistent. ChatGPT, a generative AI model, has shown promise in medical scenarios, but its role in IBD communication needs further evaluation. The objective of this study was to assess the quality of ChatGPT-4’s responses to common patient questions about IBD, compared to those provided by experienced IBD specialists. Methods: Twenty-five frequently asked questions were collected during routine IBD outpatient visits and categorized into five themes: pregnancy/breastfeeding, diet, vaccinations, lifestyle, and medical therapy/surgery. Each question was answered by ChatGPT-4 and by two expert gastroenterologists. Responses were anonymized and evaluated by 12 physicians (six IBD experts and six non-experts) using a 5-point Likert scale across four dimensions: accuracy, reliability, comprehensibility, and actionability. Evaluators also attempted to identify whether responses were AI- or human-generated. Results: ChatGPT-4 responses received significantly higher overall scores than those from human experts (mean 4.28 vs. 4.05; p < 0.001). The best-rated scenarios were medical therapy and surgery; the diet scenario consistently received lower scores. Only 33% of AI-generated responses were correctly identified as such, indicating strong similarity to human-written answers. Both expert and non-expert evaluators rated AI responses highly, though IBD specialists gave higher ratings overall. Conclusion: ChatGPT-4 generated high-quality, clear, and actionable responses to IBD-related patient questions, often outperforming human experts. Its outputs were frequently indistinguishable from those written by physicians, suggesting potential as a supportive tool for patient education. Nonetheless, further studies are needed to assess real-world application and ensure appropriate use in personalized clinical care.
Overview
CoChat Summary
TL;DR
ChatGPT-4 produced high-quality answers to common IBD patient questions that were often rated higher than answers from experienced IBD specialists and were frequently indistinguishable from human-written responses. The authors conclude it may be a useful supplementary patient-education tool, but real-world and patient-centered evaluation is still needed.
Key findings
- ChatGPT-4 responses received significantly higher overall ratings than human expert responses (mean 4.28 ± 0.82 vs. 4.05 ± 0.93; p < 0.001).
- Only 98 of 300 ChatGPT-generated answers (33%) were correctly identified as AI-generated; experts identified 34% and non-experts 31%.
- IBD experts gave significantly higher ratings than non-experts across all four dimensions (mean 4.32 ± 0.88 vs. 3.95 ± 0.88; p < 0.001).
- Medical therapy and surgery scenarios were best rated, while diet consistently received lower scores.
Methods
- Prospective study: 25 frequently asked IBD patient questions were collected during a one-month observational period in an IBD outpatient clinic (approximately 500 patients seen), then categorized into pregnancy/breastfeeding, diet, vaccinations, lifestyle, and medical therapy/surgery.
- Each question was answered by ChatGPT-4 and by two expert gastroenterologists; responses were anonymized and evaluated by 12 physicians (six IBD experts, six non-experts) using a 5-point Likert scale for accuracy, reliability, comprehensibility, and actionability. Evaluators also tried to identify whether responses were AI- or human-generated.
- Statistical analysis used SPSS version 29; normality was assessed with Shapiro–Wilk, and non-parametric tests (Friedman and Mann–Whitney U) were applied, with p < 0.05 considered significant.
Limitations
- The paper states: small sample size of physician evaluators; fixed prompt structure for AI responses; evaluation exclusively by physicians without patient input; only two experts generated the human responses; and the method does not capture the empathetic, interactive nature of doctor–patient communication.
- Evident from the text: Table 2 reports identical GPT-4 and human scores for several domains (e.g., accuracy 4.0 ± 1.14 for both; comprehensibility 4.5 ± 0.71 for both), which conflicts with the reported overall superiority of ChatGPT-4 and complicates interpretation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
8 Valid · 0 Not valid · 1 Not checked · 48 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our findings confirm the growing potential of generative AI tools in the field of patient education for IBD (Gravina et al., 2024).”
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Discussion
What the sources say
- Supportedcites Gravina et al., 2024 source
The Abstract describes ChatGPT’s potential to provide medical information in response to common questions from patients with IBD, and the Conclusion reports that some of its answers were scientifically reliable. The paper also stresses inaccuracies and the need for refinement; the citing sentence claims potential, not that the tool is ready for unrestricted use. (LLM, conf=0.89)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This likely reflects the inherent complexity and individual variability of nutritional counselling in IBD, an area where standardized information, no matter how well articulated, cannot fully replace individualized medical advice (Barberio et al., 2025).”
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Discussion
What the sources say
- Supportedcites Barberio et al., 2025 source
Section 4.1, “Personalized Nutrition Approaches” (page unavailable), says that variation among people with IBD may make standardized dietary interventions suboptimal and calls for tailored recommendations. The Conclusions (page unavailable) also describe nutritional management as complex and emphasize the need for nutrition specialists. The explanation for the citing paper’s lower diet ratings is its own inference, not a finding attributed to Barberio et al. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“that study used an earlier version of the model (GPT-3), which has notable limitations in terms of language understanding and medical reasoning compared to the more advanced GPT-4”
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Introduction
What the sources say
- Partly supportedcites Kung et al., 2023 sourceExperimental — accuracy not yet measured
The Introduction identifies the model studied as GPT3.5, not GPT-3. The Discussion describes limitations in its USMLE performance, but the paper does not compare its language understanding or medical reasoning with GPT-4. (LLM, conf=0.88)
- Partly supportedcites Harsha Nori et al., 2023 sourceExperimental — accuracy not yet measured
The Abstract (p. 1) and Conclusion (p. 22) report that GPT-4 outperforms GPT-3.5 on medical examinations and benchmarks, supporting a comparison of medical-task performance. The paper compares GPT-4 with GPT-3.5, not GPT-3, and does not establish a comparative limitation in general language understanding. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT-4 responses received significantly higher overall scores than those from human experts (mean 4.28 vs. 4.05; p < 0.001).”
- Also stated in: § Comparison between ChatGPT-4 and human responses · plain
“A statistically significant difference was observed in the overall evaluation of responses generated by ChatGPT-4 compared to those written by expert gastroenterologists, with ChatGPT responses receiving significantly higher ratings (mean 4.28 ± 0.82 vs. 4.05 ± 0.93; p < 0.001).”
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Comparison between ChatGPT-4 and human responses · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and lowest in Diet (mean 4.27 ± 0.85).”
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Results > Comparison between ChatGPT-4 and human responses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
26 more claims
Create a free account to see all 57 claims and every source verdict.
- Probably not a substantial claim. 12
- It may state two separate claims as one. 19
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Results > Comparison between ChatGPT-4 and human responses
“The highest accuracy was observed in the Medical Therapy and Surgery scenario (mean 4.40 ± 0.71)”
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Results > Comparison between ChatGPT-4 and human responses
“| Accuracy | 4.17 (±0.79) | 4.01 (±0.91) | 4.12 (±0.64) | 4.37 (±0.71) | 4.33 (±0.88) |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Discussion
“while experts were generally more critical, emphasizing accuracy and adherence to guidelines, non-experts tended to value clarity and comprehensibility, often assigning higher scores.”
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Results > Comparison between IBD experts and non-experts
“experts providing significantly higher scores across all four evaluated dimensions (mean 4.32 ± 0.88 vs. 3.95 ± 0.88; p < 0.001).”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“ChatGPT responses receiving significantly higher ratings (mean 4.28 ± 0.82 vs. 4.05 ± 0.93; p < 0.001)”
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Results > Comparison between ChatGPT-4 and human responses
Reported 4.05 · recomputed 4.25
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Results > Comparison between ChatGPT-4 and human responses
“Reliability peaked in both the Medical Therapy and Surgery and Vaccinations scenarios (mean 3.87 ± 0.97 and 3.80 ± 0.90, respectively), and was lowest in the Diet and Lifestyle scenarios (mean 3.59 ± 0.99 and 3.80 ± 0.90, respectively).”
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Results > Comparison between ChatGPT-4 and human responses
“| Reliability | 3.77 (±0.94) | 4.01 (±0.91) | 3.80 (±0.90) | 3.60 (±0.98) | 4.33 (±0.88) |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“ChatGPT responses receiving significantly higher ratings (mean 4.28 ± 0.82 vs. 4.05 ± 0.93; p < 0.001)”
§ When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease > Results > Comparison between ChatGPT-4 and human responses
Reported 4.28 · recomputed 4.125
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Aung, Y. Y. M., Wong, D. C. S., & Ting, D. S. W. (2021) · British Medical Bulletin · doi:10.1093/bmb/ldab016 · Cited by 511
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509
Disease Monitoring in Inflammatory Bowel Disease: Evolving Principles and Possibilities
Plevris, N., & Lees, C. W. (2022) · Gastroenterology · doi:10.1053/j.gastro.2022.01.024 · Cited by 168 · cited 1×
“Inflammatory bowel diseases (IBD) are chronic and often disabling conditions that require continuous patient engagement, education, and support (Monteleone et al., 2023; Plevris and Lees, 2022).”
Immune-mediated inflammatory diseases: common and different pathogenic and clinical features
Monteleone, G., Moscardelli, A., Colella, A., Marafini, I., & Salvatori, S. (2023) · Autoimmunity Reviews · doi:10.1016/j.autrev.2023.103410 · Cited by 105 · cited 1×
“Inflammatory bowel diseases (IBD) are chronic and often disabling conditions that require continuous patient engagement, education, and support (Monteleone et al., 2023; Plevris and Lees, 2022).”
Self-citationGravina, A. G., Pellegrino, R., Cipullo, M., Palladino, G., Imperio, G., Ventura, A., Auletta, S., Ciamarra, P., & Federico, A. (2024) · World Journal of Gastroenterology · doi:10.3748/wjg.v30.i1.17 · Cited by 56
Sciberras, M., Farrugia, Y., Gordon, H., Furfaro, F., Allocca, M., Torres, J., Arebi, N., Fiorino, G., Iacucci, M., Verstockt, B., et al. (2024) · Journal of Crohn's and Colitis · doi:10.1093/ecco-jcc/jjae040 · Cited by 45 · cited 2×
“5 in answering IBD-related questions based on the European Crohn’s and Colitis Organisation (ECCO) guidelines (Sciberras et al., 2024).”
Self-citationBarberio, B., Bertin, L., Facchin, S., Bonazzi, E., Cusano, S., Romanelli, G., Pesenti, F. F., Cazzaniga, E., Palestini, P., Zingone, F., et al. (2025) · Nutrients · doi:10.3390/nu17111879 · Cited by 28
Exploring ChatGPT effectiveness in addressing direct patient queries on colorectal cancer screening
Maida, M., Mori, Y., Fuccio, L., Sferrazza, S., Vitello, A., Facciorusso, A., & Hassan, C. (2025) · Endoscopy International Open · doi:10.1055/a-2568-9416 · Cited by 7 · cited 1×
“Previous studies in the field of gastroenterology have demonstrated the model’s high level of accuracy in answering scenario-specific medical questions (Maida et al., 2025; Calabrese et al., 2025).”
Calabrese, G., Maselli, R., Maida, M., Barbaro, F., Morais, R., Nardone, O. M., Sinagra, E., Di Mitri, R., & Sferrazza, S. (2025) · iGIE · doi:10.1016/j.igie.2025.01.012 · Cited by 5
Cited by
Hughes, R., Bello, A. L., Liang, R. F.-H., Pugliano, C. L., Zammarchi, I., Ghosh, S., & Iacucci, M. (2026) · Chinese Medical Journal · doi:10.1097/cm9.0000000000004170 · Cited by 1
“[83] In more specific areas such as dietary management, chat-based AI also demonstrated reliable responses, highlighting its potential as a diet education tool in IBD populations.”
Metadata
- Authors
- De Cristofaro, E., Zorzi, F., Abreu, M., Colella, A., Blanco, G. D. V., Fiorino, G., Lolli, E., Noor, N., Lopetuso, L. R., Pioche, M., Grimaldi, J., Paoluzi, O. A., Roseira, J., Sena, G., Troncone, E., Calabrese, E., Monteleone, G., & Marafini, I.
- Affiliations
- Policlinico Tor Vergata (IT)
- University of Miami (US)
- University of Rome Tor Vergata (IT)
- Azienda Ospedaliera San Camillo-Forlanini (IT)
- University of Cambridge (GB)
- Agostino Gemelli University Polyclinic (IT)
- Hospices Civils de Lyon (FR)
- Algarve Biomedical Center (PT)
- Corresponding author
- Marafini, Irene
- ORCID
- Venue
- Frontiers in Artificial Intelligence
- Publisher
- Frontiers Media SA
- Published
- 2025-10-10
- Volume
- 8
- Pages
- 1678320-1678320
- ISSN
- 2624-8212 (ISSN-L)
- PMC
- PMC12549657
- OpenAlex
- W4415017401
- Cited by
- 1
- References
- 10
- Citations by year
- 2026: 1
- FWCI
- 0.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Artificial Intelligence (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Cineca Institutional Research Information System (Tor Vergata University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT-4, inflammatory bowel disease, patient education, generative AI, patient questions, AI-generated text detection, pregnancy and lactation, vaccination, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease. https://app.cochat.ai/papers/10.3389/frai.2025.1678320
- The paper
- De Cristofaro, E., Zorzi, F., Abreu, M., Colella, A., Blanco, G. D. V., Fiorino, G., Lolli, E., Noor, N., Lopetuso, L. R., Pioche, M., Grimaldi, J., Paoluzi, O. A., Roseira, J., Sena, G., Troncone, E., Calabrese, E., Monteleone, G., & Marafini, I. (2025). When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease. Frontiers in Artificial Intelligence. https://doi.org/10.3389/frai.2025.1678320
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.