Evaluating AI-based breastfeeding chatbots: quality, readability, and reliability analysis
Open accessAbstract
Background: In recent years, expectant and breastfeeding mothers commonly use various breastfeeding-related social media applications and websites to seek breastfeeding-related information. At the same time, AI-based chatbots-such as ChatGPT, Gemini, and Copilot-have become increasingly prevalent on these platforms (or on dedicated websites), providing automated, user-oriented breastfeeding guidance. Aim: The goal of our study is to understand the relative performance of three AI-based chatbots: ChatGPT, Gemini, and Copilot, by evaluating the quality, reliability, readability, and similarity of the breastfeeding information they provide. Methods: Two researchers evaluated the information provided by three different AI-based breastfeeding chatbots: ChatGPT version 3.5, Gemini, and Copilot. A total of 50 frequently asked questions about breastfeeding were identified and used in the study, divided into two categories (Baby-Centered Questions and Mother-Centered Questions), and evaluated using five scoring criteria, including the Quality Information Provision for Patients (EQIP) scale, the Simple Measure of Gobbledygook (SMOG) scale, the Similarity Index (SI), the Modified Dependability Scoring System (mDISCERN), and the Global Quality Scale (GQS). Results: The evaluation of AI chatbots’ answers showed statistically significant differences across all criteria (p < 0.05). Copilot scored highest on the EQIP, SMOG, and SI scales, while Gemini excelled in mDISCERN and GQS evaluations. No significant difference was found between Copilot and Gemini for mDISCERN and GQS scores. All three chatbots demonstrated high reliability and quality, though their readability required university-level education. Notably, ChatGPT displayed high originality, while Copilot exhibited the greatest similarity in responses. Conclusion: AI chatbots provide reliable answers to breastfeeding questions, but the information can be hard to understand. While more reliable than other online sources, their accuracy and usability are still in question. Further research is necessary to facilitate the integration of advanced AI in healthcare.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Explicit instruction regarding AI use is ethically effective in enhancing professional ethics and critical thinking of students using AI-generated information.
“Conversely, the studies indicate that explicit instruction regarding AI use is ethically effective in enhancing the professional ethics and critical thinking of the students using AI generated information [20,21].”
From HOW AI TOOLS (LIKE CHATGPT/GEMINI) AFFECT CRITICAL THINKING IN NURSING STUDENTS. A SYSTEMATIC REVIEW (2026) · cites this paper as [21]
The Introduction and Discussion assess breastfeeding chatbots’ information quality, reliability, and readability, not instruction for students. Searches for explicit AI-use instruction, professional ethics, and students’ critical thinking found no passage addressing the claimed effect. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breastfeeding in the 21st century: epidemiology, mechanisms, and lifelong effect
Victora, C. G., Bahl, R., Barros, A. J. D., França, G. V. A., Horton, S., Krasevec, J., Murch, S., Sankar, M. J., Walker, N., & Rollins, N. C. (2016) · The Lancet · doi:10.1016/s0140-6736(15)01024-7 · Cited by 8,040 · see citing works
Why invest, and what it will take to improve breastfeeding practices?
Rollins, N. C., Bhandari, N., Hajeebhoy, N., Horton, S., Lutter, C. K., Martines, J. C., Piwoz, E. G., Richter, L. M., & Victora, C. G. (2016) · The Lancet · doi:10.1016/s0140-6736(15)01044-2 · Cited by 2,944 · cited 1× · see citing works
“The reasons for low breastfeeding rates are generally the socio-demographic characteristics of mothers and infants, their social environment, health problems and inadequate breastfeeding education received before and after birth [3].”
SMOG Grading - A New Readability Formula.
McLaughlin, G. H. (1969) · The Journal of Reading · Cited by 2,081 · see citing works
Bernard, A., Langille, M., Hughes, S., Rose, C., Leddin, D., & Veldhuyzen van Zanten, S. (2007) · The American Journal of Gastroenterology · doi:10.1111/j.1572-0241.2007.01325.x · Cited by 1,197 · see citing works
When and How Artificial Intelligence Augments Employee Creativity
Jia, N., Luo, X., Zheng, F., & Liao, C. (2023) · Academy of Management Journal · doi:10.5465/amj.2022.0426 · Cited by 820 · see citing works
Rao, A. S., Kim, J., Kamineni, M., Pang, M., Lie, W., Dreyer, K. J., & Succi, M. D. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.05.003 · Cited by 481 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · see citing works
Moult, B., Franck, L. S., & Brady, H. (2004) · Health Expectations · doi:10.1111/j.1369-7625.2004.00273.x · Cited by 334 · see citing works
Readability Assessment of Internet-Based Consumer Health Information
Walsh, T. M., & Volsko, T. A. (2008) · Respiratory Care · doi:10.4187/respcare.08531310 · Cited by 328 · see citing works
Májovský, M., Černý, M., Kasal, M., Komarc, M., & Netuka, D. (2023) · Journal of Medical Internet Research · doi:10.2196/46924 · Cited by 301 · see citing works
Joshi, G., Jain, A., Araveeti, S. R., Adhikari, S., Garg, H., & Bhandari, M. (2024) · Electronics · doi:10.3390/electronics13030498 · Cited by 294 · see citing works
Social media as social support in pregnancy and the postpartum
Baker, B., & Yang, I. H. (2018) · Sexual & Reproductive Healthcare · doi:10.1016/j.srhc.2018.05.003 · Cited by 265 · see citing works
Evaluating ChatGPT as an Adjunct for Radiologic Decision-Making
Rao, A., Kim, J., Kamineni, M., Pang, M., Lie, W., & Succi, M. D. (2023) · doi:10.1101/2023.02.02.23285399 · Cited by 257 · see citing works
Quality of patient health information on the Internet: reviewing a complex and
Fahy, E. T. (2014) · Australasian Medical Journal · doi:10.4066/amj.2014.1900 · Cited by 228 · see citing works
Are video sharing Web sites a useful source of information on hypertension?
Kumar, N., Pandey, A., Venkatraman, A., & Garg, N. (2014) · Journal of the American Society of Hypertension · doi:10.1016/j.jash.2014.05.001 · Cited by 191 · see citing works
Revilla‐León, M., Gómez‐Polo, M., Barmak, A. B., Inam, W., Kan, J. Y. K., Kois, J. C., & Akal, O. (2022) · Journal of Prosthetic Dentistry · doi:10.1016/j.prosdent.2022.01.026 · Cited by 176 · see citing works
Patient Similarity in Prediction Models Based on Health Data: A Scoping Review
Sharafoddini, A., Dubin, J. A., & Lee, J. (2017) · JMIR Medical Informatics · doi:10.2196/medinform.6730 · Cited by 122 · see citing works
A new era in Internet interventions: The advent of Chat-GPT and AI-assisted therapist guidance
Carlbring, P., Hadjistavropoulos, H. D., Kleiboer, A. M., & Andersson, G. (2023) · Internet Interventions · doi:10.1016/j.invent.2023.100621 · Cited by 120 · see citing works
Rokhshad, R., Zhang, P., Mohammad-Rahimi, H., Pitchika, V., Entezari, N., & Schwendicke, F. (2024) · E -journal of dentistry · doi:10.1016/j.jdent.2024.104938 · Cited by 109 · see citing works
Comparative Performance of ChatGPT and Bard in a Text-Based Radiology Knowledge Assessment
Patil, N. S., Huang, R. S., van der Pol, C. B., & Larocque, N. (2023) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231193716 · Cited by 101 · see citing works
Langille, M. G. I., Bernard, A., Rodgers, C., Hughes, S. F., Leddin, D., & van Zanten, S. V. (2010) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2009.12.024 · Cited by 79 · see citing works
Issaiy, M., Zarei, D., & Saghazadeh, A. (2023) · World Journal of Emergency Surgery · doi:10.1186/s13017-023-00527-2 · Cited by 78 · see citing works
The role of using ChatGPT AI in writing medical scientific articles
Benichou, L. (2023) · Journal of Stomatology, Oral and Maxillofacial Surgery · doi:10.1016/j.jormas.2023.101456 · Cited by 73 · see citing works
Super-forecasting the ‘technological singularity’ risks from artificial intelligence
Radanliev, P., Roure, D. D., Maple, C. R., & Ani, U. (2022) · Evolving Systems · doi:10.1007/s12530-022-09431-7 · Cited by 69 · see citing works
Yurdakurban, E., Topsakal, K. G., & Duran, G. S. (2023) · Journal of Stomatology Oral and Maxillofacial Surgery · doi:10.1016/j.jormas.2023.101757 · Cited by 63 · see citing works
Cangöl, E., & Şahin, N. H. (2014) · Zeynep Kamil Tıp Bülteni · doi:10.16948/zktb.80388 · Cited by 53 · see citing works
Breastfeeding mothers' use of technology while breastfeeding
Tomfohrde, O., & Reinke, J. S. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.07.057 · Cited by 50 · see citing works
YouTube as a source of medical information about epidural analgesia for labor pain
D'Souza, R. S., D’Souza, S., & Sharpe, E. (2020) · International Journal of Obstetric Anesthesia · doi:10.1016/j.ijoa.2020.11.005 · Cited by 43 · see citing works
Evaluation of Breastfeeding Web Sites for Patient Education
Dornan, B. A., & Oermann, M. H. (2005) · MCN The American Journal of Maternal/Child Nursing · doi:10.1097/00005721-200601000-00006 · Cited by 33 · see citing works
Social Media. A Literature Review
DUONG, C. T. P. (2020) · Journal of Media Research · doi:10.24193/jmr.38.7 · Cited by 33 · see citing works
Carlsson, T., & Axelssön, O. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6380 · Cited by 23 · see citing works
Mohandas, S., Rana, R., Sirwani, B., Kirubakaran, R., & Puthussery, S. (2023) · Nutrients · doi:10.3390/nu15040988 · Cited by 13 · see citing works
Review of online breastfeeding information for Aboriginal and Torres Strait Islander women
Hopkins, M., Meedya, S., Ivers, R. G., & Charlton, K. E. (2020) · Women and Birth · doi:10.1016/j.wombi.2020.06.012 · Cited by 9 · see citing works
Orbatu, D., Karaca, S. Y., Alaygut, D., & Karaca, I. O. (2021) · Breastfeeding Medicine · doi:10.1089/bfm.2020.0301 · Cited by 9 · see citing works
Azak, M., Yılmaz, B., & Şahin, N. H. (2023) · Maternal and Child Health Journal · doi:10.1007/s10995-023-03722-4 · Cited by 9 · see citing works
Kaçer, E. Ö., & Kacer, I. (2024) · PLoS ONE · doi:10.1371/journal.pone.0310508 · Cited by 5 · see citing works
Utility Evaluation of Information from YouTube on Breastfeeding for Preterm Babies
Jo, C.-K., Lee, S., & Kim, M.-J. (2019) · Neonatal Medicine · doi:10.5385/nm.2019.26.4.185 · Cited by 4 · see citing works
An analysis of breastfeeding print educational material.
Ak, V. (1997) · PubMed · Cited by 4 · see citing works
Addressing Chatbots as Artificial Intelligence Aids in Pediatric Pathology
Schukow, C. P., & Nguyễn, V. H. (2023) · Pediatric and Developmental Pathology · doi:10.1177/10935266231212340 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [10]Kacer EO, Ipekten F. Can ChatGPT provide quality information about fever in children? Journal of Paediatrics and Child Health. 2024. PMC11701195 39470298
Cited by
Sorted by citations
Aslan, İ. Ö., & Aslan, M. (2025) · Healthcare · doi:10.3390/healthcare13141756 · Cited by 11 · see citing works
Shao, X., Ruan, T., Ju, X., Sun, Y., & Cui, J. (2025) · Digital Health · doi:10.1177/20552076251367645 · Cited by 11 · see citing works
Özdemir, Ö. T., Kavan, M. Y., & Güven, Y. (2025) · BMC Oral Health · doi:10.1186/s12903-025-07298-z · Cited by 11 · see citing works
Apomediation, AI, and the illusion of autonomy: risks of misinformation in patient decision-making
Rojo, A. P., Fontaines-Ruíz, T., & Romero-Rodríguez, L. M. (2026) · Frontiers in Communication · doi:10.3389/fcomm.2025.1684370 · Cited by 3 · see citing works
HOW AI TOOLS (LIKE CHATGPT/GEMINI) AFFECT CRITICAL THINKING IN NURSING STUDENTS. A SYSTEMATIC REVIEW
Din, A. L. (2026) · Journal of Medical & Health Sciences Review · doi:10.65035/650ytk03 · Cited by 1 · see citing works
Golshan-Tafti, M., Bahrami, R., Jayervand, F., Yeganegi, M., Shiehbeiki, S., Zanbagh, L., Talebi, H., Shahbazi, A., Shiri, A., Hajizadeh, N., et al. (2026) · Egyptian Pediatric Association Gazette · doi:10.1186/s43054-026-00538-z · Cited by 1 · see citing works
Jijo, A., Jijo, M., Manoj, S., & Veerachamy, A. (2025) · Cureus · doi:10.7759/cureus.92434 · Cited by 1 · see citing works
TEKGÜNDÜZ, S. E., Gürol, A., & APAY, S. E. (2026) · Research Square · doi:10.21203/rs.3.rs-8558387/v1
Adamyan, L. V., Plavunov, I. N., & Владимировна, К. О. (2026) · Russian Journal of Human Reproduction · doi:10.17116/repro2026320216
Multidimensional Evaluation of AI-Generated Patient Information on Cone Beam–Computed Tomography
Öçbe, M., Akçay, H., Sır, E., Taş, Ç., & Erbalta, R. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-01976-2
Aljaafari, D., Aljumah, H. K., Alzuwayr, M. A., Al-Essa, Y. M., Alradhi, H. A., Alabdali, M. M., Almuslim, N. I., AlQarni, M., & Alesefir, W. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15103908
İnam, Ö., Kahvecioğlu, Y., & Okay, S. (2026) · Journal of Human Lactation · doi:10.1177/08903344261446830
SERHATLIOĞLU, S. G., & Yeşil, Y. (2026) · Breastfeeding Medicine · doi:10.1177/15568253261481438
Demirel, D., Ceviz, K., Keten, T., & Gökkaya, C. S. (2026) · Revista da Associação Médica Brasileira · doi:10.1590/1806-9282.20260541
Kkirgia, M., Magklara, E.-P., Zoupa, C., Volaki, P., Mougiou, V., Iliodromiti, Z., Boutsikou, Τ., Iacovidou, N. M., & Sokou, R. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15197513
- Authors
- Kacer, E. O.
- ORCID
- Kacer, Emine Ozdemir
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2025-03-17
- Volume
- 20
- Issue
- 3
- Pages
- e0319782
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC11913278
- OpenAlex
- W4408537572
- Cited by
- 14
- References
- 38
- Citations by year
- 2026: 10 · 2025: 4
- FWCI
- 3.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Social Sciences › Health › Social Media in Health Education
- Keywords
- AI chatbots, breastfeeding, ChatGPT, Gemini, Copilot, readability, information quality, reliability, frequently asked questions, Environmental Quality Incentives Program, smog, mDISCERN, Global Quality Scale, similarity index, online health information
- MeSH terms
- Generative Artificial Intelligence, Artificial Intelligence, Breast Feeding, Female, Humans, Mothers, Reproducibility of Results, Comprehension, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
