CoChat
Sign Up
Cited by 30Artificial Intelligence in Healthcare and EducationEthics and Social Impacts of AIArtificial Intelligence in Healthcare

Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review

Open access
Hua, Yining, Xia, Winna, Bates, David, Hartstein, George Luke, Kim, Hyungjin Tom, Li, Michael, Nelson, Benjamin W, Stromeyer IV, Charles +4 more·2025·JMIR AI
DOI 10.2196/69006PMC PMC12639340

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Health care chatbots are rapidly proliferating, while generative artificial intelligence (AI) outpaces existing evaluation standards. OBJECTIVE: We aimed to develop a structured, stakeholder-informed framework to standardize evaluation of health care chatbots. METHODS: PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses)-guided searches across multiple databases identified 266 records; 152 were screened, 21 full texts were assessed, and 11 frameworks were included. We extracted 356 questions (refined to 271 by deduplication and relevance review), mapped items to Coalition for Health AI constructs, and organized them with iterative input from clinicians, patients, developers, epidemiologists, and policymakers. RESULTS: We developed the Health Care AI Chatbot Evaluation Framework (HAICEF), a hierarchical framework with 3 priority domains (safety, privacy, and fairness; trustworthiness and usefulness; and design and operational effectiveness) and 18 second-level and 60 third-level constructs covering 271 questions. Emphasis includes data provenance and harm control; Health Insurance Portability and Accountability Act/General Data Protection Regulation-aligned privacy and security; bias management; and reliability, transparency, and workflow integration. Question distribution across domains is as follows: design and operational effectiveness, 40%; trustworthiness and usefulness, 39%; and safety, privacy and fairness, 21%. The framework accommodates both patient-facing and back-office use cases. CONCLUSIONS: HAICEF provides an adaptable scaffold for standardized evaluation and responsible implementation of health care chatbots. Planned next steps include prospective validation across settings and a Delphi consensus to extend accountability and accessibility assurances.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/69006

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Such vignette or benchmark studies are considered first-tier, foundational in recent evaluation frameworks such as the HAICEF.

    “such vignette or benchmark studies are considered first-tier, foundational in recent evaluation frameworks such as the HAICEF142,143.”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [143]

    The Discussion (Principal Findings; page unavailable) identifies safety, privacy, and fairness—not vignette or benchmark studies—as HAICEF’s foundation and step 1. It describes checking for bias as a foundational first step. The framework’s tiers are evaluation domains, not categories of study designs. (LLM, conf=0.78)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. [3]Katz U Cohen E Shachar E Somer J Fink A Morse E Shreiber B Wolf I GPT versus resident physicians — a benchmark based on official board scores NEJM AI 2024 04 25 1 5 1 8 10.1056/aidbp2300192 · DOI · Paper page
  2. [4]Meaney C Huang R Lu K Fischer A Leung F Kulasegaram K Tzanetos K Punnett A Comparing the performance of ChatGPT and GPT-4 versus a cohort of medical students on an official University of Toronto undergraduate medical education progress test MedRXiv Preprint posted online on September 14, 2023 10.1101/2023.09.14.23295571 · DOI
  3. [5]Kung TH Cheatham M Medenilla A Sillos C De Leon L Elepaño C Madriaga M Aggabao R Diaz-Candido G Maningo J Tseng V Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models PLOS Digit Health 2023 02 2 2 e0000198 10.1371/journal.pdig.0000198 36812645 PDIG-D-22-00371 36812645 PMC9931230 · DOI · Paper page
  4. [6]Singhal K Azizi S Tu T Mahdavi SS Wei J Chung HW Scales N Tanwani A Cole-Lewis H Pfohl S Payne P Seneviratne M Gamble P Kelly C Babiker A Schärli N Chowdhery A Mansfield P Demner-Fushman D Agüera Y Arcas B Webster D Corrado GS Matias Y Chou K Gottweis J Tomasev N Liu Y Rajkomar A Barral J Semturs C Karthikesalingam A Natarajan V Large language models encode clinical knowledge Nature 2023 08 620 7972 172 180 10.1038/s41586-023-06291-2 37438534 10.1038/s41586-023-06291-2 37438534 PMC10396962 · DOI · Paper page
  5. [8]Torous J Blease C Generative artificial intelligence in mental health care: potential benefits and current challenges World Psychiatry 2024 02 23 1 1 2 10.1002/wps.21148 38214643 38214643 PMC10785974 · DOI · Paper page
  6. [9]Ethics and governance of artificial intelligence for health: guidance on large multi-modal models World Health Organization 2025-08-18 https://www.who.int/publications/i/item/9789240084759 40518431
  7. [10]Blumenthal D Patel B The regulation of clinical artificial intelligence NEJM AI 2024 07 25 1 8 AIpc2400545 10.1056/aipc2400545 · DOI
  8. [11]Zhou H Liu F Gu B Zou X Huang J Wu J Li Y Chen S Zhou P Liu J Hua Y Mao C You C Wu X Zheng Y Clifton L Li Z Luo J Clifton D A survey of large language models in medicine: progress, application, and challenge ArXiv Preprint posted online on July 22, 2024 10.48550/arXiv.2311.05112 · DOI · Paper page
  9. [12]Denecke K Warren J How to evaluate health applications with conversational user interface? Stud Health Technol Inform 2020 06 16 270 976 980 10.3233/SHTI200307 32570527 SHTI200307 32570527 · DOI
  10. [13]Denecke K Framework for guiding the development of high-quality conversational agents in healthcare Healthcare (Basel) 2023 04 07 11 8 1061 10.3390/healthcare11081061 37107895 healthcare11081061 37107895 PMC10137907 · DOI
  11. [14]Denecke K May R Developing a technical-oriented taxonomy to define archetypes of conversational agents in health care: literature review and cluster analysis J Med Internet Res 2023 01 30 25 e41583 10.2196/41583 36716093 v25i1e41583 36716093 PMC9926340 · DOI
  12. [15]Liu C Zowghi D Peng G Kong S Information quality of conversational agents in healthcare Inf Dev 2023 05 10 41 3 1080 1102 10.1177/02666669231172434 · DOI
  13. [16]Martinengo L Lin X Jabir AI Kowatsch T Atun R Car J Tudor Car L Conversational agents in health care: expert interviews to inform the definition, classification, and conceptual framework J Med Internet Res 2023 11 01 25 e50767 10.2196/50767 37910153 v25i1e50767 37910153 PMC10652195 · DOI
  14. [17]Coghlan S Leins K Sheldrick S Cheong M Gooding P D'Alfonso S To chat or bot to chat: ethical issues with using chatbots in mental health Digit Health 2023 9 20552076231183542 10.1177/20552076231183542 37377565 10.1177_20552076231183542 37377565 PMC10291862 · DOI
  15. [18]Xue J Zhang B Zhao Y Zhang Q Zheng C Jiang J Li H Liu N Li Z Fu W Peng Y Logan J Zhang J Xiang X Evaluation of the current state of chatbots for digital health: scoping review J Med Internet Res 2023 12 19 25 e47217 10.2196/47217 38113097 v25i1e47217 38113097 PMC10762606 · DOI
  16. [19]Abbasian M Khatibi E Azimi I Oniani D Shakeri Hossein Abad Z Thieme A Sriram R Yang Z Wang Y Lin B Gevaert O Li L Jain R Rahmani AM Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI NPJ Digit Med 2024 03 29 7 1 82 10.1038/s41746-024-01074-z 38553625 10.1038/s41746-024-01074-z 38553625 PMC10980701 · DOI · Paper page
  17. [20]Shlobin NA Ward M Shah HA Brown ED Sciubba DM Langer D D'Amico RS Ethical incorporation of artificial intelligence into neurosurgery: a generative pretrained transformer chatbot-based, human-modified approach World Neurosurg 2024 07 187 e769 e791 10.1016/j.wneu.2024.04.165 38723944 S1878-8750(24)00738-1 38723944 · DOI
  18. [21]Ding H Simmich J Vaezipour A Andrews N Russell T Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review J Am Med Inform Assoc 2024 02 16 31 3 746 761 10.1093/jamia/ocad222 38070173 7467291 38070173 PMC10873847 · DOI
  19. [22]Nadarzynski T Knights N Husbands D Graham CA Llewellyn CD Buchanan T Montgomery I Ridge D Achieving health equity through conversational AI: a roadmap for design and implementation of inclusive chatbots in healthcare PLOS Digit Health 2024 05 3 5 e0000492 10.1371/journal.pdig.0000492 38696359 PDIG-D-23-00467 38696359 PMC11065243 · DOI
  20. [23]Henson P David G Albright K Torous J Deriving a practical framework for the evaluation of health apps Lancet Digit Health 2019 06 1 2 e52 e54 10.1016/s2589-7500(19)30013-5 33323229 · DOI
  21. [25]Wongvibulsin S Yan MJ Pahalyants V Murphy W Daneshjou R Rotemberg V Current state of dermatology mobile applications with artificial intelligence features JAMA Dermatol 2024 06 01 160 6 646 650 10.1001/jamadermatol.2024.0468 38452263 2815800 38452263 PMC10921342 · DOI · Paper page
  22. [26]Sharp G Torous J West ML Ethical challenges in AI approaches to eating disorders J Med Internet Res 2023 08 14 25 e50696 10.2196/50696 37578836 v25i1e50696 37578836 PMC10463082 · DOI
  23. [27]Flores L Kim S Young SD Addressing bias in artificial intelligence for public health surveillance J Med Ethics 2024 02 20 50 3 190 194 10.1136/jme-2022-108875 37130756 jme-2022-108875 37130756 · DOI
  24. [28]King M Harmful biases in artificial intelligence Lancet Psychiatry 2022 11 9 11 e48 10.1016/s2215-0366(22)00312-1 36244361 · DOI
  25. [29]De Freitas J Cohen IG The health risks of generative AI-based wellness apps Nat Med 2024 05 30 5 1269 1275 10.1038/s41591-024-02943-6 38684859 10.1038/s41591-024-02943-6 38684859 · DOI · Paper page
  26. [30]Hua Y Liu F Yang K Li Z Na H Sheu Y Zhou P Moran LV Ananiadou S Clifton DA Beam A Torous J Large language models in mental health care: a scoping review Curr Treat Options Psych 2025 07 25 12 1 27 10.1007/s40501-025-00363-y · DOI · Paper page
  27. [31]Chen S Kann BH Foote MB Aerts HJWL Savova GK Mak RH Bitterman DS Use of artificial intelligence chatbots for cancer treatment information JAMA Oncol 2023 10 01 9 10 1459 1462 10.1001/jamaoncol.2023.2954 37615976 2808731 37615976 PMC10450584 · DOI · Paper page
  28. [32]Maslow AH A theory of human motivation Psychol Rev 1943 07 50 4 370 396 10.1037/h0054346 · DOI · Paper page

Cited by

Sorted by citations

Authors
Hua, Y., Xia, W., Bates, D., Hartstein, G. L., Kim, H. T., Li, M., Nelson, B. W., Stromeyer IV, C., King, D., Suh, J., Zhou, L., & Torous, J.
Affiliations
  • Harvard University (US)
  • Beth Israel Deaconess Medical Center (US)
  • Brigham and Women's Hospital (US)
  • Thomas Jefferson University (US)
  • Brown University (US)
  • Microsoft (United States) (US)
Corresponding author
Torous, John
ORCID
Venue
JMIR AI
Publisher
JMIR Publications Inc.
Published
2025-11-07
Volume
4
Pages
e69006
ISSN
2817-1705 (ISSN-L)
DOI
10.2196/69006
PMC
PMC12639340
OpenAlex
W4413242629
Cited by
30
References
1
Citations by year
2026: 28 · 2025: 2
FWCI
1.77
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR AI (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • The Medicine Forum (submittedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
generative AI, AI evaluation framework, evaluation standards, data provenance, privacy and security, transparency, workflow integration, stakeholder feedback

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review