CoChat
Sign Up
Cited by 57Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareClinical Reasoning and Diagnostic Skills

Assessing ChatGPT’s capacity for clinical decision support in pediatrics: A comparative study with pediatricians using KIDMAP of Rasch analysis

Open access
Kao, Hsu-Ju, Chien, Tsair-Wei, Wang, Wen-Chung, Chou, Willy, Chow, Julie Chi·2023·Medicine
DOI 10.1097/md.0000000000034068PMC PMC10289633
Open this paper in CoChat — free account

Abstract

Background: The application of large language models in clinical decision support (CDS) is an area that warrants further investigation. ChatGPT, a prominent large language models developed by OpenAI, has shown promising performance across various domains. However, there is limited research evaluating its use specifically in pediatric clinical decision-making. This study aimed to assess ChatGPT’s potential as a CDS tool in pediatrics by evCDSaluating its performance on 8 common clinical symptom prompts. Study objectives were to answer the 2 research questions: the ChatGPT’s overall grade in a range from A (high) to E (low) compared to a normal sample and the difference in assessment of ChatGPT between 2 pediatricians. Methods: We compared ChatGPT’s responses to 8 items related to clinical symptoms commonly encountered by pediatricians. Two pediatricians independently assessed the answers provided by ChatGPT in an open-ended format. The scoring system ranged from 0 to 100, which was then transformed into 5 ordinal categories. We simulated 300 virtual students with a normal distribution to provide scores on items based on Rasch rating scale model and their difficulties in a range between −2 to 2.5 logits. Two visual presentations (Wright map and KIDMAP) were generated to answer the 2 research questions outlined in the objectives of the study. Results: The 2 pediatricians’ assessments indicated that ChatGPT’s overall performance corresponded to a grade of C in a range from A to E, with average scores of −0.89 logits and 0.90 logits (=log odds), respectively. The assessments revealed a significant difference in performance between the 2 pediatricians (P < .05), with scores of −0.89 (SE = 0.37) and 0.90 (SE = 0.41) in log odds units (logits in Rasch analysis). Conclusion: This study demonstrates the feasibility of utilizing ChatGPT as a CDS tool for patients presenting with common pediatric symptoms. The findings suggest that ChatGPT has the potential to enhance clinical workflow and aid in responsible clinical decision-making. Further exploration and refinement of ChatGPT’s capabilities in pediatric care can potentially contribute to improved healthcare outcomes and patient management.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1097/md.0000000000034068

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. [2]Biswas S. ChatGPT and the future of medical writing. Radiology. 2023; 307: e223312. 36728748
  2. [3]Curtis N; ChatGPT. To ChatGPT or not to ChatGPT? the impact of artificial intelligence on academic publishing. Pediatr Infect Dis J. 2023; 42: 275. 36757192
  3. [12]Gilson A Safranek CW Huang T. How does ChatGPT perform on the United States medical licensing examination? the implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023; 9: e45312. 36753318 PMC9947764
  4. [13]Jin Q Dhingra B Liu Z. PubMedQA: a dataset for biomedical research question answering. arXiv. Preprint posted online on September 13, 2019.
  5. [14]Rao A Kim J Kamineni M. Evaluating ChatGPT as an adjunct for radiologic decision-making. Preprint. medRxiv. 2023;2023.02.02.23285399.
  6. [15]Lai Y Feng M Deng J. Medication analysis and pharmaceutical care for a child with Kawasaki disease: a case report and review of the literature. Medicine (Baltimore). 2023; 102: e32488. 36607867 PMC9829272
  7. [16]Cai WJ Ding SG. Retrospective analysis of clinical characteristics and related influencing factors of Kawasaki disease. Medicine (Baltimore). 2022; 101: e32430. 36596080 PMC9803503
  8. [17]Choi J Chang S Kim E. Integrative treatment of herbal medicine with western medicine on coronary artery lesions in children with Kawasaki disease. Medicine (Baltimore). 2022; 101: e28802. 35363167 PMC9281920
  9. [18]Li C Du Y Wang H. Neonatal Kawasaki disease: case report and literature review. Medicine (Baltimore). 2021; 100: e24624. 33607798 PMC7899894
  10. [19]Zheng X Li Y Yue P. Diagnostic significance of circulating miRNAs in Kawasaki disease in China: current evidence based on a meta-analysis. Medicine (Baltimore). 2021; 100: e24174. 33578520 PMC7886432
  11. [20]Curtis N. Examples of ChatGPT responses (generated in Jan 2023). Available at: http://links.lww.com/INF/E931 [access date February 21, 2022].
  12. [21]Ha LA Yaneva V. Automatic question answering for medical MCQs: can it go further than information retrieval? In: Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019). 2019 Presented at: RANLP 2019; September 2–4, 2019; Varna, Bulgaria: 418– 22.
  13. [22]Puthenpura V Nadkarni S DiLuna M. Personality changes and staring spells in a 12-year-old child: a case report incorporating ChatGPT, a natural language processing tool driven by Artificial Intelligence (AI). Cureus. 2023; 15: e36408. 37090271 PMC10115215
  14. [23]Almazyad M Aljofan F Abouammoh NA. Enhancing expert panel discussions in pediatric palliative care: innovative scenario development and summarization with ChatGPT-4. Cureus. 2023; 15: e38249. 37122982 PMC10143975
  15. [24]Liu S Wright AP Patterson BL. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc. 2023: ocad072. PMC10280357 37087108
  16. [26]Andrich D. A rating formulation for ordered response categories. Psychometrika. 1978; 43: 561– 73.
  17. [27]Bond T Yan Z Heene M. Applying the Rasch Model: Fundamental Measurement in the Human Science. ISBN 9780367141424, Published July 20, 2020. New York, NY: Routledge.
  18. [28]Wright BD Stone MH. Best Test Design: Rasch Measurement. MESA Press; 1979.
  19. [29]Müller M. Item fit statistics for Rasch analysis: can we trust them? J Stat Distrib App. 2020; 7: 5.
  20. [30]Linacre JM. An all-purpose person fit statistic? Rasch Meas Trans. 1997; 11: 582– 3.
  21. [31]Smith EV. Detecting and evaluating the impact of multidimensionality using item fit statistics and principal component analysis of residuals. J Appl Meas. 2002; 3: 205– 31. 12011501
  22. [34]Wilson M. Some notes on the term: “Wright Map.”. Rasch Meas Trans. 2011; 25: 1331.
  23. [35]Yeh CH Chien TW Lin JJ. Comparing the similarity and differences in MeSH terms associated with spine-specific journals using the forest plot: a bibliometric analysis. Medicine (Baltimore). 2022; 101: e31441. 36343077 PMC9646558
  24. [36]Masters GN. Rasch KIDMAP - a history. Rasch Meas Trans. 1994; 8: 366.
  25. [37]Linacre JM. How to simulate Rasch data. Rasch Meas Trans. 2007; 21: 1125.
  26. [40]Yang TY Chien TW Lai FJ. Web-based skin cancer assessment and classification using machine learning and mobile computerized adaptive testing in a Rasch Model: development study. JMIR Med Inform. 2022; 10: e33006. 35262505 PMC9282670
  27. [41]Shao Y Nadkarni S Niu K. A note on the Newton–Raphson iteration method in the Rasch model. Rasch Meas Trans. 2022; 35: 1851– 6.
  28. [42]Shou Y Sellbom M Chen H-F. Fundamentals of measurement in clinical psychology. In: Asmundson GJG, ed. Comprehensive Clinical Psychology. Amsterdam, The Netherlands: Elsevier; 2022: 13– 35.
  29. [46]Linacre JM. Optimizing rating scale category effectiveness. J Appl Meas. 2002; 3: 85– 106. 11997586
  30. [53]Kung TH Cheatham M Medenilla A. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023; 2: e0000198. 36812645 PMC9931230
  31. [56]Arif TB Munaf U Ul-Haque I. The future of medical education and research: is ChatGPT a blessing or blight in disguise? Med Educ Online. 2023; 28: 2181052. 36809073 PMC9946299
  32. [57]Wu HM Shao Y Chien TW. Student’s performance is shown on Google Maps using online Rasch analysis. J Appl Meas. 2020; 21: 1– 10.32129766
  33. [64]Wu M Tam HP Hen TH. Educational Measurement for Applied Researchers. Springer; 2017.

Cited by

Sorted by citations

Authors
Kao, H.-J., Chien, T.-W., Wang, W.-C., Chou, W., & Chow, J. C.
Affiliations
  • Chi Mei Medical Center (TW)
  • Education University of Hong Kong (HK)
  • Chung Shan Medical University Hospital (TW)
Corresponding author
Chow, Julie Chi
ORCID
Venue
Medicine
Publisher
Ovid Technologies (Wolters Kluwer Health)
Published
2023-06-23
Volume
102
Issue
25
Pages
e34068
ISSN
0025-7974 (ISSN-L)
DOI
10.1097/md.0000000000034068
PMC
PMC10289633
OpenAlex
W4381838911
Cited by
57
References
5
Citations by year
2026: 7 · 2025: 18 · 2024: 27 · 2023: 5
FWCI
2.28
Type
journal article
Language
en
Open access
Licence
Locations
  • Medicine (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
ChatGPT, large language models, clinical decision support, pediatrics, Rasch analysis, clinical symptoms, AI performance evaluation
MeSH terms
Pediatricians, Child, Delivery of Health Care, Humans, Pediatrics, Software, Decision Support Systems, Clinical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review