Using AI-generated suggestions from ChatGPT to optimize clinical decision support
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Objective: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods: We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. Results: Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. Conclusion: AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The integration of AI-powered tools into clinical pharmacy practices could bolster efficiency, decision-making, and patient outcomes.9”
From ChatGPT and Clinical Training: Perception, Concerns, and Practice of Pharm-D Students (2023) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
A General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703
Effect of Clinical Decision-Support Systems: A Systematic Review
Bright, T. J., Wong, A., Dhurjati, R., Bristow, E., Bastian, L., Coeytaux, R. R., Samsa, G., Hasselblad, V., Williams, J. W., Musty, M. D., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-1-201207030-00450 · Cited by 1,200
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185
Research works printed but not linked: 1
- de Winter, 2010, Five-point Likert items: t test versus Mann-Whitney-Wilcoxon, Pract Assess Res Eval, 15, 11
Cited by
Items 501–542 of 542
Navigating the utility of generative artificial intelligence in healthcare delivery
Kulkarni, M., Sheth, M., & Chaudhary, P. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-45252-9.00015-3
Zhu, L., Zhong, H., Bai, J., Wang, X., Huang, P., & Xu, Z. (2026) · Digital Health · doi:10.1177/20552076261438675
Understanding AI-Assisted Learning Acceptance Among Vocational Students: An AIDUA Model Approach
Ye, J.-H., Yang, X., Li, B., Nong, W., Hong, J., & Wang, L. (2026) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2026.16.4.2557
Leveraging a Large Language Model to Generate Quality Improvement Feedback for Clinical Notes
Kim, C., Gelfinbein, J., Gencerliler, N., Jahan, N., Udaikumar, J., Heery, L., Goodman, A., Ng, S., Attard, J., Asha, S., et al. (2026) · Applied Clinical Informatics · doi:10.1055/a-2851-0739
Pioneers & Pathfinders: 10 Years of Frontiers in Medicine
(2026) · Frontiers research topics · doi:10.3389/978-2-8325-7221-4
Implementation of Generative AI in Biomedical Research and Healthcare
Nikolopoulos, A., & Karalis, V. D. (2026) · Applied Biosciences · doi:10.3390/applbiosci5020034
Building Toward a Future for Electronic Health Record Systems
Rosenbloom, S. T., Holmgren, A. J., & Steitz, B. D. (2026) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-092724-050949
Can we rely on generative AI for emergency patient triage classification?
Nwandikom, U., & Namin, A. S. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100301
Artificial intelligence-driven medicine repurposing for chronic pyelonephritis: A potential solution
Wiwanitkit, S., & Wiwanitkit, V. (2026) · Journal of Medical Society · doi:10.4103/jms.jms_8_24
Pereira, R. A. D. C., Schwingel, P. A., Estrada, L. C. N. C. D., & Gambassi, B. B. (2026) · Editora Científica Digital eBooks · doi:10.37885/260321502
Konukoğlu, O., Kaya, M., Eseoğlu, Ş., & Gunaydin, I. (2026) · Ankara Üniversitesi Tıp Fakültesi Mecmuası · doi:10.65092/autfm.1878887
Xiaoyuan, W., Tang, Y., Lin, Y., Huang, Z., Huang, X., & Cai, J. (2026) · doi:10.2196/preprints.106057
Assessing Pain Catastrophizing Through Free-Text Responses: A Validation of Large Language ModelsPreprint
Lee, A., You, D. S., & Dildine, T. C. (2026) · medRxiv · doi:10.64898/2026.07.22.26358710
Rao, A. S., Nwokemodo-Ihejirika, C., Kincaid, J. W. R., Крылова, М. В., Esmail, K. P., Nguyen, D. S., Rivera, C., Koranteng, E., & Succi, M. D. (2026) · Journal of Medical Internet Research · doi:10.2196/87818
Wang, H.-J., Lee, W., Gau, T., Cheng, K., & Wei, C.-R. (2026) · JMIR Medical Informatics · doi:10.2196/84260
TumorBoard: Evidence-Grounded Multi-Agent Decision Support for Longitudinal Neuro-OncologyPreprint
Liu, Y., Zhang, Z., Liu, R., Xitang, M., Shin, S.-Y., & Lee, H.-A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.03190
Dual-process theory and decision-making in large language modelsPreprint
Ward, T., Brady, O., Zhang, L., McGovern, D. P., & Nulty, P. (2026) · PsyArXiv (OSF Preprints)
Jayakody, J. A. U. S., Jayawardhana, S. M. M. S., & P.R.H.N.G.Thilakarathne (2025) · Proceedings of the sliit international conference on engineering and technology · doi:10.54389/jocb4687
Fischer, D., Meier, L., Koch, S., & Braun, T. (2026) · Archives of Pharmacy Practice · doi:10.51847/whawvgp2pm
Improving Collaborative Management in Chronic Diseases through AI-Based Decision Support Tools
Barbaric, A. (2026) · TSpace (University of Toronto)
The Role of Advanced Language Models in MedicalDiagnostics: A Case Study on Breast Cancer Prediction
Karayiğit, H., & Kalelıoğlu, F. (2026) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2026.2221
Novak, P., & Svoboda, J. (2024) · Journal of Health Informatics and Digital Systems · doi:10.68159/m602539487
Hall, N., Byrom, B., Smith, T., James, J., & Eliasson, L. (2026) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2026.2717295
Large Language Model–Driven Automation of Microbiome Diagnostic Reporting in Clinical Laboratories
Alvarez, M., Herrera, S., Cruz, D., & Castro, A. (2026) · Journal of Medical Sciences and Interdisciplinary Research · doi:10.51847/ttj3atzob6
Garces, I. A., Wong, A. G., Brutti, J. A., Bini, S. A., McDaid, A., & Lawrie, C. M. (2026) · JBJS Open Access · doi:10.2106/jbjs.oa.26.00077
Ceylan, İ., Ekin, S., Özyaprak, B., & Gülen, D. (2026) · Frontiers in Pharmacology · doi:10.3389/fphar.2026.1911849
Rethinking Objectivity in Clinical AI: A Qualitative Study of Concussion Evaluators
Zaidi, A. M., Saw, J. J.-W., Fu, L., Arneson, K., Kim, I., Cross, A., & Karahalios, K. G. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816992
Explainable and Knowledge-Enhanced Machine Learning for Biomedical Representation Learning
Gao, L. (2026) · ThinkTech (Texas Tech University)
Wright, A. T., Liu, S., & Wright, A. P. (2026) · medRxiv · doi:10.64898/2026.09.24.26363906
Self-citationPopescu, A., Ionescu, M., Stan, E., Dumitrescu, S., & Pavel, I. (2026) · Journal of Health Informatics and Digital Systems · doi:10.68159/j827489866
Implementing generative artificial intelligence in a tertiary hospital health screening center
Song, J., Park, J. S., Park, S. H., Kim, H., Baek, M. Y., Lee, C. M., Lee, Y. J., Yu, H. W., cho, S., & Choe, E. K. (2026) · Scientific Reports · doi:10.1038/s41598-026-73939-0
Qiu, X., Chen, C., & Zhang, L. (2026) · Scientific Reports · doi:10.1038/s41598-026-72931-y
Generative Ai In Medicine And Health Care: Promises, Opportunities And Challenges
Bhatol, M. D., Chaudhary, K. K., & Patel, V. P. (2024) · Educational Administration Theory and Practice journal
Nedal, N., El‐Dalahmeh, A., & Li, J. (2026) · Authorea · doi:10.22541/authorea.15010000/v1
Mauda-Havakuk, M., Golan, O., Freitas, V. A. R., Filyavich, A., Diai, S., Neeman, R., Mercer, D. G., & Amitai, Y. (2026) · Journal of Clinical Imaging Science · doi:10.25259/jcis_132_2026
Lee, J., Shin, J., & Cho, B. (2026) · Journal of Korean Medical Science · doi:10.3346/jkms.2026.41.e339
Jaffri, S. U., Koo, A. C., Ting, C., & Hussain, S. (2026) · Healthcare · doi:10.3390/healthcare14213643
Zhang, H., Yu, H., Deng, H., Wang, H., & Dong, Y. (2026) · European Journal of Education · doi:10.1111/ejed.70900
Alexiou, V. G., Vassiliou, A., Sumpio, B. E., Geroulakos, G., & Kakkos, S. K. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag178
Li, Y., Liao, Y., Lin, L., & Zhao, T. (2026) · Artificial Intelligence and Autonomous Systems · doi:10.55092/aias20260007
Ethical Issues and Bias Considerations in GenAI Feedback: Analytics, Design, and Impact
Yan, J., & King, N. R. (2026) · Advances in analytics for learning and teaching · doi:10.1007/978-3-032-31540-3_9
Human–Agentic AI Collaboration Workflow for Trend Analysis of ChatGPT Applications in HealthcarePreprint
Chou, P. J., Lin, Y., & Li, E. Y. (2026) · medRxiv · doi:10.64898/2026.11.14.26368120
Metadata
- Authors
- Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A.
- Affiliations
- Vanderbilt University Medical Center (US)
- The University of Texas Southwestern Medical Center (US)
- The University of Texas Health Science Center at Houston (US)
- Corresponding author
- Liu, Siru
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-04-22
- Volume
- 30
- Issue
- 7
- Pages
- 1237-1245
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC10280357
- OpenAlex
- W4366769280
- Cited by
- 397
- References
- 34
- Citations by year
- 2026: 64 · 2025: 126 · 2024: 144 · 2023: 61
- FWCI
- 15.73
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- Funders
- NIH (K99LM014097-01, R01AG062499-01, R01LM013995-01)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- ChatGPT, clinical decision support, large language models, non-inferiority, reinforcement learning from human feedback, learning health systems
- MeSH terms
- Learning Health System, Artificial Intelligence, Humans, Language, Decision Support Systems, Clinical, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.