Assessing the Value of ChatGPT for Clinical Decision Support Optimization
Open accessAbstract
Objective: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods: We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. Results: Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. Conclusion: AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT offers numerous benefits in health care, including optimizing radiology reporting, generating patient discharge summaries, improving patient care, providing antimicrobial advice, and improving clinical decision support.
“In health care, ChatGPT offers numerous benefits [17], including optimizing radiology reporting [18], generating patient discharge summaries [19], improving patient care [20], providing antimicrobial advice [21], and improving clinical decision support [22].”
From Ethical Considerations of Using ChatGPT in Health Care (2023) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 29,639 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115 · see citing works
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703 · see citing works
Effect of Clinical Decision-Support Systems: A Systematic Review
Bright, T. J., Wong, A., Dhurjati, R., Bristow, E., Bastian, L., Coeytaux, R. R., Samsa, G., Hasselblad, V., Williams, J. W., Musty, M. D., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-1-201207030-00450 · Cited by 1,200 · see citing works
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · see citing works
Overriding of Drug Safety Alerts in Computerized Physician Order Entry
van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1809 · Cited by 1,117 · see citing works
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · see citing works
From Mindless to Mindful Practice — Cognitive Bias and Clinical Decision Making
Croskerry, P. (2013) · New England Journal of Medicine · doi:10.1056/nejmp1303712 · Cited by 572 · see citing works
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549 · see citing works
Physicians' Decisions to Override Computerized Drug Alerts in Primary Care
Weingart, S. N., Toth, M., Sands, D. Z., Aronson, M. D., Davis, R. B., & Phillips, R. S. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.21.2625 · Cited by 487 · see citing works
Tiering Drug-Drug Interaction Alerts by Severity Increases Compliance Rates
Paterno, M. D., Maviglia, S. M., Gorman, P. N., Seger, D. L., Yoshida, E., Seger, A. C., Bates, D. W., & Gandhi, T. K. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2808 · Cited by 263 · see citing works
Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon (Addendum added October 2012)
de Winter, J. F. C., & Dodou, D. (2010) · University of Massachusetts Amherst · doi:10.7275/bj1p-ts64 · Cited by 231 · see citing works
A "Fundamental Theorem" of Biomedical Informatics
Friedman, C. P. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3092 · Cited by 201 · see citing works
A protocol for dynamic model calibration
Villaverde, A. F., Pathirana, D., Fröhlich, F., Hasenauer, J., & Banga, J. R. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab387 · Cited by 139 · see citing works
Powers, E. M., Shiffman, R. N., Melnick, E. R., Hickner, A., & Sharifi, M. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy112 · Cited by 128 · see citing works
Kane‐Gill, S. L., O’Connor, M. F., Rothschild, J. M., Selby, N. M., McLean, B. A., Bonafide, C. P., Cvach, M. M., Hu, X., Konkani, A., Pelter, M. M., et al. (2017) · Critical Care Medicine · doi:10.1097/ccm.0000000000002580 · Cited by 115 · see citing works
Eliminating Health Care Disparities With Mandatory Clinical Decision Support
Lau, B. D., Haider, A. H., Streiff, M. B., Lehmann, C. U., Kraus, P. S., Hobson, D. B., Kraenzlin, F. S., Zeidan, A. M., Pronovost, P. J., & Haut, E. R. (2014) · Medical Care · doi:10.1097/mlr.0000000000000251 · Cited by 113 · see citing works
What, if all alerts were specific - estimating the potential impact on drug interaction alert burden
Seidling, H. M., Klein, U., Schaier, M., Czock, D., Theile, D., Pruszydlo, M. G., Kaltschmidt, J., Mikus, G., & Haefeli, W. E. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2013.12.006 · Cited by 91 · see citing works
Olakotan, O. O., & Yusof, M. M. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103453 · Cited by 75 · see citing works
Clinician collaboration to improve clinical decision support: the Clickbusters initiative
McCoy, A. B., Russo, E. M., Johnson, K. B., Addison, B., Patel, N., Wanderer, J. P., Mize, D. E., Jackson, J. G., Reese, T. J., Littlejohn, S., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac027 · Cited by 41 · see citing works
Self-citationFundamentals of Measurement in Clinical Psychology
Shou, Y.-Y., Sellbom, M., & Chen, H.-F. (2021) · Reference Module in Neuroscience and Biobehavioral Psychology · doi:10.1016/b978-0-12-818697-8.00110-2 · Cited by 33 · see citing works
Sorace, J. M., Wong, H., DeLeire, T., et al. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.104037 · Cited by 32 · see citing works
Mitchell, J. P., Probst, J. C., Brock‐Martin, A., Bennett, K. J., Glover, S. H., & Hardin, J. W. (2013) · The Journal of Rural Health · doi:10.1111/jrh.12043 · Cited by 29 · see citing works
Reduction of clinical support warnings through recategorization of severity levels
Parke, C. K., Santiago, E., Zussy, B., & Klipa, D. (2014) · American Journal of Health-System Pharmacy · doi:10.2146/ajhp140095 · Cited by 29 · see citing works
Premature closure? Not so fast
Dhaliwal, G. (2016) · BMJ Quality & Safety · doi:10.1136/bmjqs-2016-005267 · Cited by 29 · see citing works
The potential for leveraging machine learning to filter medication alerts
Liu, S., Kawamoto, K., Fiol, G. D., Weir, C., Malone, D. C., Reese, T. J., Morgan, K. L., ElHalta, D., & Abdelrahman, S. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab292 · Cited by 28 · see citing works
Self-citationHorn, J., & Ueng, S. (2019) · Annals of Pharmacotherapy · doi:10.1177/1060028019863419 · Cited by 26 · see citing works
A Crowdsourcing Framework for Medical Data Sets.
Ye, C., Coco, J., Epishova, A., Hajaj, C., Bogardus, H., Novak, L. L., Denny, J. C., Vorobeychik, Y., Lasko, T. A., Malin, B., et al. (2018) · Europe PMC (PubMed Central) · Cited by 24 · see citing works
Craig, K. J. T., Fusco, N., Lindsley, K. B., Snowdon, J. L., Willis, V. C., Arriaga, Y. E., & Dankwa‐Mullan, I. (2020) · Cardiovascular Digital Health Journal · doi:10.1016/j.cvdhj.2020.11.001 · Cited by 23 · see citing works
Optimizing Drug-Drug Interaction Alerts Using a Multidimensional Approach
Daniels, C. C., Burlison, J. D., Baker, D. K., Robertson, J., Sablauer, A., Flynn, P. M., Campbell, P. K., & Hoffman, J. M. (2019) · PEDIATRICS · doi:10.1542/peds.2017-4111 · Cited by 21 · see citing works
Improving the specificity of drug-drug interaction alerts: Can it be done?
Reese, T., Wright, A. T., Liu, S., Boyce, R. D., Romero, A. V., Fiol, G. D., Kawamoto, K., & Malone, D. C. (2022) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxac045 · Cited by 14 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Sirajuddin AM, Osheroff JA, Sittig DF, et al. Implementation pearls from a new guidebook on improving medication use and outcomes with clinical decision support. Effective CDS is essential for addressing healthcare performance improvement imperatives. J Healthc Inf Manag 2009;23:38-45./pmc/articles/PMC3316472/ (accessed 1 Jun 2021).
- Brown TB, Mann B, Ryder N, et al. Language Models are Few-Shot Learners. Published Online First: 28 May 2020.http://arxiv.org/abs/2005.14165 (accessed 30 Jan 2023).
Cited by
Sorted by citations
Ethical Considerations of Using ChatGPT in Health Care
Wang, C., Liu, S., Yang, H., Guo, J., Wu, Y., & Liu, J. (2023) · Journal of Medical Internet Research · doi:10.2196/48009 · Cited by 541 · see citing works
Self-citationUnlocking the opportunities through ChatGPT Tool towards ameliorating the education system
Javaid, M., Haleem, A., Singh, R. P., Khan, S., & Khan, I. H. (2023) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100115 · Cited by 535 · see citing works
Utility of ChatGPT in Clinical Practice
Liu, J., Wang, C., & Liu, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48568 · Cited by 505 · see citing works
Self-citationChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452 · see citing works
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Assessing the research landscape and clinical utility of large language models: a scoping review
Park, Y.-J., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02459-6 · Cited by 187 · see citing works
Temsah, M.-H., Aljamaan, F., Malki, K. H., Alhasan, K., Altamimi, I., Aljarbou, R., Bazuhair, F., Alsubaihin, A., Abdulmajeed, N., Alshahrani, F. S., et al. (2023) · Healthcare · doi:10.3390/healthcare11131812 · Cited by 179 · see citing works
Williamson, S. M., & Prybutok, V. (2024) · Information · doi:10.3390/info15060299 · Cited by 146 · see citing works
Shao, C.-Y., Li, H., Liu, X.-L., Li, C., Yang, L.-Q., Zhang, Y.-J., Luo, J., & Zhao, J. (2023) · Interactive Journal of Medical Research · doi:10.2196/46900 · Cited by 49 · see citing works
ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis.
Ahmed, W., Saturno, M., Rajjoub, R., Duey, A. H., Zaidat, B., Hoang, T., Restrepo Mejia, M., Gallate, Z. S., Shrestha, N., Tang, J., et al. (2024) · European Spine Journal · doi:10.1007/s00586-024-08198-6 · Cited by 37 · see citing works
A bibliometric analysis of the advance of artificial intelligence in medicine
Lin, M., Lin, L., Lin, L., Lin, Z., & Yan, X. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1504428 · Cited by 36 · see citing works
Nasef, H., Patel, H., Amin, Q., Baum, S., Ratnasekera, A., Ang, D., Havron, W. S., Nakayama, D., & Elkbuli, A. (2025) · The American Surgeon™ · doi:10.1177/00031348241256075 · Cited by 4 · see citing works
- Authors
- Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A.
- Affiliations
- Vanderbilt University Medical Center (US)
- The University of Texas Southwestern Medical Center (US)
- The University of Texas Health Science Center at Houston (US)
- Corresponding author
- Liu, Siru
- ORCID
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2023-02-23
- PMC
- PMC9980251
- OpenAlex
- W4321605784
- Cited by
- 90
- References
- 33
- Citations by year
- 2026: 6 · 2025: 21 · 2024: 31 · 2023: 32
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- medRxiv (acceptedVersion)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion)
- Funders
- National Institutes of Health (R01LM013995-01, K99LM014097-01)
- National Institute on Aging (R01AG062499)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- ChatGPT, clinical decision support, large language models, non-inferiority, learning health systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
