Using AI-generated suggestions from ChatGPT to optimize clinical decision support
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Objective: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods: We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. Results: Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. Conclusion: AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The integration of AI-powered tools into clinical pharmacy practices could bolster efficiency, decision-making, and patient outcomes.9”
From ChatGPT and Clinical Training: Perception, Concerns, and Practice of Pharm-D Students (2023) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
A General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703
Effect of Clinical Decision-Support Systems: A Systematic Review
Bright, T. J., Wong, A., Dhurjati, R., Bristow, E., Bastian, L., Coeytaux, R. R., Samsa, G., Hasselblad, V., Williams, J. W., Musty, M. D., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-1-201207030-00450 · Cited by 1,200
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150
Overriding of Drug Safety Alerts in Computerized Physician Order Entry
van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1809 · Cited by 1,117
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · cited 1×
“For example, using a publicly available clinical language model, GatorTron, based on 90 billion words of deidentified clinical notes from the University of Florida Health, PubMed articles, and Wikipedia, might be a good start.(36) Another example is BioGPT, a GPT-like model trained on PubMed articles.”
MethodFrom Mindless to Mindful Practice — Cognitive Bias and Clinical Decision Making
Croskerry, P. (2013) · New England Journal of Medicine · doi:10.1056/nejmp1303712 · Cited by 571
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549
Physicians' Decisions to Override Computerized Drug Alerts in Primary Care
Weingart, S. N., Toth, M., Sands, D. Z., Aronson, M. D., Davis, R. B., & Phillips, R. S. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.21.2625 · Cited by 487
Willits, F. K., Theodori, G. L., & Luloff, A. E. (2016) · Journal of Rural Social Sciences · Cited by 379
Tiering Drug-Drug Interaction Alerts by Severity Increases Compliance Rates
Paterno, M. D., Maviglia, S. M., Gorman, P. N., Seger, D. L., Yoshida, E., Seger, A. C., Bates, D. W., & Gandhi, T. K. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2808 · Cited by 263
Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon (Addendum added October 2012)
de Winter, J. F. C., & Dodou, D. (2010) · University of Massachusetts Amherst · doi:10.7275/bj1p-ts64 · Cited by 231 · cited 1×
“metric Mann-Whitney Wilcoxon test to compare expert ratings of AI-generated suggestions and human-generated suggestions.(32) In the”
MethodA "Fundamental Theorem" of Biomedical Informatics
Friedman, C. P. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3092 · Cited by 201
Sirajuddin, A. M., Osheroff, J. A., Sittig, D. F., Chuo, J., Velasco, F. T., & Collins, D. A. (2009) · PubMed · Cited by 131
Self-citationPowers, E. M., Shiffman, R. N., Melnick, E. R., Hickner, A., & Sharifi, M. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy112 · Cited by 128
Kane-Gill, S. L., O’Connor, M. F., Rothschild, J. M., Selby, N. M., McLean, B., Bonafide, C. P., Cvach, M. M., Hu, X., Konkani, A., Pelter, M. M., et al. (2017) · Critical Care Medicine · doi:10.1097/ccm.0000000000002580 · Cited by 115
Lau, B. D., Haider, A. H., Streiff, M. B., Lehmann, C. U., Kraus, P. S., Hobson, D. B., Kraenzlin, F. S., Zeidan, A. M., Pronovost, P. J., & Haut, E. R. (2015) · Medical Care · doi:10.1097/mlr.0000000000000251 · Cited by 113
ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B., Ricke, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.14882 · Cited by 97
What, if all alerts were specific - estimating the potential impact on drug interaction alert burden
Seidling, H. M., Klein, U., Schaier, M., Czock, D., Theile, D., Pruszydlo, M. G., Kaltschmidt, J., Mikus, G., & Haefeli, W. E. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2013.12.006 · Cited by 91
Olakotan, O. O., & Yusof, M. M. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103453 · Cited by 70
Fundamentals of Measurement in Clinical Psychology
Shou, Y., Sellbom, M., & Chen, H.-F. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-818697-8.00110-2 · Cited by 60
Clinician collaboration to improve clinical decision support: the Clickbusters initiative
McCoy, A. B., Russo, E. M., Johnson, K. B., Addison, B., Patel, N., Wanderer, J. P., Mize, D. E., Jackson, J. G., Reese, T. J., Littlejohn, S., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac027 · Cited by 41 · cited 4×
“We previously conducted an alert optimization program—Clickbusters—involving a 10-step process of reviewing alert related data and clinical evidence, identifying possible improvements, discussing improvements with stakeholders, making changes in the test environment, testing, and evaluating.(19) The entire process was…”
BackgroundMethodInfluentialSelf-citationSorace, J. M., Wong, H., DeLeire, T., Xu, D., Handler, S., Garcia, B., & MaCurdy, T. E. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.104037 · Cited by 34
Reduction of clinical support warnings through recategorization of severity levels
Parke, C., Santiago, E., Zussy, B., & Klipa, D. (2015) · American Journal of Health-System Pharmacy · doi:10.2146/ajhp140095 · Cited by 29
Premature closure? Not so fast
Dhaliwal, G. (2017) · BMJ Quality & Safety · doi:10.1136/bmjqs-2016-005267 · Cited by 29
Mitchell, J. P., Probst, J. C., Brock‐Martin, A., Bennett, K. J., Glover, S. H., & Hardin, J. W. (2013) · The Journal of Rural Health · doi:10.1111/jrh.12043 · Cited by 27
A Crowdsourcing Framework for Medical Data Sets.
Ye, C., Coco, J., Epishova, A., Hajaj, C., Bogardus, H., Novak, L. L., Denny, J. C., Vorobeychik, Y., Lasko, T. A., Malin, B., et al. (2018) · PubMed Central · Cited by 27
Craig, K. J. T., Fusco, N., Lindsley, K. B., Snowdon, J. L., Willis, V. C., Arriaga, Y. E., & Dankwa‐Mullan, I. (2020) · Cardiovascular Digital Health Journal · doi:10.1016/j.cvdhj.2020.11.001 · Cited by 27
The potential for leveraging machine learning to filter medication alerts
Liu, S., Kawamoto, K., Del Fiol, G., Weir, C., Malone, D. C., Reese, T. J., Morgan, K., ElHalta, D., & Abdelrahman, S. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab292 · Cited by 27
Self-citationHorn, J., & Ueng, S. (2019) · Annals of Pharmacotherapy · doi:10.1177/1060028019863419 · Cited by 26
Optimizing Drug-Drug Interaction Alerts Using a Multidimensional Approach
Daniels, C. C., Burlison, J. D., Baker, D. K., Robertson, J., Sablauer, A., Flynn, P. M., Campbell, P. K., & Hoffman, J. M. (2019) · Pediatrics · doi:10.1542/peds.2017-4111 · Cited by 21
Improving the specificity of drug-drug interaction alerts: Can it be done?
Reese, T., Wright, A. T., Liu, S., Boyce, R. D., Romero, A. V., Fiol, G. D., Kawamoto, K., & Malone, D. C. (2022) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxac045 · Cited by 15
Self-citation
Research works printed but not linked: 1
- de Winter, 2010, Five-point Likert items: t test versus Mann-Whitney-Wilcoxon, Pract Assess Res Eval, 15, 11
Cited by
Dwivedi, Y. K., Pandey, N., Currie, W., & Micu, A. (2024) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-05-2023-0686 · Cited by 570
Ethical Considerations of Using ChatGPT in Health Care
Wang, C., Liu, S., Yang, H., Guo, J., Wu, Y., & Liu, J. (2023) · Journal of Medical Internet Research · doi:10.2196/48009 · Cited by 541
Self-citationUnlocking the opportunities through ChatGPT Tool towards ameliorating the education system
Javaid, M., Haleem, A., Singh, R. P., Khan, S., & Khan, I. H. (2023) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100115 · Cited by 535
Utility of ChatGPT in Clinical Practice
Liu, J., Wang, C., & Liu, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48568 · Cited by 505
Self-citationAccuracy and Reliability of Chatbot Responses to Physician Questions
Goodman, R. S., Patrinely, J. R., Stone, C. A., Zimmerman, E., Donald, R. R., Chang, S. S., Berkowitz, S. T., Finn, A. P., Jahangir, E., Scoville, E. A., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.36483 · Cited by 484
Metadata
- Authors
- Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A.
- Affiliations
- Vanderbilt University Medical Center (US)
- The University of Texas Southwestern Medical Center (US)
- The University of Texas Health Science Center at Houston (US)
- Corresponding author
- Liu, Siru
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-04-22
- Volume
- 30
- Issue
- 7
- Pages
- 1237-1245
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC10280357
- OpenAlex
- W4366769280
- Cited by
- 397
- References
- 34
- Citations by year
- 2026: 64 · 2025: 126 · 2024: 144 · 2023: 61
- FWCI
- 15.73
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- Funders
- NIH (K99LM014097-01, R01AG062499-01, R01LM013995-01)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- ChatGPT, clinical decision support, large language models, non-inferiority, reinforcement learning from human feedback, learning health systems
- MeSH terms
- Learning Health System, Artificial Intelligence, Humans, Language, Decision Support Systems, Clinical, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.