PharmacyGPT: exploration of artificial intelligence for medication management in the intensive care unit
Open accessAbstract
BACKGROUND: The purpose of this evaluation was to develop a novel framework for large language model (LLMs) iterative prompt optimization for the specialized domain of medication decision-making in the intensive care unit (ICU). This serves as a first step towards the use of LLMs as clinical decision support (CDS) tools capable of performing pharmacy related tasks. METHODS: Using a cohort of 1,000 adult patients managed in the ICU for greater than 24 h, an iterative optimization process in the GPT-4 framework was applied to enhance performance of various medication-related tasks, including patient disease state clustering by medication regimen, medication regimen generation, and outcome prediction with the intent to develop PharmacyGPT. Within this feedback loop, the input prompts were adjusted based on model performance in previous iterations. RESULTS: The iterative prompt optimization process housed within PharmacyGPT was able to develop meaningful disease state clusters based on input information, develop medication regimens with inclusion of drug, dose, and frequency, and predict mortality with an accuracy of 0.75, precision 0.37, and recall 0.70. CONCLUSION: Iterative prompt optimization shows promise as a rapid means to improve LLM functionality to specific tasks, even in highly domain specific areas like medication management in the ICU. Such domain specific engineering may serve as a strategy to develop LLMs as viable CDS tools.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some general-purpose LLMs, including GPT-4-based models, can be customized through methods such as prompt engineering and in-context learning and augmented with domain-specific external knowledge sources, enabling responses grounded in relevant databases or documents.
“Some general-purpose LLMs, including GPT-4-based models, can be customized through methods such as prompt engineering and in-context learning (e.g., PharmacyGPT), 10 and augmented with domain-specific external knowledge sources, enabling responses grounded in relevant databases or documents.114”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [10]
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1× · see citing works
“These models are primarily trained on textual data and may not have sufficient exposure to specialized medical terminology, complex concepts, or rare conditions, which are vital for effective decision-making in healthcare [56, 57].”
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · see citing works
Pharmacist Participation on Physician Rounds and Adverse Drug Events in the Intensive Care Unit
Leape, L. L. (1999) · JAMA · doi:10.1001/jama.282.3.267 · Cited by 1,606 · see citing works
Addressing bias in big data and AI for health care: A call for open science
Norori, N., Hu, Q., Aellen, F. M., Faraci, F. D., & Tzovara, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100347 · Cited by 903 · see citing works
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855 · see citing works
Self-citationPreventable adverse drug events in hospitalized patients
Cullen, D. J., Sweitzer, B. J., Bates, D. W., Burdick, E., Edmondson, A. C., & Leape, L. L. (1997) · Critical Care Medicine · doi:10.1097/00003246-199708000-00014 · Cited by 611 · see citing works
Modern hierarchical, agglomerative clustering algorithms
Müllner, D. (2011) · arXiv · doi:10.48550/arxiv.1109.2378 · Cited by 529 · see citing works
Mathematical Capabilities of ChatGPT
Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., & Berner, J. (2023) · arXiv · doi:10.48550/arxiv.2301.13867 · Cited by 385 · see citing works
A Systematic Review of the Barriers to the Implementation of Artificial Intelligence in Healthcare
Ahmed, M. I., Spooner, B., Isherwood, J., Lane, M., Orrock, E., & Dennison, A. (2023) · Cureus · doi:10.7759/cureus.46454 · Cited by 364 · see citing works
ChatGPT: Is this version good for healthcare and research?
Vaishya, R., Misra, A., & Vaish, A. (2023) · Diabetes & Metabolic Syndrome: Clinical Research & Reviews · doi:10.1016/j.dsx.2023.102744 · Cited by 293 · see citing works
Valentin, A., Capuzzo, M., Guidet, B., Moreno, R. P., Dolanski-Aghamanoukjan, L., Bauer, P., & Metnitz, P. (2006) · Intensive Care Medicine · doi:10.1007/s00134-006-0290-7 · Cited by 268 · see citing works
Halpern, N. A., Goldman, D. A., Tan, K. S., & Pastores, S. M. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000001722 · Cited by 224 · see citing works
Evaluating large language models on a highly-specialized topic, radiation oncology physics
Holmes, J., Liu, Z., Zhang, L., Ding, Y., Sio, T. T. W., McGee, L. A., Ashman, J. B., Li, X., Liu, T., Shen, J., et al. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1219326 · Cited by 166 · see citing works
Self-citationAnalysis of risk factors for adverse drug events in critically ill patients*
Kane‐Gill, S. L., Kirisci, L., Verrico, M. M., & Rothschild, J. M. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e318236f473 · Cited by 151 · see citing works
Prompt Engineering for Healthcare: Methodologies and Applications
Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.14670 · Cited by 150 · see citing works
Self-citationAdverse drug events in intensive care units: Risk factors, impact, and the role of team care
Kane‐Gill, S. L., Jacobi, J., & Rothschild, J. M. (2010) · Critical Care Medicine · doi:10.1097/ccm.0b013e3181dd8364 · Cited by 147 · see citing works
Position Paper on Critical Care Pharmacy Services: 2020 Update
Lat, I., Paciullo, C. A., Daley, M. J., MacLaren, R., Bolesta, S., McCann, J., Stollings, J. L., Gross, K., Foos, S. A., Roberts, R. J., et al. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004437 · Cited by 143 · see citing works
A path to precision in the ICU
Maslove, D. M., Lamontagne, F., Marshall, J. C., & Heyland, D. K. (2017) · Critical Care · doi:10.1186/s13054-017-1653-x · Cited by 109 · see citing works
Costs of adverse events in intensive care units*
Kaushal, R., Bates, D. W., Franz, C., Soukup, J. R., & Rothschild, J. M. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000284510.04248.66 · Cited by 100 · see citing works
When brain-inspired AI meets AGI
Zhao, L., Zhang, L., Wu, Z., Chen, Y., Dai, H., Yu, X., Liu, Z., Zhang, T., Hu, X., Jiang, X., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100005 · Cited by 96 · see citing works
Self-citationDeID-GPT: Zero-shot Medical Text De-Identification by GPT-4Preprint
Liu, Z., Huang, Y., Yu, X., Zhang, L., Wu, Z., Cao, C., Dai, H., Zhao, L., Li, Y., Shu, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.11032 · Cited by 85 · see citing works
Self-citationArtificial General Intelligence for Medical Imaging Analysis
Li, X., Zhao, L., Zhang, L., Wu, Z., Liu, Z., Jiang, H., Cao, C., Xu, S., Li, Y., Dai, H., et al. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3493775 · Cited by 68 · see citing works
Self-citationGwynn, M. E., Poisson, M. O., Waller, J. L., et al. (2019) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxy054 · Cited by 67 · see citing works
Sikora, A. N., Ayyala, D. N., Rech, M. A., Blackwell, S. B., Campbell, J., Caylor, M. M., Smith, M. N., DePriest, A., Dzierba, A. L., Flannery, A. H., et al. (2022) · Critical Care Medicine · doi:10.1097/ccm.0000000000005585 · Cited by 62 · see citing works
Self-citationRadiology-GPT: A Large Language Model for Radiology
Liu, Z., Zhong, A., Li, Y., Yang, L., Ju, C., Wu, Z., Ma, C., Shu, P., Chen, C., Kim, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.08666 · Cited by 59 · see citing works
Self-citationPHarmacist Avoidance or Reductions in Medical Costs in CRITically Ill Adults: PHARM-CRIT Study
Rech, M. A., Gurnani, P. K., Peppard, W., Smetana, K. S., Berkel, M. V., Hammond, D. A., & Flannery, A. H. (2021) · Critical Care Explorations · doi:10.1097/cce.0000000000000594 · Cited by 57 · see citing works
AgriBERT: Knowledge-Infused Agricultural Language Models for Matching Food and Nutrition
Rezayi, S., Liu, Z., Wu, Z., Dhakal, C., Ge, B., Zhen, C., Liu, T., & Li, S. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/715 · Cited by 55 · see citing works
Self-citationOptimization of critical care pharmacy clinical services: A gap analysis approach
Sikora, A. N., Murray, B., Smith, S. E., Brothers, T. N., Al-Mamun, M. A., Chase, A. M., Rowe, S., Buckley, M. S., Murphy, D. J., & Devlin, J. W. (2021) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxab237 · Cited by 53 · see citing works
Self-citationBoßelmann, C. M., Leu, C., & Lal, D. (2023) · Epilepsia · doi:10.1111/epi.17570 · Cited by 52 · see citing works
Sikora, A. N., Smith, S. E., Olney, W. J., et al. (2019) · JACCP JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY · doi:10.1002/jac5.1146 · Cited by 50 · see citing works
Self-citationDrug utilization pattern in critical care unit in a tertiary care teaching hospital in India
Tripathi, C., Patel, M., Barvaliya, M. J., et al. (2013) · International Journal of Critical Illness and Injury Science · doi:10.4103/2229-5151.124128 · Cited by 48 · see citing works
Characterization of changes in medication complexity using a modified scoring tool
Sikora, A. N., Anderson, D. T., Gwynn, M. E., et al. (2019) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxz213 · Cited by 37 · see citing works
Self-citationAl-Mamun, M. A., Brothers, T. N., & Sikora, A. N. (2020) · Annals of Pharmacotherapy · doi:10.1177/1060028020959042 · Cited by 33 · see citing works
Self-citationMulticenter validation of a novel medication-regimen complexity scoring tool
Sikora, A. N., Smith, S. E., Olney, W. J., et al. (2020) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxz330 · Cited by 32 · see citing works
Self-citationSmith, S. E., Shelley, R., & Sikora, A. N. (2021) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxab460 · Cited by 31 · see citing works
Self-citationClinicalRadioBERT: Knowledge-Infused Few Shot Learning for Clinical Notes Named Entity Recognition
Rezayi, S., Dai, H., Liu, Z., Wu, Z., Hebbar, A., Burns, A. H., Zhao, L., Zhu, D., Li, Q., Liu, W., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21014-3_28 · Cited by 31 · see citing works
Self-citationMedication Regimen Complexity Score as an Indicator of Fluid Balance in Critically Ill Patients
Olney, W. J., Chase, A. M., Hannah, S., Smith, S. E., & Sikora, A. N. (2021) · Journal of Pharmacy Practice · doi:10.1177/0897190021999792 · Cited by 28 · see citing works
Self-citationSikora, A. N. (2023) · Critical Care Clinics · doi:10.1016/j.ccc.2023.01.006 · Cited by 27 · see citing works
Self-citationWebb, A. J., Rowe, S., & Sikora, A. N. (2022) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxac059 · Cited by 26 · see citing works
Self-citationXie, Q., Han, W., Lai, Y., et al. (2023) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2304.05351 · Cited by 24 · see citing works
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and OpportunitiesPreprint
Song, Y., Wang, T., Mondal, S. K., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.06743 · Cited by 24 · see citing works
Evaluation of medication regimen complexity as a predictor for mortality
Sikora, A. N., Devlin, J. W., Yu, M., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-37908-1 · Cited by 21 · see citing works
Self-citationSmetana, K. S., Flannery, A. H., Gurnani, P. K., et al. (2023) · JACCP JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY · doi:10.1002/jac5.1798 · Cited by 16 · see citing works
Sikora, A. N., Rafiei, A., Rad, M. G., Keats, K. R., Smith, S. E., Devlin, J. W., Murphy, D. J., Murray, B., Kamaleswaran, R., & MRC-ICU Investigator Team (2023) · Critical Care · doi:10.1186/s13054-023-04437-2 · Cited by 15 · see citing works
Self-citationPharmacyGPT: The AI PharmacistPreprint
Liu, Z., Wu, Z., Hu, M., Zhao, B., Zhao, L., Zhang, T., Dai, H., Chen, X., Shen, Y., Li, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10432 · Cited by 14 · see citing works
Self-citationUnderstanding LLMs: A Comprehensive Overview from Training to InferencePreprint
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.02038 · Cited by 14 · see citing works
Self-citationChase, A. M., Azimi, H. A., Forehand, C. C., Keats, K. R., Taylor, A. N., Wu, S., Blotske, K., & Sikora, A. N. (2023) · Hospital Pharmacy · doi:10.1177/00185787231170386 · Cited by 11 · see citing works
Self-citationRadOnc-GPT: A Large Language Model for Radiation OncologyPreprint
Liu, Z., Wang, P., Li, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.10160 · Cited by 11 · see citing works
Self-citationRafiei, A., Rad, M. G., Sikora, A. N., et al. (2023) · medRxiv · doi:10.1101/2023.06.20.23291680 · Cited by 8 · see citing works
Self-citationSikora, A. N., Jeong, H., Yu, M., et al. (2022) · Research Square · doi:10.21203/rs.3.rs-1745568/v1 · Cited by 7 · see citing works
Self-citationMachine learning vs. traditional regression analysis for fluid overload prediction in the ICUPreprint
Sikora, A. N., Zhang, T., Murphy, D. J., et al. (2023) · medRxiv · doi:10.1101/2023.06.16.23291493 · Cited by 5 · see citing works
Self-citationWhen Brain-inspired AI Meets AGIPreprint
Zhao, L., Zhang, L., Wu, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.15935 · Cited by 2 · see citing works
Self-citationAn Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT
Ma, C., Wu, Z., Wang, J., Xu, S., Wei, Y., Zeng, F., Liu, Z., Jiang, X., Guo, L., Cai, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08448
Self-citation
Flagged first, then by citations
Research works printed but not linked: 9
- [5]Bedi S, Orr-Ewing L, Dash S, Koyejo A, Callahan JA, Fries M, Wornow A, Swaminathan LS, Lehmann HJ, Hong M, Kashyap AR, Chaurasia NR, Shah K, Singh T, Tazbaz A, Milstein MA, Pfeffer NH, Shah A. A systematic review of testing and evaluation of healthcare applications of large language models (LLMs). medrixv. https://www.medrxiv.org/content/10.1101/2024.04.15.24305869v1.full · DOI
- [32]van der Maaten LH. Visualizing data using t-SNE. J Mach Learn Res. 2008;9:2579–605.
- [38]Homgmin Cai WL, Zhengliang Liu X, Huang Y, Zhang S, Ding S, Li Q, Li T, Liu, Li X. Coarse-to-fine knowledge graph domain adaptation based on distantly-supervised iterative training. arXiv. 2022. https://arxiv.org/abs/2211.02849. · arXiv
- [40]Sikora A. Critical care pharmacists: A focus on horizons. Crit Care Clin. 2023;39(3):503–27. 10.1016/j.ccc.2023.01.006. 37230553 · DOI
- [48]Webb ARS, Sikora A. Automated MRC-ICU calculations in the electronic medical record of an academic medical center: applications and considerations for critical care pharmacist practice. Am J Health-System Pharm. 2021. [In press].
- [54]Lin C-Y, Rouge. A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. 2004.
- [55]Lavie SBA. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. 2005.
- [61]Holmes J, Liu Z, Zhang L, et al. Evaluating large language models on a highly-specialized topic, radiation oncology physics. arXiv. 2023. https://arxiv.org/abs/2304.01938. PMC10388568 37529688 · arXiv
- [65]Tariq RAVR, Sinha A, et al. Medication dispensing errors and prevention. [Updated 2023 May 2]. In: StatPearls [Internet]. Treasure Island (FL): statpearls publishing; 2023 Jan. Available from: https://www.ncbi.nlm.nih.gov/books/NBK519065/ 30085607
Cited by
Sorted by citations
Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations
Yakob, R. A., Bamgboje-Ayodele, A., Collins, J. C., & Aslani, P. (2026) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2026.09.007
- Authors
- Liu, Z., Xu, S., Wu, Z., Murray, B., F. Barreto, E., Li, S., Liu, W., Li, X., Liu, T., & Sikora, A.
- Affiliations
- University of Georgia (US)
- University of North Carolina at Chapel Hill (US)
- University of Virginia (US)
- Mayo Clinic Hospital (US)
- Harvard University (US)
- University of Colorado Anschutz (US)
- Corresponding author
- Sikora, Andrea
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2025-10-28
- Volume
- 25
- Issue
- 1
- Pages
- 398-398
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC12570815
- OpenAlex
- W4415620200
- Cited by
- 8
- References
- 55
- Citations by year
- 2026: 8
- FWCI
- 1.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, prompt optimization, clinical decision support, medication management, intensive care unit, GPT-4, mortality prediction
- MeSH terms
- Adult, Artificial Intelligence, Humans, Intensive Care Units, Decision Support Systems, Clinical, Medication Therapy Management
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
