Optimal medication dosing from suboptimal clinical examples: a deep reinforcement learning approach
Abstract
Misdosing medications with sensitive therapeutic windows, such as heparin, can place patients at unnecessary risk, increase length of hospital stay, and lead to wasted hospital resources. In this work, we present a clinician-in-the-loop sequential decision making framework, which provides an individualized dosing policy adapted to each patient's evolving clinical phenotype. We employed retrospective data from the publicly available MIMIC II intensive care unit database, and developed a deep reinforcement learning algorithm that learns an optimal heparin dosing policy from sample dosing trails and their associated outcomes in large electronic medical records. Using separate training and testing datasets, our model was observed to be effective in proposing heparin doses that resulted in better expected outcomes than the clinical guidelines. Our results demonstrate that a sequential modeling approach, learned from retrospective data, could potentially be used at the bedside to derive individualized patient dosing policies.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Nemati et al. used deep reinforcement learning to optimize medication dosing.
“For example, Nemati et al. used deep reinforcement learning to optimize medication dosing (35),”
From Accessing Artificial Intelligence for Clinical Decision-Making (2021) · cites this paper as [35]
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083 · see citing works
A New, Evidence-based Estimate of Patient Harms Associated with Hospital Care
James, J. T. (2013) · Journal of Patient Safety · doi:10.1097/pts.0b013e3182948a69 · Cited by 1,399 · see citing works
Saeed, M., Villarroel, M., Reisner, A. T., Clifford, G., Lehman, L.-W., Moody, G., Heldt, T., Kyaw, T. H., Moody, B., & Mark, R. G. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e31820a92c6 · Cited by 1,162 · see citing works
Self-citationRiedmiller, M. (2005) · Lecture notes in computer science · doi:10.1007/11564096_32 · Cited by 771 · see citing works
Achieving a Nationwide Learning Health System
Friedman, C. P., Wong, A. K., & Blumenthal, D. (2010) · Science Translational Medicine · doi:10.1126/scitranslmed.3001456 · Cited by 539 · see citing works
Chakraborty, B., & Murphy, S. A. (2013) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-022513-115553 · Cited by 288 · see citing works
Landefeld, C. S., Cook, E. F., Flatley, M., Weisberg, M. C., & Goldman, L. (1987) · The American Journal of Medicine · doi:10.1016/0002-9343(87)90004-0 · Cited by 264 · see citing works
The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas
Bothe, M. K., Dickens, L., Reichel, K., Tellmann, A., Ellger, B., Westphal, M., & Faisal, A. A. (2013) · Expert Review of Medical Devices · doi:10.1586/17434440.2013.827515 · Cited by 126 · see citing works
The Effectiveness of Implementing the Weight-Based Heparin Nomogram as a Practice Guideline
Raschke, R. A. (1996) · Archives of Internal Medicine · doi:10.1001/archinte.1996.00440140067006 · Cited by 118 · see citing works
Alban, S. (2011) · Handbook of experimental pharmacology · doi:10.1007/978-3-642-23056-1_10 · Cited by 106 · see citing works
A data-driven approach to optimized medication dosing: a focus on heparin
Ghassemi, M. M., Richter, S. E., Eche, I. M., et al. (2014) · Intensive Care Medicine · doi:10.1007/s00134-014-3406-5 · Cited by 66 · see citing works
Self-citationIntrastromal corneal ring implants for corneal thinning disorders: an evidence-based analysis.
(2009) · Europe PMC (PubMed Central) · Cited by 39 · see citing works
Ghassemi, M. M., Lehman, L., Snoek, J., et al. (2014) · RePEc: Research Papers in Economics · Cited by 12 · see citing works
Self-citationSupervised Learning in Dynamic Bayesian NetworksPreprint
Nemati, S., & Adams, R. (2014) · RePEc: Research Papers in Economics · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310 · see citing works
Artificial intelligence to deep learning: machine intelligence approach for drug discovery
Gupta, R., Srivastava, D., Sahu, M., Tiwari, S., Ambasta, R. K., & Kumar, P. (2021) · Molecular Diversity · doi:10.1007/s11030-021-10217-3 · Cited by 1,473 · see citing works
“Likewise, Nemati et al. [126] developed a deep reinforcement learning model trained on multipa-rameter intelligent monitoring in intensive care II database (MIMIC II) to find an ideal dose of another anticoagulant drug, heparin.”
BackgroundArtificial intelligence, bias and clinical safety
Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008370 · Cited by 956 · see citing works
Accessing Artificial Intelligence for Clinical Decision-Making
Giordano, C., Brennan, M., Mohamed, B., Rashidi, P., Modave, F., & Tighe, P. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.645232 · Cited by 326 · see citing works
Reinforcement learning for intelligent healthcare applications: A survey
Coronato, A., Naeem, M., De Pietro, G., & Paragliola, G. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101964 · Cited by 320 · see citing works
Reinforcement learning for clinical decision support in critical care: comprehensive review
Liu, S., See, K. C., Ngiam, K. Y., Celi, L. A., Sun, X., & Feng, M. (2020) · Journal of Medical Internet Research · doi:10.2196/18477 · Cited by 225 · see citing works
Bridging the implementation gap of machine learning in healthcare
Seneviratne, M. G., Shah, N. H., & Chu, L. (2020) · BMJ Innovations · doi:10.1136/bmjinnov-2019-000359 · Cited by 173 · see citing works
Deep learning in pharmacogenomics: from gene regulation to patient stratification
Kalinin, A. A., Higgins, G. A., Reamaroon, N., Soroushmehr, S., Allyn-Feuer, A., Dinov, I. D., Najarian, K., & Athey, B. D. (2018) · Pharmacogenomics · doi:10.2217/pgs-2018-0008 · Cited by 207 · see citing works
Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial
Wang, G., Liu, X., Ying, Z., Yang, G., Chen, Z., Liu, Z., Zhang, M., Yan, H., Lu, Y., Gao, Y., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02552-9 · Cited by 156 · see citing works
Order-free Medicine Combination Prediction with Graph Convolutional Reinforcement Learning
Wang, S., Ren, P., Chen, Z., Ren, Z., Ma, J., & de Rijke, M. (2019) · Proceedings of the 28th ACM International Conference on Information and Knowledge Management · doi:10.1145/3357384.3357965 · Cited by 76 · see citing works
Yuan, H., Kang, L., Li, Y., & Fan, Z. (2024) · Medicine Advances · doi:10.1002/med4.70 · Cited by 32 · see citing works
- Authors
- Nemati, S., Ghassemi, M. M., & Clifford, G. D.
- Affiliations
- Emory University (US)
- Massachusetts Institute of Technology (US)
- Georgia Institute of Technology (US)
- Venue
- 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
- Publisher
- IEEE
- Published
- 2016-08
- Volume
- 2016
- Pages
- 2978-2981
- ISSN
- 2375-7477 (ISSN-L)
- OpenAlex
- W2535327675
- Cited by
- 195
- References
- 17
- Citations by year
- 2026: 9 · 2025: 21 · 2024: 18 · 2023: 24 · 2022: 18 · 2021: 34 · 2020: 29 · 2019: 23 · 2018: 13 · 2017: 6
- FWCI
- 7.54
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Georgia Institute of Technology
- National Institutes of Health
- National Institute of Environmental Health Sciences (K01ES025445)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- heparin dosing, deep reinforcement learning, intensive care unit, anticoagulation, therapeutic window, electronic health records, clinical decision support
- MeSH terms
- Algorithms, Dose-Response Relationship, Drug, Heparin, Humans, Learning, Length of Stay, Markov Chains, Reinforcement, Psychology, Retrospective Studies, Databases, Factual
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
