Cited by 266Health Systems, Economic Evaluations, Quality of LifeProbability and Statistical ResearchComplex Systems and Decision Making

The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index.

Markov Decision Processes: A Tool for Sequential Decision Making under Uncertainty

Open access

Alagoz, Oguzhan, Hsu, Heather, Schaefer, Andrew J., Roberts, Mark S.·2010·Medical Decision Making

DOI 10.1177/0272989x09353194PMC PMC3060044

Open this paper in CoChat — free account

Abstract

We provide a tutorial on the construction and evaluation of Markov decision processes (MDPs), which are powerful analytical tools used for sequential decision making under uncertainty that have been widely used in many industrial and manufacturing applications but are underutilized in medical decision making (MDM). We demonstrate the use of an MDP to solve a sequential clinical treatment problem under uncertainty. Markov decision processes generalize standard Markov models in that a decision process is embedded in the model and multiple decisions are made over time. Furthermore, they have significant advantages over standard decision analysis. We compare MDPs to standard Markov-based simulation models by solving the problem of the optimal timing of living-donor liver transplantation using both methods. Both models result in the same optimal transplantation policy and the same total life expectancies for the same patient and living donor. The computation time for solving the MDP model is significantly smaller than that for solving the Markov model. We briefly describe the growing literature of MDPs applied to medical decisions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1177/0272989x09353194

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Dynamic Programming
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Cost-Effectiveness in Health and Medicine
(1996) · doi:10.1093/oso/9780195108248.001.0001 · Cited by 7,193
Markov Decision Processes
Puterman, M. L. (1994) · Wiley series in probability and statistics · doi:10.1002/9780470316887 · Cited by 6,057
Approximate Dynamic Programming
Powell, W. B. (2011) · Wiley series in probability and statistics · doi:10.1002/9781118029176 · Cited by 2,359
Approximate Dynamic Programming
Powell, W. B. (2007) · Wiley series in probability and statistics · doi:10.1002/9780470182963 · Cited by 1,582
Dynamic Programming and Stochastic Control
Bertsekas, D. P., & White, C. C. (1977) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1977.4309612 · Cited by 1,211
The Markov process in medical prognosis
Beck, J. R., & Pauker, S. G. (1983) · Medical Decision Making · doi:10.1177/0272989x8300300403 · Cited by 957
Operations research : applications and algorithms
Winston, W. L. (1994) · Cited by 638
Dynamic programming: Models and applications
Denardo, E. V. (1982) · CERN Document Server (European Organization for Nuclear Research) · Cited by 411
Survival after liver transplantation in the United States: A disease-specific analysis of the UNOS database
Roberts, M. S., Angus, D. C., Bryce, C. L., Valenta, Z., & Weissfeld, L. A. (2004) · Liver Transplantation · doi:10.1002/lt.20137 · Cited by 380
Self-citation

Show all 34 references

Research works printed but not linked: 5

Bellman RE, 1957, Dynamic Programming
Faissol D., 2006, Swann J. Timing of Testing and Treatment of Hepatitis C and Other Diseases
Kreke JE, 2007, Decisions for Patients with Pneumonia-Related Sepsis [PhD dissertation]
Kurt M., At what lipid ratios should a patient with type 2 diabetes initiate statins?
2006, Pro T Suite [computer program]. Version release 1

Sorted by citations

Active inference and learning
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., O⿿Doherty, J., & Pezzulo, G. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.022 · Cited by 674
State-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3
Siebert, U., Alagoz, O., Bayoumi, A. M., Jahn, B., Owens, D. K., Cohen, D. J., & Kuntz, K. M. (2012) · Value in Health · doi:10.1016/j.jval.2012.06.014 · Cited by 495
Decision Making in Health and Medicine: Integrating Evidence and Values
Hunink, M. G. M., Weinstein, M. C., Wittenberg, E., Drummond, M. F., Pliskin, J. S., Wong, J. B., & Glasziou, P. P. (2014) · doi:10.1017/cbo9781139506779 · Cited by 425
Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach
Bennett, C. C., & Hauser, K. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.12.003 · Cited by 378
“Markov decision processes (MDPs) are one efficient technique for determining such optimal sequential decisions (termed a “policy”) in dynamic and uncertain environments [18,19], and have been explored in medical decision-making problems in recent years [18,20].”
Influential
State-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force–3
Siebert, U., Alagoz, O., Bayoumi, A. M., Jahn, B., Owens, D. K., Cohen, D. J., & Kuntz, K. M. (2012) · Medical Decision Making · doi:10.1177/0272989x12455463 · Cited by 348
Hepatology
Schenker, S., Perkins, H. S., & Sorrell, M. F. (2006) · Hepatology · doi:10.1002/(issn)1527-3350 · Cited by 262
Artificial Intelligence in Pathology
Chang, H. Y., Jung, C. K., Woo, J. I., Lee, S.-H., Cho, J., Kim, S. W., & Kwak, T.-Y. (2018) · Journal of Pathology and Translational Medicine · doi:10.4132/jptm.2018.12.16 · Cited by 256
Confronting Deep Uncertainties in Risk Analysis
Cox, L. A. (2012) · Risk Analysis · doi:10.1111/j.1539-6924.2012.01792.x · Cited by 211
A review on lubricant condition monitoring information analysis for maintenance decision support
Wakiru, J., Pintelon, L. M., Muchiri, P., & Chemweno, P. K. (2018) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2018.08.039 · Cited by 207
Active Inference: Demystified and Compared
Sajid, N., Ball, P. J., Parr, T., & Friston, K. J. (2021) · Neural Computation · doi:10.1162/neco_a_01357 · Cited by 197
Development and validation of a reinforcement learning algorithm to dynamically optimize mechanical ventilation in critical care
Peine, A., Hallawa, A., Bickenbach, J., Dartmann, G., Fazlic, L. B., Schmeink, A., Ascheid, G., Thiemermann, C., Schuppert, A., Kindle, R. D., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00388-6 · Cited by 165
Computational Phenotyping in Psychiatry: A Worked Example
Schwartenbeck, P., & Friston, K. J. (2016) · eNeuro · doi:10.1523/eneuro.0049-16.2016 · Cited by 134
An evidential Markov decision making model
He, Z., & Jiang, W. R. (2018) · Information Sciences · doi:10.1016/j.ins.2018.08.013 · Cited by 103
From data to optimal decision making: a data-driven, probabilistic machine learning approach to decision support for patients with sepsis
Tsoukalas, A., Albertson, T., & Tagkopoulos, I. (2015) · JMIR Medical Informatics · doi:10.2196/medinform.3445 · Cited by 96
“MDP has been used for decision support in determining liver acceptance for liver transplants [18-20], HIV therapy initiation [21], breast cancer screening [22], treatment of hepatitis C [23], statin therapy timing [24], among others [25].”
Deep Reinforcement Learning Approach for Trading Automation in the Stock Market
Kabbani, T., & Duman, E. (2022) · IEEE Access · doi:10.1109/access.2022.3203697 · Cited by 88
Optimization of anemia treatment in hemodialysis patients via reinforcement learning
Escandell-Montero, P., Chermisi, M., Martínez-Martínez, J. M., Gómez‐Sanchís, J., Barbieri, C., Soria‐Olivas, E., Mari, F., Vila‐Francés, J., Stopper, A., Gatti, E., et al. (2014) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2014.07.004 · Cited by 87
Artificial Intelligence in Plastic Surgery: Applications and Challenges
Liang, X., Yang, X., Yin, S., Malay, S., Chung, K. C., Ma, J., & Wang, K. (2020) · Aesthetic Plastic Surgery · doi:10.1007/s00266-019-01592-2 · Cited by 88
Robust Markov Decision Processes: Beyond Rectangularity
Goyal, V., & Grand-Clément, J. (2022) · Mathematics of Operations Research · doi:10.1287/moor.2022.1259 · Cited by 82
Opportunistic decision making and complexity in emergency care
Franklin, A., Liu, Y., Li, Z., Nguyen, V., Johnson, T. R., Robinson, D. J., Okafor, N. G., King, B. R., Patel, V. L., & Zhang, J. (2011) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.04.001 · Cited by 82
MDP and Machine Learning-Based Cost-Optimization of Dynamic Resource Allocation for Network Function Virtualization
Shi, R., Zhang, J., Chu, W., Bao, Q., Jin, X., Gong, C., Zhu, Q., Yu, C., & Rosenberg, S. (2015) · 2015 IEEE International Conference on Services Computing · doi:10.1109/scc.2015.19 · Cited by 74
Markov Decision Processes for Screening and Treatment of Chronic Diseases
Steimle, L. N., & Denton, B. T. (2017) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-319-47766-4_6 · Cited by 74
UAV Path Planning Based on the Average TD3 Algorithm With Prioritized Experience Replay
Luo, X., Wang, Q., Gong, H., & Tang, C. (2024) · IEEE Access · doi:10.1109/access.2024.3375083 · Cited by 71
Value of information analysis in civil and infrastructure engineering: a review
Zhang, W., Lü, D., Qin, J., Thöns, S., & Faber, M. H. (2021) · Journal of Infrastructure Preservation and Resilience · doi:10.1186/s43065-021-00027-0 · Cited by 70
Elective Patient Admission and Scheduling under Multiple Resource Constraints
Barz, C., & Rajaram, K. (2015) · Production and Operations Management · doi:10.1111/poms.12395 · Cited by 69
The decision-making process of genetically at-risk couples considering preimplantation genetic diagnosis: Initial findings from a grounded theory study
Hershberger, P. E., Gallo, A. M., Kavanaugh, K. L., Olshansky, E. F., Schwartz, A., & Tur-Kaspa, I. (2012) · Social Science & Medicine · doi:10.1016/j.socscimed.2012.02.003 · Cited by 66
Flexible expansion planning of distribution system integrating multiple renewable energy sources: An approximate dynamic programming approach
Sun, Q., Wu, Z. S., Gu, W., Zhu, T., Zhong, L., & Gao, T. (2021) · Energy · doi:10.1016/j.energy.2021.120367 · Cited by 63
Markov Decision Process for Image-Guided Additive Manufacturing
Yao, B., Imani, F., & Yang, H. Y. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2839973 · Cited by 63
District Cooling System Control for Providing Operating Reserve Based on Safe Deep Reinforcement Learning
Yu, P., Zhang, H., Song, Y., Hui, H., & Chen, G. (2023) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2023.3237888 · Cited by 58
Identifying Appropriate Recipients for CDC Infectious Risk Donor Kidneys
Chow, E. K. H., Massie, A. B., Muzaale, A. D., Singer, A. L., Kucirka, L. M., Montgomery, R. A., Lehmann, H. P., & Segev, D. L. (2013) · American Journal of Transplantation · doi:10.1002/ajt.12206 · Cited by 55
Guidance and control of autonomous surface underwater vehicles for target tracking in ocean environment by deep reinforcement learning
Song, D., Gan, W., Yao, P., Zang, W., Zhang, Z., & Qu, X. (2022) · Ocean Engineering · doi:10.1016/j.oceaneng.2022.110947 · Cited by 53
Personalized screening intervals for biomarkers using joint models for longitudinal and survival data
Rizopoulos, D., Taylor, J. M. G., van Rosmalen, J., Steyerberg, E. W., & Takkenberg, J. J. M. (2015) · Biostatistics · doi:10.1093/biostatistics/kxv031 · Cited by 52
Cache-Assisted Collaborative Task Offloading and Resource Allocation Strategy: A Metareinforcement Learning Approach
Chen, S., Rui, L., Gao, Z., Li, W., & Qiu, X. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3168885 · Cited by 51
Estimated prevalence of undiagnosed HCV infected individuals in Italy: A mathematical model by route of transmission and fibrosis progression
Kondili, L. A., Andreoni, M., Alberti, A., Lobello, S., Babudieri, S., Roscini, A. S., Merolla, R. C. D., Marrocco, W., & Craxı̀, A. (2021) · Epidemics · doi:10.1016/j.epidem.2021.100442 · Cited by 51
Cost-effectiveness of one-time genetic testing to minimize lifetime adverse drug reactions
Alagöz, O., Durham, D., & Kasirajan, K. (2015) · The Pharmacogenomics Journal · doi:10.1038/tpj.2015.39 · Cited by 50
Multi-Objective Markov Decision Processes for Data-Driven Decision Support.
Lizotte, D. J., & Laber, E. B. (2016) · PubMed · Cited by 50
A Deep Learning Approach for Task Offloading in Multi-UAV Aided Mobile Edge Computing
Ebrahim, M. A., Ebrahim, G. A., Mohamed, H. K., & Abdellatif, S. O. (2022) · IEEE Access · doi:10.1109/access.2022.3208584 · Cited by 45
Optimization of PSA Screening Policies
Zhang, J., Denton, B. T., Balasubramanian, H., Shah, N. D., & Inman, B. A. (2011) · Medical Decision Making · doi:10.1177/0272989x11416513 · Cited by 45
Clinical Course among Cases of Acute Liver Failure of Indeterminate Diagnosis
Li, R., Belle, S. H., Horslen, S. P., Chen, L.-W., Zhang, S., Squires, R. H., Bukauskas, K., Narkewicz, M. R., Hite, M., Loomes, K. M., et al. (2016) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2015.12.065 · Cited by 45
The structure of optimal statin initiation policies for patients with Type 2 diabetes
Kurt, M., Denton, B. T., Schaefer, A. J., Shah, N. D., & Smith, S. A. (2011) · IIE Transactions on Healthcare Systems Engineering · doi:10.1080/19488300.2010.550180 · Cited by 42
Self-citation
Transplantation at the Nexus of Behavioral Economics and Health Care Delivery
Schnier, K. E., Cox, J. C., McIntyre, C. W., Ruhil, R., Sadiraj, V., & Turgeon, N. A. (2012) · American Journal of Transplantation · doi:10.1111/j.1600-6143.2012.04343.x · Cited by 42
Healthcare management through organizational simulation
Basole, R. C., Bodner, D. A., & Rouse, W. B. (2012) · Decision Support Systems · doi:10.1016/j.dss.2012.10.012 · Cited by 41
Reinforcement learning-based expanded personalized diabetes treatment recommendation using South Korean electronic health records
Oh, S. H., Park, J., Lee, S. J., Kang, S. S., & Mo, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117932 · Cited by 41
Applications of machine learning in decision analysis for dose management for dofetilide
Levy, A. E., Biswas, M., Weber, R. A., Tarakji, K. G., Chung, M. K., Noseworthy, P. A., Newton‐Cheh, C., & Rosenberg, M. A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0227324 · Cited by 41
Handling decision-making in Intuitionistic Fuzzy environment: PyIFDM package
Więckowski, J., Kizielewicz, B., & Sałabun, W. (2023) · SoftwareX · doi:10.1016/j.softx.2023.101344 · Cited by 40
Choosing the Order of Deceased Donor and Living Donor Kidney Transplantation in Pediatric Recipients
Arendonk, K. J. V., Chow, E. K. H., James, N. T., Orandi, B. J., Ellison, T. A., Smith, J. M., Colombani, P. M., & Segev, A. D. L. (2015) · Transplantation · doi:10.1097/tp.0000000000000588 · Cited by 38
Long-term adaption decisions via fully and partially observable Markov decision processes
Špačková, O., & Straub, D. (2017) · Sustainable and Resilient Infrastructure · doi:10.1080/23789689.2017.1278995 · Cited by 37
From Data to Improved Decisions: Operations Research in Healthcare Delivery
Capan, M., Khojandi, A., Denton, B. T., Williams, K. D., Ayer, T., Chhatwal, J., Kurt, M., Mason, J. E., Roberts, M. S., Zaric, G. S., et al. (2017) · Medical Decision Making · doi:10.1177/0272989x17705636 · Cited by 37
Self-citation
Economic Assessment of an Anti-HCV Screening Program in Italy
Ruggeri, M., Coretti, S., Gasbarrini, A., & Cicchetti, A. (2013) · Value in Health · doi:10.1016/j.jval.2013.07.005 · Cited by 36
A new belief Markov chain model and its application in inventory prediction
He, Z., & Jiang, W. R. (2017) · International Journal of Production Research · doi:10.1080/00207543.2017.1405166 · Cited by 36
The applications and prospects of big data in perioperative anesthetic management
Zhu, Y., Liu, X., Li, Y., & Yi, B. (2024) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-024-00068-0 · Cited by 33
Standoff tracking control of underwater glider to moving target
Zang, W., Yao, P., & Song, D. (2021) · Applied Mathematical Modelling · doi:10.1016/j.apm.2021.09.011 · Cited by 33
Lubricating Oil Remaining Useful Life Prediction Using Multi-Output Gaussian Process Regression
Tanwar, M., & Raghavan, N. (2020) · IEEE Access · doi:10.1109/access.2020.3008328 · Cited by 31
Optimizing the dynamic treatment regime of in-hospital warfarin anticoagulation in patients after surgical valve replacement using reinforcement learning
Zeng, J., Shao, J., Lin, S., Zhang, H., Su, X., Lian, X., Zhao, Y., Ji, X., & Zheng, Z. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac088 · Cited by 30
Modeling cost‐effectiveness and health gains of a “universal” versus “prioritized” hepatitis C virus treatment policy in a real‐life cohort
Kondili, L. A., Romano, F., Rolli, F. R., Ruggeri, M., Rosato, S., Brunetto, M. R., Zignego, A. L., Ciancio, A., Leo, A. D., Raimondo, G., et al. (2017) · Hepatology · doi:10.1002/hep.29399 · Cited by 29
Human-Augmented Prescriptive Analytics With Interactive Multi-Objective Reinforcement Learning
Lepenioti, K., Bousdekis, A., Apostolou, D., & Mentzas, G. N. (2021) · IEEE Access · doi:10.1109/access.2021.3096662 · Cited by 31
The Decision to Emigrate: A Simulation Model Based on the Theory of Planned Behaviour
Willekens, F. (2016) · The Springer series on demographic methods and population analysis · doi:10.1007/978-3-319-32283-4_10 · Cited by 26
Modelling and solving resource allocation problems via a dynamic programming approach
Forootani, A., Tipaldi, M., Zarch, M. G., Liuzza, D., & Glielmo, L. (2019) · International Journal of Control · doi:10.1080/00207179.2019.1661521 · Cited by 24
Expert Systems in Mental Health Care
Bennett, C. C., & Doub, T. W. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-420248-1.00002-7 · Cited by 6
A radiobiological Markov simulation tool for aiding decision making in proton therapy referral
Austin, A. M., Douglass, M. J. J., Nguyen, G. T., & Penfold, S. N. (2017) · Physica Medica · doi:10.1016/j.ejmp.2017.11.013 · Cited by 24
PrescDRL: deep reinforcement learning for herbal prescription planning in treatment of chronic diseases
Yang, K., Yu, Z., Su, X., Zhang, F., He, X., Wang, N., Zheng, Q., Yu, F., Wen, T., & Zhou, X. (2024) · Chinese Medicine · doi:10.1186/s13020-024-01005-w · Cited by 26
Traded Control of Human–Machine Systems for Sequential Decision-Making Based on Reinforcement Learning
Zhang, Q., Kang, Y., Zhao, Y., Li, P., & You, S. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3127857 · Cited by 23
Robustness of Proactive Intensive Care Unit Transfer Policies
Grand-Clément, J., Chan, C. W., Goyal, V., & Escobar, G. J. (2022) · Operations Research · doi:10.1287/opre.2022.2403 · Cited by 23
Decision modeling in donation after circulatory death liver transplantation
McLean, K. A., Camilleri‐Brennan, J., Knight, S. R., Drake, T. M., Ots, R., Shaw, C. A., Wigmore, S. J., & Harrison, E. M. (2016) · Liver Transplantation · doi:10.1002/lt.24715 · Cited by 22
An Intelligent Algorithm for USVs Collision Avoidance Based on Deep Reinforcement Learning Approach with Navigation Characteristics
Sun, Z., Fan, Y., & Wang, G. (2023) · Journal of Marine Science and Engineering · doi:10.3390/jmse11040812 · Cited by 21
DRL-Based Load-Balancing Routing Scheme for 6G Space–Air–Ground Integrated Networks
Dong, F., Song, J., Zhang, Y., Wang, Y., & Huang, T. (2023) · Remote Sensing · doi:10.3390/rs15112801 · Cited by 21
A MDP Model for Breast and Ovarian Cancer Intervention Strategies for BRCA1/2 Mutation Carriers
Abdollahian, M., & Das, T. K. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2014.2319246 · Cited by 21
Optimization of Sequential Decision Making for Chronic Diseases: From Data to Decisions
Denton, B. T. (2018) · doi:10.1287/educ.2018.0184 · Cited by 20
Optimal treatment recommendations for diabetes patients using the Markov decision process along with the South Korean electronic health records
Oh, S. H., Lee, S. J., Noh, J., & Mo, J. (2021) · Scientific Reports · doi:10.1038/s41598-021-86419-4 · Cited by 20
Data-Driven Markov Decision Process Approximations for Personalized Hypertension Treatment Planning
Schell, G. J., Marrero, W. J., Lavieri, M. S., Sussman, J. B., & Hayward, R. A. (2016) · MDM Policy & Practice · doi:10.1177/2381468316674214 · Cited by 20
Sensitivity Analysis in Sequential Decision Models
Chen, Q., Ayer, T., & Chhatwal, J. (2016) · Medical Decision Making · doi:10.1177/0272989x16670605 · Cited by 19
An End-to-End Deep Reinforcement Learning Approach for Job Shop Scheduling
Zhao, L., Shen, W., Zhang, C., & Peng, K. (2022) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd54268.2022.9776116 · Cited by 18
Information Entropy in Predicting Location of Observation Points for Long Tunnel
Xu, C. G., Hu, C., Liu, X., & Wang, S. (2017) · Entropy · doi:10.3390/e19070332 · Cited by 18
Establishing the Value of Diagnostic and Prognostic Tests in Health Technology Assessment
Soares, M. O., Walker, S., Palmer, S., & Sculpher, M. (2018) · Medical Decision Making · doi:10.1177/0272989x17749829 · Cited by 18
What Is the Optimal Threshold at Which to Recommend Breast Biopsy?
Burnside, E. S., Chhatwal, J., & Alagöz, O. (2012) · PLoS ONE · doi:10.1371/journal.pone.0048820 · Cited by 18
Optimal Policy Learning for Disease Prevention Using Reinforcement Learning
Khan, Z. A., Feng, Z., Uddin, M. I., Mast, N., Shah, S. A. A., Imtiaz, M., Al‐Khasawneh, M. A., & Mahmoud, M. (2020) · Scientific Programming · doi:10.1155/2020/7627290 · Cited by 17
Constrained Markov Decision Process Modeling for Optimal Sensing of Cardiac Events in Mobile Health
Yao, B., Chen, Y., & Yang, H. Y. (2021) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2021.3052483 · Cited by 17
Exploring Potential Energy Surfaces Using Reinforcement Machine Learning
Mills, A. W., Goings, J. J., Beck, D. A. C., Yang, C., & Li, X. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c00373 · Cited by 18
Optimisation and control of the supply of blood bags in hemotherapic centres via Markov decision process with discounted arrival rate
Soares, H., Arruda, E. F., Bahiense, L., Gärtner, D., & Filho, L. A. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101791 · Cited by 16
Realizing Midcourse Penetration With Deep Reinforcement Learning
Liang, J., Nan, Y., & Li, Z. (2021) · IEEE Access · doi:10.1109/access.2021.3091605 · Cited by 16
Operations research applications in hospital operations: Part I
Abe, T. K., Beamon, B. M., Storch, R. L., & Agus, J. (2016) · IIE Transactions on Healthcare Systems Engineering · doi:10.1080/19488300.2015.1134727 · Cited by 16
A Simple Reinforcement Learning Mechanism for Resource Allocation in LTE-A Networks with Markov Decision Process and Q-LearningPreprint
Santos, E. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.09312 · Cited by 16
Goals and Objectives to Optimize the Value of an Acute Pain Service in Perioperative Pain Management
Le-Wendling, L. T., Glick, W., & Tighe, P. J. (2017) · Techniques in Orthopaedics · doi:10.1097/bto.0000000000000245 · Cited by 15
A finite-horizon Markov decision process model for cancer chemotherapy treatment planning: an application to sequential treatment decision making in clinical trials
Bazrafshan, N., & lotfi, M. M. (2020) · Annals of Operations Research · doi:10.1007/s10479-020-03706-5 · Cited by 15
AI-Driven Decision Making for Auxiliary Diagnosis of Epidemic Diseases
Lin, K., Liu, J., & Gao, J. (2021) · IEEE Transactions on Molecular Biological and Multi-Scale Communications · doi:10.1109/tmbmc.2021.3120646 · Cited by 15
Cost-Effectiveness Analysis of Early Treatment of Chronic HCV with Sofosbuvir/Velpatasvir in Italy
Ruggeri, M., Romano, F., Basile, M., Coretti, S., Rolli, F. R., Drago, C., & Cicchetti, A. (2018) · Applied Health Economics and Health Policy · doi:10.1007/s40258-018-0410-x · Cited by 15
Predicting engine oil degradation across diverse vehicles and identifying key factors
Omiya, T., Hanyuda, K., & Nagatomi, E. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112524 · Cited by 14
Risk‐based breast cancer follow‐up stratified by age
Witteveen, A., Otten, J. W. M., Vliegen, I. M. H., Siesling, S., Timmer, J., & IJzerman, M. J. (2018) · Cancer Medicine · doi:10.1002/cam4.1760 · Cited by 14
A Survey on Event Prediction Methods from a Systems Perspective: Bringing Together Disparate Research Areas
Benzin, J.-V., & Rinderle‐Ma, S. (2025) · ACM Computing Surveys · doi:10.1145/3743672 · Cited by 13
Dynamic Transitions of Pediatric Sepsis: A Markov Chain Analysis
Kausch, S. L., Lobo, J. M., Spaeder, M. C., Sullivan, B. A., & Keim‐Malpass, J. (2021) · Frontiers in Pediatrics · doi:10.3389/fped.2021.743544 · Cited by 12
A Markov Decision Processes Modeling for Curricular Analytics
Slim, A., Yusuf, H. A., Abbas, N., Abdallah, C. T., Heileman, G. L., & Slim, A. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00071 · Cited by 12
A Promising Approach to Optimizing Sequential Treatment Decisions for Depression: Markov Decision Process
Li, F., Jörg, F., Li, X., & Feenstra, T. (2022) · PharmacoEconomics · doi:10.1007/s40273-022-01185-z · Cited by 12
Computer Disease Simulation Models
Rutter, C. M., Knudsen, A. B., & Pandharipande, P. V. (2011) · Academic Radiology · doi:10.1016/j.acra.2011.02.004 · Cited by 12
Decision-Making Process Regarding Fertility Among Reproductive-Age Women With Cancer in Taiwan
Huang, S., Hsieh, P.-L., Hsiung, Y., Tseng, L., Chen, P., & Hung, C. (2016) · Cancer Nursing · doi:10.1097/ncc.0000000000000439 · Cited by 12
Was Angelina Jolie Right? Optimizing Cancer Prevention Strategies Among BRCA Mutation Carriers
Nohdurft, E., Long, E. F., & Spinler, S. (2017) · Decision Analysis · doi:10.1287/deca.2017.0352 · Cited by 12
Studying sequential processes of criminal defendant decision-making using a choose-your-own-adventure research paradigm
Frazier, A., & Gonzales, J. E. (2021) · Psychology Crime and Law · doi:10.1080/1068316x.2021.1972107 · Cited by 14
Operations research applications in hospital operations: Part III
Abe, T. K., Beamon, B. M., Storch, R. L., & Agus, J. (2016) · IIE Transactions on Healthcare Systems Engineering · doi:10.1080/19488300.2016.1199613 · Cited by 12
Markov chain evaluation of acute postoperative pain transition states
Tighe, P. J., Bzdega, M., Fillingim, R. B., Rashidi, P., & Aytuğ, H. (2015) · Pain · doi:10.1097/j.pain.0000000000000429 · Cited by 11
Automation Reliability and Decision Strategy: A Sequential Decision Making Model for Automation Interaction
Acharya, A., Howes, A., Baber, C., & Marshall, T. (2018) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931218621033 · Cited by 14
An integrated supply chain risk mitigation tool – model, analysis and insights
Balan, S., & Handfield, R. (2024) · International Journal of Management Science and Engineering Management · doi:10.1080/17509653.2024.2441783 · Cited by 10
Stochastic Artificial Intelligence benefits and Supply Chain Management inventory prediction
Haoud, N. E., & Bachiri, Z. (2019) · International Colloquium on Logistics and Supply Chain Management (LOGISTIQUA) · doi:10.1109/logistiqua.2019.8907271 · Cited by 10

Authors: Alagoz, O., Hsu, H., Schaefer, A. J., & Roberts, M. S.
Affiliations: Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI,
University of Pittsburgh (US)
ORCID: Hsu, Heather
Schaefer, Andrew J.
Roberts, Mark S.
Venue: Medical Decision Making
Publisher: SAGE Publications
Published: 2009-12-31
Volume: 30
Issue: 4
Pages: 474-483
ISSN: 0272-989X (ISSN-L)
1552-681X
DOI: 10.1177/0272989x09353194
PMC: PMC3060044
OpenAlex: W2036336974
Cited by: 266
References: 34
Citations by year: 2026: 15 · 2025: 18 · 2024: 22 · 2023: 21 · 2022: 28 · 2021: 24 · 2020: 16 · 2019: 12 · 2018: 26 · 2017: 16 · 2016: 14 · 2015: 15 · 2014: 8 · 2013: 5 · 2012: 16
FWCI: 17.22
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Medical Decision Making (publishedVersion)
PubMed (publishedVersion)
PubMed Central (acceptedVersion)
RePEc: Research Papers in Economics (submittedVersion)
Funders: National Science Foundation
U.S. National Library of Medicine (R21LM008273)
National Center for Research Resources (UL1RR025011)
Division of Civil, Mechanical and Manufacturing Innovation
Topics: Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
Social Sciences › Decision Sciences › Management Science and Operations Research › Complex Systems and Decision Making
Keywords: Markov decision process, sequential decision making under uncertainty, living donor liver transplantation, Markov model, decision analysis, computational efficiency, life expectancy
MeSH terms: Decision Making, Humans, Markov Chains, Liver Transplantation, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Markov Decision Processes: A Tool for Sequential Decision Making under Uncertainty

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks