Markov decision processes: a tool for sequential decision making under uncertainty
Open accessAbstract
We provide a tutorial on the construction and evaluation of Markov decision processes (MDPs), which are powerful analytical tools used for sequential decision making under uncertainty that have been widely used in many industrial and manufacturing applications but are underutilized in medical decision making (MDM). We demonstrate the use of an MDP to solve a sequential clinical treatment problem under uncertainty. Markov decision processes generalize standard Markov models in that a decision process is embedded in the model and multiple decisions are made over time. Furthermore, they have significant advantages over standard decision analysis. We compare MDPs to standard Markov-based simulation models by solving the problem of the optimal timing of living-donor liver transplantation using both methods. Both models result in the same optimal transplantation policy and the same total life expectancies for the same patient and living donor. The computation time for solving the MDP model is significantly smaller than that for solving the Markov model. We briefly describe the growing literature of MDPs applied to medical decisions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There have been relatively few applications of sequential decision-making techniques like MDPs in healthcare.
“However, despite such applicability of sequential decisionmaking techniques like MDPs to medical decision-making, there have been relatively few applications in healthcare [18,19].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [19]
Markov decision processes (MDPs) are one efficient technique for determining optimal sequential decisions in dynamic and uncertain environments.
“Markov decision processes (MDPs) are one efficient technique for determining such optimal sequential decisions (termed a 'policy') in dynamic and uncertain environments [18,19], and have been explored in medical decision-making problems in recent years [18,20].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [19]
Other methods such as Markov decision processes generalize STMs by allowing embedding of sequential decisions, so that multiple decisions can be made in multiple time periods.
“Other methods such as Markov decision processes generalize STMs by allowing embedding of sequential decisions, and thus, multiple decisions can be made in multiple time periods [39,40].”
From State-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3 (2012) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
Cost-Effectiveness in Health and Medicine
(1996) · doi:10.1093/oso/9780195108248.001.0001 · Cited by 7,193 · see citing works
Puterman, M. L. (1994) · Wiley series in probability and statistics · doi:10.1002/9780470316887 · Cited by 6,057 · see citing works
Approximate Dynamic Programming
Powell, W. B. (2011) · Wiley series in probability and statistics · doi:10.1002/9781118029176 · Cited by 2,359 · see citing works
Approximate Dynamic Programming
Powell, W. B. (2007) · Wiley series in probability and statistics · doi:10.1002/9780470182963 · Cited by 1,582 · see citing works
Dynamic Programming and Stochastic Control
Bertsekas, D. P., & White, C. C. (1977) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1977.4309612 · Cited by 1,211 · see citing works
The Markov process in medical prognosis
Beck, J. R., & Pauker, S. G. (1983) · Medical Decision Making · doi:10.1177/0272989x8300300403 · Cited by 957 · see citing works
Operations research : applications and algorithms
Winston, W. L. (1994) · Cited by 638 · see citing works
Dynamic programming: Models and applications
Denardo, E. V. (1982) · CERN Document Server (European Organization for Nuclear Research) · Cited by 409 · see citing works
Roberts, M. S., Angus, D. C., Bryce, C. L., Valenta, Z., & Weissfeld, L. A. (2004) · Liver Transplantation · doi:10.1002/lt.20137 · Cited by 380 · see citing works
Self-citationOperations Research: Applications and Algorithms
Zieyel, E. R. (1988) · Technometrics · doi:10.1080/00401706.1988.10488426 · Cited by 360 · see citing works
Primer on Medical Decision Analysis: Part 1—Getting Started
Detsky, A. S., Naglie, G., Krahn, M. D., Naimark, D. M. J., & Redelmeier, D. A. (1997) · Medical Decision Making · doi:10.1177/0272989x9701700201 · Cited by 230 · see citing works
The Optimal Timing of Living-Donor Liver Transplantation
Alagöz, O., Maillart, L. M., Schaefer, A. J., & Roberts, M. S. (2004) · Management Science · doi:10.1287/mnsc.1040.0287 · Cited by 213 · see citing works
Self-citationMotion Planning Under Uncertainty for Image-guided Medical Needle Steering
Alterovitz, R., Branicky, M. S., & Goldberg, K. (2008) · The International Journal of Robotics Research · doi:10.1177/0278364908097661 · Cited by 188 · see citing works
Planning treatment of ischemic heart disease with partially observable Markov decision processes
Hauskrecht, M., & Fraser, H. (2000) · Artificial Intelligence in Medicine · doi:10.1016/s0933-3657(99)00042-1 · Cited by 176 · see citing works
Assessing Dynamic Breast Cancer Screening Policies
Maillart, L. M., Ivy, J. S., Ransom, S. B., & Diehl, K. M. (2008) · Operations Research · doi:10.1287/opre.1080.0614 · Cited by 163 · see citing works
Modeling Medical Treatment Using Markov Decision Processes
Schaefer, A. J., Bailey, M. D., Shechter, S. M., & Roberts, M. S. (2005) · International Series in Operations Research & Management Science · doi:10.1007/1-4020-8066-2_23 · Cited by 155 · see citing works
Self-citationThe Optimal Time to Initiate HIV Therapy Under Ordered Health States
Shechter, S. M., Bailey, M. D., Schaefer, A. J., & Roberts, M. S. (2008) · Operations Research · doi:10.1287/opre.1070.0480 · Cited by 148 · see citing works
Self-citationDetermining the Acceptance of Cadaveric Livers Using an Implicit Model of the Waiting List
Alagöz, O., Maillart, L. M., Schaefer, A. J., & Roberts, M. S. (2007) · Operations Research · doi:10.1287/opre.1060.0329 · Cited by 145 · see citing works
Self-citationDynamic programming: Models and applications
Wijngaard, J. (1984) · European Journal of Operational Research · doi:10.1016/0377-2217(84)90117-6 · Cited by 101 · see citing works
Ahn, J., & Hornberger, J. C. (1996) · Management Science · doi:10.1287/mnsc.42.5.629 · Cited by 89 · see citing works
Optimizing the Start Time of Statin Therapy for Patients with Diabetes
Denton, B. T., Kurt, M., Shah, N. D., Bryant, S. C., & Smith, S. A. (2009) · Medical Decision Making · doi:10.1177/0272989x08329462 · Cited by 88 · see citing works
Estimating the Patient's Price of Privacy in Liver Transplantation
Sandıkçı, B., Maillart, L. M., Schaefer, A. J., Alagöz, O., & Roberts, M. S. (2008) · Operations Research · doi:10.1287/opre.1080.0648 · Cited by 81 · see citing works
Self-citationChoosing Among Living-Donor and Cadaveric Livers
Alagöz, O., Maillart, L. M., Schaefer, A. J., & Roberts, M. S. (2007) · Management Science · doi:10.1287/mnsc.1070.0726 · Cited by 78 · see citing works
Self-citationOptimal Control of a Birth and Death Epidemic Process
Lefèvre, C. (1981) · Operations Research · doi:10.1287/opre.29.5.971 · Cited by 58 · see citing works
Alagöz, O., Bryce, C. L., Shechter, S. M., Schaefer, A. J., Chang, C.-C. H., Angus, D. C., & Roberts, M. S. (2005) · Medical Decision Making · doi:10.1177/0272989x05282719 · Cited by 56 · see citing works
Self-citationComparison of Some Suboptimal Control Policies in Medical Drug Therapy
Hu, C., Lovejoy, W. S., & Shafer, S. L. (1996) · Operations Research · doi:10.1287/opre.44.5.696 · Cited by 52 · see citing works
Alterovitz, R., Branicky, M. S., & Goldberg, K. (2008) · Springer tracts in advanced robotics · doi:10.1007/978-3-540-68405-3_20 · Cited by 34 · see citing works
Modeling hospital discharge policies for patients with pneumonia-related sepsis
Kreke, J. E., Bailey, M. D., Schaefer, A. J., Angus, D. C., & Roberts, M. S. (2008) · IIE Transactions · doi:10.1080/07408170802165872 · Cited by 19 · see citing works
Self-citationOPTIMAL POLICIES FOR THE ACCEPTANCE OF LIVING- AND CADAVERIC-DONOR LIVERS
Alagöz, O. (2004) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 16 · see citing works
Markov process-based Monte Carlo simulation
Roberts, M. S. (1992) · Winter Simulation Conference · doi:10.1145/167293.167816 · Cited by 15 · see citing works
Self-citationModeling disease management decisions for patients with pneumonia-related sepsis
Kreke, J. E. (2007) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 9 · see citing works
35th Annual Meeting for the Society for Medical Decision Making
N/A (2014) · Medical Decision Making · doi:10.1177/0272989x13519751 · Cited by 3 · see citing works
Flagged first, then by citations
1 of the 34 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [8]Bellman RE Dynamic Programming 1957 Princeton University Press Princeton, NJ
Cited by
Sorted by citations
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., O⿿Doherty, J., & Pezzulo, G. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.022 · Cited by 674 · see citing works
State-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3
Siebert, U., Alagoz, O., Bayoumi, A. M., Jahn, B., Owens, D. K., Cohen, D. J., & Kuntz, K. M. (2012) · Value in Health · doi:10.1016/j.jval.2012.06.014 · Cited by 495 · see citing works
Decision Making in Health and Medicine: Integrating Evidence and Values
Hunink, M. G. M., Weinstein, M. C., Wittenberg, E., Drummond, M. F., Pliskin, J. S., Wong, J. B., & Glasziou, P. P. (2014) · doi:10.1017/cbo9781139506779 · Cited by 425 · see citing works
Bennett, C. C., & Hauser, K. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.12.003 · Cited by 378 · see citing works
“Markov decision processes (MDPs) are one efficient technique for determining such optimal sequential decisions (termed a “policy”) in dynamic and uncertain environments [18,19], and have been explored in medical decision-making problems in recent years [18,20].”
InfluentialState-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force–3
Siebert, U., Alagoz, O., Bayoumi, A. M., Jahn, B., Owens, D. K., Cohen, D. J., & Kuntz, K. M. (2012) · Medical Decision Making · doi:10.1177/0272989x12455463 · Cited by 348 · see citing works
Schenker, S., Perkins, H. S., & Sorrell, M. F. (2006) · Hepatology · doi:10.1002/(issn)1527-3350 · Cited by 262 · see citing works
Artificial Intelligence in Pathology
Chang, H. Y., Jung, C. K., Woo, J. I., Lee, S.-H., Cho, J., Kim, S. W., & Kwak, T.-Y. (2018) · Journal of Pathology and Translational Medicine · doi:10.4132/jptm.2018.12.16 · Cited by 249 · see citing works
Confronting Deep Uncertainties in Risk Analysis
Cox, L. A. (2012) · Risk Analysis · doi:10.1111/j.1539-6924.2012.01792.x · Cited by 211 · see citing works
A review on lubricant condition monitoring information analysis for maintenance decision support
Wakiru, J., Pintelon, L. M., Muchiri, P., & Chemweno, P. K. (2018) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2018.08.039 · Cited by 207 · see citing works
Active Inference: Demystified and Compared
Sajid, N., Ball, P. J., Parr, T., & Friston, K. J. (2021) · Neural Computation · doi:10.1162/neco_a_01357 · Cited by 178 · see citing works
Peine, A., Hallawa, A., Bickenbach, J., Dartmann, G., Fazlic, L. B., Schmeink, A., Ascheid, G., Thiemermann, C., Schuppert, A., Kindle, R. D., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00388-6 · Cited by 152 · see citing works
Computational Phenotyping in Psychiatry: A Worked Example
Schwartenbeck, P., & Friston, K. J. (2016) · eNeuro · doi:10.1523/eneuro.0049-16.2016 · Cited by 124 · see citing works
An evidential Markov decision making model
He, Z., & Jiang, W. R. (2018) · Information Sciences · doi:10.1016/j.ins.2018.08.013 · Cited by 103 · see citing works
Tsoukalas, A., Albertson, T., & Tagkopoulos, I. (2015) · JMIR Medical Informatics · doi:10.2196/medinform.3445 · Cited by 96 · see citing works
“MDP has been used for decision support in determining liver acceptance for liver transplants [18-20], HIV therapy initiation [21], breast cancer screening [22], treatment of hepatitis C [23], statin therapy timing [24], among others [25].”
Deep Reinforcement Learning Approach for Trading Automation in the Stock Market
Kabbani, T., & Duman, E. (2022) · IEEE Access · doi:10.1109/access.2022.3203697 · Cited by 88 · see citing works
Optimization of anemia treatment in hemodialysis patients via reinforcement learning
Escandell-Montero, P., Chermisi, M., Martínez-Martínez, J. M., Gómez‐Sanchís, J., Barbieri, C., Soria‐Olivas, E., Mari, F., Vila‐Francés, J., Stopper, A., Gatti, E., et al. (2014) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2014.07.004 · Cited by 86 · see citing works
Artificial Intelligence in Plastic Surgery: Applications and Challenges
Liang, X., Yang, X., Yin, S., Malay, S., Chung, K. C., Ma, J., & Wang, K. (2020) · Aesthetic Plastic Surgery · doi:10.1007/s00266-019-01592-2 · Cited by 85 · see citing works
Robust Markov Decision Processes: Beyond Rectangularity
Goyal, V., & Grand-Clément, J. (2022) · Mathematics of Operations Research · doi:10.1287/moor.2022.1259 · Cited by 82 · see citing works
Opportunistic decision making and complexity in emergency care
Franklin, A., Liu, Y., Li, Z., Nguyen, V., Johnson, T. R., Robinson, D. J., Okafor, N. G., King, B. R., Patel, V. L., & Zhang, J. (2011) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.04.001 · Cited by 79 · see citing works
Shi, R., Zhang, J., Chu, W., Bao, Q., Jin, X., Gong, C., Zhu, Q., Yu, C., & Rosenberg, S. (2015) · 2015 IEEE International Conference on Services Computing · doi:10.1109/scc.2015.19 · Cited by 74 · see citing works
Markov Decision Processes for Screening and Treatment of Chronic Diseases
Steimle, L. N., & Denton, B. T. (2017) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-319-47766-4_6 · Cited by 74 · see citing works
UAV Path Planning Based on the Average TD3 Algorithm With Prioritized Experience Replay
Luo, X., Wang, Q., Gong, H., & Tang, C. (2024) · IEEE Access · doi:10.1109/access.2024.3375083 · Cited by 71 · see citing works
Value of information analysis in civil and infrastructure engineering: a review
Zhang, W., Lü, D., Qin, J., Thöns, S., & Faber, M. H. (2021) · Journal of Infrastructure Preservation and Resilience · doi:10.1186/s43065-021-00027-0 · Cited by 70 · see citing works
Elective Patient Admission and Scheduling under Multiple Resource Constraints
Barz, C., & Rajaram, K. (2015) · Production and Operations Management · doi:10.1111/poms.12395 · Cited by 69 · see citing works
Hershberger, P. E., Gallo, A. M., Kavanaugh, K. L., Olshansky, E. F., Schwartz, A., & Tur-Kaspa, I. (2012) · Social Science & Medicine · doi:10.1016/j.socscimed.2012.02.003 · Cited by 66 · see citing works
Sun, Q., Wu, Z. S., Gu, W., Zhu, T., Zhong, L., & Gao, T. (2021) · Energy · doi:10.1016/j.energy.2021.120367 · Cited by 63 · see citing works
Markov Decision Process for Image-Guided Additive Manufacturing
Yao, B., Imani, F., & Yang, H. Y. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2839973 · Cited by 63 · see citing works
Yu, P., Zhang, H., Song, Y., Hui, H., & Chen, G. (2023) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2023.3237888 · Cited by 58 · see citing works
Identifying Appropriate Recipients for CDC Infectious Risk Donor Kidneys
Chow, E. K. H., Massie, A. B., Muzaale, A. D., Singer, A. L., Kucirka, L. M., Montgomery, R. A., Lehmann, H. P., & Segev, D. L. (2013) · American Journal of Transplantation · doi:10.1002/ajt.12206 · Cited by 55 · see citing works
Song, D., Gan, W., Yao, P., Zang, W., Zhang, Z., & Qu, X. (2022) · Ocean Engineering · doi:10.1016/j.oceaneng.2022.110947 · Cited by 53 · see citing works
Rizopoulos, D., Taylor, J. M. G., van Rosmalen, J., Steyerberg, E. W., & Takkenberg, J. J. M. (2015) · Biostatistics · doi:10.1093/biostatistics/kxv031 · Cited by 52 · see citing works
Chen, S., Rui, L., Gao, Z., Li, W., & Qiu, X. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3168885 · Cited by 51 · see citing works
Kondili, L. A., Andreoni, M., Alberti, A., Lobello, S., Babudieri, S., Roscini, A. S., Merolla, R. C. D., Marrocco, W., & Craxı̀, A. (2021) · Epidemics · doi:10.1016/j.epidem.2021.100442 · Cited by 51 · see citing works
Cost-effectiveness of one-time genetic testing to minimize lifetime adverse drug reactions
Alagöz, O., Durham, D., & Kasirajan, K. (2015) · The Pharmacogenomics Journal · doi:10.1038/tpj.2015.39 · Cited by 50 · see citing works
Multi-Objective Markov Decision Processes for Data-Driven Decision Support.
Lizotte, D. J., & Laber, E. B. (2016) · PubMed · Cited by 50 · see citing works
A Deep Learning Approach for Task Offloading in Multi-UAV Aided Mobile Edge Computing
Ebrahim, M. A., Ebrahim, G. A., Mohamed, H. K., & Abdellatif, S. O. (2022) · IEEE Access · doi:10.1109/access.2022.3208584 · Cited by 45 · see citing works
Optimization of PSA Screening Policies
Zhang, J., Denton, B. T., Balasubramanian, H., Shah, N. D., & Inman, B. A. (2011) · Medical Decision Making · doi:10.1177/0272989x11416513 · Cited by 45 · see citing works
Clinical Course among Cases of Acute Liver Failure of Indeterminate Diagnosis
Li, R., Belle, S. H., Horslen, S. P., Chen, L.-W., Zhang, S., Squires, R. H., Bukauskas, K., Narkewicz, M. R., Hite, M., Loomes, K. M., et al. (2016) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2015.12.065 · Cited by 45 · see citing works
The structure of optimal statin initiation policies for patients with Type 2 diabetes
Kurt, M., Denton, B. T., Schaefer, A. J., Shah, N. D., & Smith, S. A. (2011) · IIE Transactions on Healthcare Systems Engineering · doi:10.1080/19488300.2010.550180 · Cited by 42 · see citing works
Self-citationTransplantation at the Nexus of Behavioral Economics and Health Care Delivery
Schnier, K. E., Cox, J. C., McIntyre, C. W., Ruhil, R., Sadiraj, V., & Turgeon, N. A. (2012) · American Journal of Transplantation · doi:10.1111/j.1600-6143.2012.04343.x · Cited by 42 · see citing works
Healthcare management through organizational simulation
Basole, R. C., Bodner, D. A., & Rouse, W. B. (2012) · Decision Support Systems · doi:10.1016/j.dss.2012.10.012 · Cited by 41 · see citing works
Oh, S. H., Park, J., Lee, S. J., Kang, S. S., & Mo, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117932 · Cited by 41 · see citing works
Applications of machine learning in decision analysis for dose management for dofetilide
Levy, A. E., Biswas, M., Weber, R. A., Tarakji, K. G., Chung, M. K., Noseworthy, P. A., Newton‐Cheh, C., & Rosenberg, M. A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0227324 · Cited by 40 · see citing works
Handling decision-making in Intuitionistic Fuzzy environment: PyIFDM package
Więckowski, J., Kizielewicz, B., & Sałabun, W. (2023) · SoftwareX · doi:10.1016/j.softx.2023.101344 · Cited by 40 · see citing works
Choosing the Order of Deceased Donor and Living Donor Kidney Transplantation in Pediatric Recipients
Arendonk, K. J. V., Chow, E. K. H., James, N. T., Orandi, B. J., Ellison, T. A., Smith, J. M., Colombani, P. M., & Segev, A. D. L. (2015) · Transplantation · doi:10.1097/tp.0000000000000588 · Cited by 38 · see citing works
Long-term adaption decisions via fully and partially observable Markov decision processes
Špačková, O., & Straub, D. (2017) · Sustainable and Resilient Infrastructure · doi:10.1080/23789689.2017.1278995 · Cited by 37 · see citing works
From Data to Improved Decisions: Operations Research in Healthcare Delivery
Capan, M., Khojandi, A., Denton, B. T., Williams, K. D., Ayer, T., Chhatwal, J., Kurt, M., Mason, J. E., Roberts, M. S., Zaric, G. S., et al. (2017) · Medical Decision Making · doi:10.1177/0272989x17705636 · Cited by 37 · see citing works
Self-citationEconomic Assessment of an Anti-HCV Screening Program in Italy
Ruggeri, M., Coretti, S., Gasbarrini, A., & Cicchetti, A. (2013) · Value in Health · doi:10.1016/j.jval.2013.07.005 · Cited by 36 · see citing works
A new belief Markov chain model and its application in inventory prediction
He, Z., & Jiang, W. R. (2017) · International Journal of Production Research · doi:10.1080/00207543.2017.1405166 · Cited by 36 · see citing works
The applications and prospects of big data in perioperative anesthetic management
Zhu, Y., Liu, X., Li, Y., & Yi, B. (2024) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-024-00068-0 · Cited by 33 · see citing works
Standoff tracking control of underwater glider to moving target
Zang, W., Yao, P., & Song, D. (2021) · Applied Mathematical Modelling · doi:10.1016/j.apm.2021.09.011 · Cited by 33 · see citing works
Lubricating Oil Remaining Useful Life Prediction Using Multi-Output Gaussian Process Regression
Tanwar, M., & Raghavan, N. (2020) · IEEE Access · doi:10.1109/access.2020.3008328 · Cited by 31 · see citing works
Zeng, J., Shao, J., Lin, S., Zhang, H., Su, X., Lian, X., Zhao, Y., Ji, X., & Zheng, Z. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac088 · Cited by 30 · see citing works
Kondili, L. A., Romano, F., Rolli, F. R., Ruggeri, M., Rosato, S., Brunetto, M. R., Zignego, A. L., Ciancio, A., Leo, A. D., Raimondo, G., et al. (2017) · Hepatology · doi:10.1002/hep.29399 · Cited by 29 · see citing works
Human-Augmented Prescriptive Analytics With Interactive Multi-Objective Reinforcement Learning
Lepenioti, K., Bousdekis, A., Apostolou, D., & Mentzas, G. N. (2021) · IEEE Access · doi:10.1109/access.2021.3096662 · Cited by 28 · see citing works
The Decision to Emigrate: A Simulation Model Based on the Theory of Planned Behaviour
Willekens, F. (2016) · The Springer series on demographic methods and population analysis · doi:10.1007/978-3-319-32283-4_10 · Cited by 26 · see citing works
Modelling and solving resource allocation problems via a dynamic programming approach
Forootani, A., Tipaldi, M., Zarch, M. G., Liuzza, D., & Glielmo, L. (2019) · International Journal of Control · doi:10.1080/00207179.2019.1661521 · Cited by 24 · see citing works
Expert Systems in Mental Health Care
Bennett, C. C., & Doub, T. W. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-420248-1.00002-7 · Cited by 24 · see citing works
A radiobiological Markov simulation tool for aiding decision making in proton therapy referral
Austin, A. M., Douglass, M. J. J., Nguyen, G. T., & Penfold, S. N. (2017) · Physica Medica · doi:10.1016/j.ejmp.2017.11.013 · Cited by 24 · see citing works
Yang, K., Yu, Z., Su, X., Zhang, F., He, X., Wang, N., Zheng, Q., Yu, F., Wen, T., & Zhou, X. (2024) · Chinese Medicine · doi:10.1186/s13020-024-01005-w · Cited by 23 · see citing works
Zhang, Q., Kang, Y., Zhao, Y., Li, P., & You, S. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3127857 · Cited by 23 · see citing works
Robustness of Proactive Intensive Care Unit Transfer Policies
Grand-Clément, J., Chan, C. W., Goyal, V., & Escobar, G. J. (2022) · Operations Research · doi:10.1287/opre.2022.2403 · Cited by 23 · see citing works
Decision modeling in donation after circulatory death liver transplantation
McLean, K. A., Camilleri‐Brennan, J., Knight, S. R., Drake, T. M., Ots, R., Shaw, C. A., Wigmore, S. J., & Harrison, E. M. (2016) · Liver Transplantation · doi:10.1002/lt.24715 · Cited by 22 · see citing works
Sun, Z., Fan, Y., & Wang, G. (2023) · Journal of Marine Science and Engineering · doi:10.3390/jmse11040812 · Cited by 21 · see citing works
DRL-Based Load-Balancing Routing Scheme for 6G Space–Air–Ground Integrated Networks
Dong, F., Song, J., Zhang, Y., Wang, Y., & Huang, T. (2023) · Remote Sensing · doi:10.3390/rs15112801 · Cited by 21 · see citing works
A MDP Model for Breast and Ovarian Cancer Intervention Strategies for BRCA1/2 Mutation Carriers
Abdollahian, M., & Das, T. K. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2014.2319246 · Cited by 21 · see citing works
Optimization of Sequential Decision Making for Chronic Diseases: From Data to Decisions
Denton, B. T. (2018) · doi:10.1287/educ.2018.0184 · Cited by 20 · see citing works
Oh, S. H., Lee, S. J., Noh, J., & Mo, J. (2021) · Scientific Reports · doi:10.1038/s41598-021-86419-4 · Cited by 20 · see citing works
Data-Driven Markov Decision Process Approximations for Personalized Hypertension Treatment Planning
Schell, G. J., Marrero, W. J., Lavieri, M. S., Sussman, J. B., & Hayward, R. A. (2016) · MDM Policy & Practice · doi:10.1177/2381468316674214 · Cited by 20 · see citing works
Sensitivity Analysis in Sequential Decision Models
Chen, Q., Ayer, T., & Chhatwal, J. (2016) · Medical Decision Making · doi:10.1177/0272989x16670605 · Cited by 19 · see citing works
An End-to-End Deep Reinforcement Learning Approach for Job Shop Scheduling
Zhao, L., Shen, W., Zhang, C., & Peng, K. (2022) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd54268.2022.9776116 · Cited by 18 · see citing works
Information Entropy in Predicting Location of Observation Points for Long Tunnel
Xu, C. G., Hu, C., Liu, X., & Wang, S. (2017) · Entropy · doi:10.3390/e19070332 · Cited by 18 · see citing works
Establishing the Value of Diagnostic and Prognostic Tests in Health Technology Assessment
Soares, M. O., Walker, S., Palmer, S., & Sculpher, M. J. (2018) · Medical Decision Making · doi:10.1177/0272989x17749829 · Cited by 18 · see citing works
What Is the Optimal Threshold at Which to Recommend Breast Biopsy?
Burnside, E. S., Chhatwal, J., & Alagöz, O. (2012) · PLoS ONE · doi:10.1371/journal.pone.0048820 · Cited by 18 · see citing works
Optimal Policy Learning for Disease Prevention Using Reinforcement Learning
Khan, Z. A., Feng, Z., Uddin, M. I., Mast, N., Shah, S. A. A., Imtiaz, M., Al‐Khasawneh, M. A., & Mahmoud, M. (2020) · Scientific Programming · doi:10.1155/2020/7627290 · Cited by 17 · see citing works
Constrained Markov Decision Process Modeling for Optimal Sensing of Cardiac Events in Mobile Health
Yao, B., Chen, Y., & Yang, H. Y. (2021) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2021.3052483 · Cited by 17 · see citing works
Exploring Potential Energy Surfaces Using Reinforcement Machine Learning
Mills, A. W., Goings, J. J., Beck, D. A. C., Yang, C., & Li, X. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c00373 · Cited by 17 · see citing works
Soares, H., Arruda, E. F., Bahiense, L., Gärtner, D., & Filho, L. A. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101791 · Cited by 16 · see citing works
Realizing Midcourse Penetration With Deep Reinforcement Learning
Liang, J., Nan, Y., & Li, Z. (2021) · IEEE Access · doi:10.1109/access.2021.3091605 · Cited by 16 · see citing works
Operations research applications in hospital operations: Part I
Abe, T. K., Beamon, B. M., Storch, R. L., & Agus, J. (2016) · IIE Transactions on Healthcare Systems Engineering · doi:10.1080/19488300.2015.1134727 · Cited by 16 · see citing works
Santos, E. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.09312 · Cited by 16 · see citing works
Goals and Objectives to Optimize the Value of an Acute Pain Service in Perioperative Pain Management
Le-Wendling, L. T., Glick, W., & Tighe, P. J. (2017) · Techniques in Orthopaedics · doi:10.1097/bto.0000000000000245 · Cited by 15 · see citing works
Bazrafshan, N., & lotfi, M. M. (2020) · Annals of Operations Research · doi:10.1007/s10479-020-03706-5 · Cited by 15 · see citing works
AI-Driven Decision Making for Auxiliary Diagnosis of Epidemic Diseases
Lin, K., Liu, J., & Gao, J. (2021) · IEEE Transactions on Molecular Biological and Multi-Scale Communications · doi:10.1109/tmbmc.2021.3120646 · Cited by 15 · see citing works
Cost-Effectiveness Analysis of Early Treatment of Chronic HCV with Sofosbuvir/Velpatasvir in Italy
Ruggeri, M., Romano, F., Basile, M., Coretti, S., Rolli, F. R., Drago, C., & Cicchetti, A. (2018) · Applied Health Economics and Health Policy · doi:10.1007/s40258-018-0410-x · Cited by 15 · see citing works
Predicting engine oil degradation across diverse vehicles and identifying key factors
Omiya, T., Hanyuda, K., & Nagatomi, E. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112524 · Cited by 14 · see citing works
Risk‐based breast cancer follow‐up stratified by age
Witteveen, A., Otten, J. W. M., Vliegen, I. M. H., Siesling, S., Timmer, J., & IJzerman, M. J. (2018) · Cancer Medicine · doi:10.1002/cam4.1760 · Cited by 14 · see citing works
Benzin, J.-V., & Rinderle‐Ma, S. (2025) · ACM Computing Surveys · doi:10.1145/3743672 · Cited by 13 · see citing works
Dynamic Transitions of Pediatric Sepsis: A Markov Chain Analysis
Kausch, S. L., Lobo, J. M., Spaeder, M. C., Sullivan, B. A., & Keim‐Malpass, J. (2021) · Frontiers in Pediatrics · doi:10.3389/fped.2021.743544 · Cited by 12 · see citing works
A Markov Decision Processes Modeling for Curricular Analytics
Slim, A., Yusuf, H. A., Abbas, N., Abdallah, C. T., Heileman, G. L., & Slim, A. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00071 · Cited by 12 · see citing works
Li, F., Jörg, F., Li, X., & Feenstra, T. (2022) · PharmacoEconomics · doi:10.1007/s40273-022-01185-z · Cited by 12 · see citing works
Computer Disease Simulation Models
Rutter, C. M., Knudsen, A. B., & Pandharipande, P. V. (2011) · Academic Radiology · doi:10.1016/j.acra.2011.02.004 · Cited by 12 · see citing works
Decision-Making Process Regarding Fertility Among Reproductive-Age Women With Cancer in Taiwan
Huang, S., Hsieh, P.-L., Hsiung, Y., Tseng, L., Chen, P., & Hung, C. (2016) · Cancer Nursing · doi:10.1097/ncc.0000000000000439 · Cited by 12 · see citing works
Was Angelina Jolie Right? Optimizing Cancer Prevention Strategies Among BRCA Mutation Carriers
Nohdurft, E., Long, E. F., & Spinler, S. (2017) · Decision Analysis · doi:10.1287/deca.2017.0352 · Cited by 12 · see citing works
Frazier, A., & Gonzales, J. E. (2021) · Psychology Crime and Law · doi:10.1080/1068316x.2021.1972107 · Cited by 12 · see citing works
Operations research applications in hospital operations: Part III
Abe, T. K., Beamon, B. M., Storch, R. L., & Agus, J. (2016) · IIE Transactions on Healthcare Systems Engineering · doi:10.1080/19488300.2016.1199613 · Cited by 12 · see citing works
Markov chain evaluation of acute postoperative pain transition states
Tighe, P. J., Bzdega, M., Fillingim, R. B., Rashidi, P., & Aytuğ, H. (2015) · Pain · doi:10.1097/j.pain.0000000000000429 · Cited by 11 · see citing works
Acharya, A., Howes, A., Baber, C., & Marshall, T. (2018) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931218621033 · Cited by 14 · see citing works
An integrated supply chain risk mitigation tool – model, analysis and insights
Balan, S., & Handfield, R. (2024) · International Journal of Management Science and Engineering Management · doi:10.1080/17509653.2024.2441783 · Cited by 10 · see citing works
Stochastic Artificial Intelligence benefits and Supply Chain Management inventory prediction
Haoud, N. E., & Bachiri, Z. (2019) · International Colloquium on Logistics and Supply Chain Management (LOGISTIQUA) · doi:10.1109/logistiqua.2019.8907271 · Cited by 10 · see citing works
- Authors
- Alagoz, O., Hsu, H., Schaefer, A. J., & Roberts, M. S.
- Affiliations
- Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI,
- University of Pittsburgh (US)
- Venue
- Medical Decision Making
- Publisher
- SAGE Publications
- Published
- 2009-12-31
- Volume
- 30
- Issue
- 4
- Pages
- 474-483
- ISSN
- 0272-989X (ISSN-L)
- 1552-681X
- PMC
- PMC3060044
- OpenAlex
- W2036336974
- Cited by
- 266
- References
- 34
- Citations by year
- 2026: 15 · 2025: 18 · 2024: 22 · 2023: 21 · 2022: 28 · 2021: 24 · 2020: 16 · 2019: 12 · 2018: 26 · 2017: 16 · 2016: 14 · 2015: 15 · 2014: 8 · 2013: 5 · 2012: 16
- FWCI
- 17.22
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Decision Making (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Science Foundation
- U.S. National Library of Medicine (R21LM008273)
- National Center for Research Resources (UL1RR025011)
- Division of Civil, Mechanical and Manufacturing Innovation
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Social Sciences › Decision Sciences › Management Science and Operations Research › Complex Systems and Decision Making
- Keywords
- Markov decision process, sequential decision making under uncertainty, living donor liver transplantation, Markov model, decision analysis, computational efficiency, life expectancy
- MeSH terms
- Decision Making, Humans, Markov Chains, Liver Transplantation, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
