Cited by 266Health Systems, Economic Evaluations, Quality of LifeProbability and Statistical ResearchComplex Systems and Decision Making

The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index.

Markov Decision Processes: A Tool for Sequential Decision Making under Uncertainty

Open access

Alagoz, Oguzhan, Hsu, Heather, Schaefer, Andrew J., Roberts, Mark S.·2010·Medical Decision Making

DOI 10.1177/0272989x09353194PMC PMC3060044

Open this paper in CoChat — free account

Abstract

We provide a tutorial on the construction and evaluation of Markov decision processes (MDPs), which are powerful analytical tools used for sequential decision making under uncertainty that have been widely used in many industrial and manufacturing applications but are underutilized in medical decision making (MDM). We demonstrate the use of an MDP to solve a sequential clinical treatment problem under uncertainty. Markov decision processes generalize standard Markov models in that a decision process is embedded in the model and multiple decisions are made over time. Furthermore, they have significant advantages over standard decision analysis. We compare MDPs to standard Markov-based simulation models by solving the problem of the optimal timing of living-donor liver transplantation using both methods. Both models result in the same optimal transplantation policy and the same total life expectancies for the same patient and living donor. The computation time for solving the MDP model is significantly smaller than that for solving the Markov model. We briefly describe the growing literature of MDPs applied to medical decisions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1177/0272989x09353194

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Dynamic Programming
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Cost-Effectiveness in Health and Medicine
(1996) · doi:10.1093/oso/9780195108248.001.0001 · Cited by 7,193
Markov Decision Processes
Puterman, M. L. (1994) · Wiley series in probability and statistics · doi:10.1002/9780470316887 · Cited by 6,057
Approximate Dynamic Programming
Powell, W. B. (2011) · Wiley series in probability and statistics · doi:10.1002/9781118029176 · Cited by 2,359
Approximate Dynamic Programming
Powell, W. B. (2007) · Wiley series in probability and statistics · doi:10.1002/9780470182963 · Cited by 1,582
Dynamic Programming and Stochastic Control
Bertsekas, D. P., & White, C. C. (1977) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1977.4309612 · Cited by 1,211
The Markov process in medical prognosis
Beck, J. R., & Pauker, S. G. (1983) · Medical Decision Making · doi:10.1177/0272989x8300300403 · Cited by 957
Operations research : applications and algorithms
Winston, W. L. (1994) · Cited by 638
Dynamic programming: Models and applications
Denardo, E. V. (1982) · CERN Document Server (European Organization for Nuclear Research) · Cited by 411
Survival after liver transplantation in the United States: A disease-specific analysis of the UNOS database
Roberts, M. S., Angus, D. C., Bryce, C. L., Valenta, Z., & Weissfeld, L. A. (2004) · Liver Transplantation · doi:10.1002/lt.20137 · Cited by 380
Self-citation

Show all 34 references

Research works printed but not linked: 5

Bellman RE, 1957, Dynamic Programming
Faissol D., 2006, Swann J. Timing of Testing and Treatment of Hepatitis C and Other Diseases
Kreke JE, 2007, Decisions for Patients with Pneumonia-Related Sepsis [PhD dissertation]
Kurt M., At what lipid ratios should a patient with type 2 diabetes initiate statins?
2006, Pro T Suite [computer program]. Version release 1

Sorted by citations

Items 101–200 of 264

Reinforcement Learning for Interactive QoS-Aware Services Composition
Alizadeh, P., Osmani, A., Khanouche, M. E., Chibani, A., & Amirat, Y. (2020) · IEEE Systems Journal · doi:10.1109/jsyst.2020.2997069 · Cited by 10
Decision Modeling for Economic Evaluation in Otolaryngology–Head and Neck Surgery: Review of Techniques
Forner, D. W. A., Hoit, G., Noel, C. W., Eskander, A. E., de Almeida, J. R., Rigby, M. H., & Naimark, D. M. J. (2020) · Otolaryngology · doi:10.1177/0194599820957288 · Cited by 9
An MDP Model for Integrated Operational Decision Making Under Uncertainty in a Gas and Oil Separation Plant
AlDurgam, M. M., Alzahrani, A. S., & Nasir, A. (2021) · IEEE Access · doi:10.1109/access.2021.3111775 · Cited by 9
Rebalancing Docked Bicycle Sharing System with Approximate Dynamic Programming and Reinforcement Learning
Seo, Y.-H., Kim, D., Kang, S., Byon, Y.-J., & Kho, S. (2022) · Journal of Advanced Transportation · doi:10.1155/2022/2780711 · Cited by 9
Kidney-related operations research: A review
Fathi, M., & Khakifirooz, M. (2019) · IISE Transactions on Healthcare Systems Engineering · doi:10.1080/24725579.2019.1640318 · Cited by 9
Development of the simulation-based German albuminuria screening model (S-GASM) for estimating the cost-effectiveness of albuminuria screening in Germany
Kairys, P., Frese, T., Voigt, P. W., Horn, J., Girndt, M., & Mikolajczyk, R. (2022) · PLoS ONE · doi:10.1371/journal.pone.0262227 · Cited by 9
A Belief Markov Model and Its Application
Deng, X., Deng, Y., Zhang, Y., & Liu, Q. (2012) · Acta Automatica Sinica · doi:10.3724/sp.j.1004.2012.00666 · Cited by 9
A Multi-Fidelity Rollout Algorithm for Dynamic Resource Allocation in Population Disease Management
Ho, T.-Y., Liu, S., & Zabinsky, Z. B. (2018) · Health Care Management Science · doi:10.1007/s10729-018-9454-6 · Cited by 9
An Adaptive Approach to Locating Mobile HIV Testing Services
Gonsalves, G., Crawford, F. W., Cleary, P. D., Kaplan, E. H., & Paltiel, A. D. (2017) · Medical Decision Making · doi:10.1177/0272989x17716431 · Cited by 9
Interpretable Machine Learning for Resource Allocation with Application to Ventilator TriagePreprint
Grand-Clément, J., Goh, Y. H., Chan, C. W., Goyal, V., & Chuang, E. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10994 · Cited by 9
LSTM & attention-based meta-reinforcement learning for trajectory tracking of underwater gliders with varying buoyancy loss and current disturbance
Tian, X., Zhou, H., Zhang, H., Zhang, L., & Shi, L. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.120906 · Cited by 9
Optimization, Simulation and Predictive Analytics in Healthcare
Chanchaichujit, J., Tan, A. W. K., Meng, F., & Eaimkhong, S. (2019) · Springer eBooks · doi:10.1007/978-981-13-8114-0_5 · Cited by 8
Combining simulation and optimization to derive operating policies for a concentrating solar power plant
Wales, J. G., Zolan, A. J., Hamilton, W. T., Newman, A. M., & Wagner, M. J. (2022) · OR Spectrum · doi:10.1007/s00291-022-00688-7 · Cited by 8
Markov models for clinical decision‐making in radiation oncology: A systematic review
McCullum, L. B., Karagoz, A., Dede, C., Garcia, R. V., Nosrat, F., Hemmati, M., Hosseinian, S., Schaefer, A. J., Fuller, C. D., & MD Anderson Head and Neck Cancer Symptom Working Group (2024) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13656 · Cited by 8
Self-citation
Robust AI-enabled Simulation of Treatment Paths with Markov Decision Process for Breast Cancer Patients
Hossain, S. S., Ebrahimi, M. R., Padmanabhan, B., Naqa, I. E., Kuo, P. C., Beard, A., & Merkel, S. (2023) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai54212.2023.00053 · Cited by 8
Bayesian learning of dose–response parameters from a cohort under response-guided dosing
Kotas, J., & Ghate, A. (2015) · European Journal of Operational Research · doi:10.1016/j.ejor.2017.07.034 · Cited by 8
Population-level intervention and information collection in dynamic healthcare policy
Cipriano, L. E., & Weber, T. A. (2017) · Health Care Management Science · doi:10.1007/s10729-017-9415-5 · Cited by 8
Optimal action sequence generation for assistive agents in fixed horizon tasks
Baraka, K., Melo, F. S., Couto, M., & Veloso, M. (2020) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-020-09458-7 · Cited by 8
Use of a Markov decision process model for treatment selection in an asymptomatic disease with consideration of risk sensitivity
Tilson, V., & Tilson, D. (2012) · Socio-Economic Planning Sciences · doi:10.1016/j.seps.2012.09.003 · Cited by 8
Expert Systems in Behavioral and Mental Healthcare: Applications of AI in Decision‐Making and Consultancy
Kulkarni, S. (2022) · Wiley eBooks · doi:10.1002/9781119792376.ch6 · Cited by 8
Comparing tapentadol to oxycodone/naloxone combination: imagining castles in the air … while building sandcastles! Re: Coluzzi F, Ruggeri M. Clinical and economic evaluation of tapentadol extended release and oxycodone/naloxone extended release in comparison with controlled release oxycodone in musculoskeletal pain. Curr Med Res Opin 2014;30:1139-51
Coluzzi, F., & Ruggeri, M. (2014) · Current Medical Research and Opinion · doi:10.1185/03007995.2014.971361 · Cited by 7
A novel intelligent hyper-heuristic algorithm for solving optimization problems
Tong, Z., Chen, H., Liu, B., Cai, J., & Cai, S. (2022) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-211250 · Cited by 7
Temporal Modeling in Clinical Artificial Intelligence, Decision-Making, and Cognitive Computing: Empirical Exploration of Practical Challenges
Bennett, C. C., & Doub, T. W. (2014) · Cited by 7
Response-guided dosing for rheumatoid arthritis
Kotas, J., & Ghate, A. (2015) · IIE Transactions on Healthcare Systems Engineering · doi:10.1080/19488300.2015.1126873 · Cited by 7
Economic evaluation of colorectal cancer screening programs: Affordability for the health service
Coretti, S., Ruggeri, M., Bidino, R. D., Gitto, L., Marcellusi, A., Mennini, F. S., & Cicchetti, A. (2020) · Journal of Medical Screening · doi:10.1177/0969141319898732 · Cited by 6
Artificial Intelligence, Bioinformatics, and Pathology
Levy, J., Lu, Y., Montivero, M. K. S., Ramwala, O. A., McFadden, J. R., Miles, C., Diamond, A. G., Reddy, R., Reddy, R., Hudson, T. R., et al. (2022) · Advances in Molecular Pathology · doi:10.1016/j.yamp.2023.01.001 · Cited by 7
Comparative effectiveness research on patients with acute ischemic stroke using Markov decision processes
Wu, D., Cai, Y., Cai, J., Liu, Q., Zhao, Y., Cai, J., Zhao, M., Huang, Y., Ye, L., Lu, Y., et al. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-23 · Cited by 6
Economic evaluation of Zepatier for the management of HCV in the Italian scenario
Rolli, F. R., Ruggeri, M., Kheiraoui, F., Drago, C., Basile, M., Favaretti, C., & Cicchetti, A. (2018) · The European Journal of Health Economics · doi:10.1007/s10198-018-0980-4 · Cited by 6
Comparing Strategies to Prevent Stroke and Ischemic Heart Disease in the Tunisian Population: Markov Modeling Approach Using a Comprehensive Sensitivity Analysis Algorithm
Saidi, O., O’Flaherty, M., Zoghlami, N., Malouche, D., Capewell, S., Critchley, J., Bandosz, P., Romdhane, H. B., & Guzman-Castillo, M. (2019) · Computational and Mathematical Methods in Medicine · doi:10.1155/2019/2123079 · Cited by 8
Autonomous Rear Parking via Rapidly Exploring Random-Tree-Based Reinforcement Learning
Shahi, S., & Lee, H. (2022) · Sensors · doi:10.3390/s22176655 · Cited by 5
RT-VeD
Wang, S., Lu, J., Guo, B., & Dong, Z. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539183 · Cited by 5
Optimal discharge of patients from intensive care via a data-driven policy learning framework
Lejarza, F., Calvert, J. S., Attwood, M. M., Evans, D. A., & Mao, Q. (2023) · Operations Research for Health Care · doi:10.1016/j.orhc.2023.100400 · Cited by 5
Multimodality treatment planning using the Markov decision process: a comprehensive study of applications and challenges
Singh, P., Singh, S., Mishra, A., & Mishra, S. K. (2024) · Research on Biomedical Engineering · doi:10.1007/s42600-024-00349-4 · Cited by 5
Personalized Cotesting Policies for Cervical Cancer Screening: A POMDP Approach
Ebadi, M., & Akhavan‐Tabatabaei, R. (2021) · Mathematics · doi:10.3390/math9060679 · Cited by 5
Probabilistic System Modeling for Complex Systems Operating in Uncertain Environments
Pouya, P., & Madni, A. M. (2022) · Springer eBooks · doi:10.1007/978-3-030-82083-1_11 · Cited by 5
Tiered Service Architecture for Remote Patient Monitoring
Chandak, S., Thapa, I., Bambos, N., & Scheinker, D. (2024) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom60970.2024.10880780 · Cited by 4
Dynamic decision models for managing the major complications of diabetes
Kurt, M. (2012) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 4
Risk modelling in blood safety – review of methods, strengths and limitations
Custer, B. (2010) · ISBT Science Series · doi:10.1111/j.1751-2824.2010.01384.x · Cited by 4
Analysis via Markov decision process to evaluate glycemic control strategies of a large retrospective cohort with type 2 diabetes: the ameliorate study
Meng, F., Sun, Y., Heng, B. H., & Leow, M. K. (2020) · Acta Diabetologica · doi:10.1007/s00592-020-01492-x · Cited by 4
Research on Sports Dance Video Recommendation Method Based on Style
Sun, J., & Tang, H. (2022) · Scientific Programming · doi:10.1155/2022/7089057 · Cited by 4
Quantifying the benefits of increasing decision-making frequency for health applications with regular decision epochs
Zhang, S., Suen, S., Sundaram, V., & Gong, C. L. (2024) · IISE Transactions · doi:10.1080/24725854.2024.2321492 · Cited by 4
Quantum error correction for heavy hexagonal code using deep reinforcement learning with policy reuse
Ji, Y., Chen, Q., Wang, R., Ji, N., & Ma, H. (2024) · Quantum Information Processing · doi:10.1007/s11128-024-04377-y · Cited by 4
Exploration Decay Policy (EDP) to Enhanced Exploration-Exploitation Trade-Off in DDPG for Continuous Action Control Optimization
Sumiea, E. H., Abdulkadir, S. J., Alhussian, H., Al-Selwi, S. M., Ragab, M. G., & Alqushaibi, A. (2023) · IEEE Student Conference on Research and Development (SCOReD) · doi:10.1109/scored60679.2023.10563810 · Cited by 4
Cost-effectiveness analysis of Daclatasvir/Sofosbuvir for the treatment of the HCV patients failed after the first line with second generation of DAAs in Italy
Ruggeri, M., Rolli, F. R., Kondili, L. A., Drago, C., Solda, F. D., Nappi, C., & Cicchetti, A. (2018) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2019.1537784 · Cited by 4
Active Acquisition for multimodal neuroimaging
Cole, J. H., Lorenz, R., Geranmayeh, F., Wood, T., Hellyer, P. J., Williams, S., Turkheimer, F., & Leech, R. (2018) · Wellcome Open Research · doi:10.12688/wellcomeopenres.14918.2 · Cited by 4
Optimization and Simulation of Kidney Paired Donation Programs.
Li, Y. (2012) · Deep Blue (University of Michigan) · Cited by 4
Personalized Medicine in Chronic Disease Management.
Schell, G. J. (2015) · Deep Blue (University of Michigan) · Cited by 4
Markov Decision Processes
Scheller-Wolf, A. A. (2018) · Wiley eBooks · doi:10.1002/9781119300977.ch14 · Cited by 4
Optimizing Vital Signs in Patients With Traumatic Brain Injury: Reinforcement Learning Algorithm Development and Validation
Zhang, H., Diao, M., Zhang, S., Ni, P., Zhang, W., Wu, C., Zhu, Y. J., & Hu, W. (2025) · Journal of Medical Internet Research · doi:10.2196/63847 · Cited by 3
Antibiotics or Tonsillectomy for Adult Recurrent Tonsillitis: Analyzing the Lesser of Two Evils
Jacob, T., Leshno, M., Neiderman, N. N. C., Kampel, L., Warshavsky, A., Mansour, J., Assadi, N., Muhanna, N., & Horowitz, G. (2023) · The Laryngoscope · doi:10.1002/lary.31139 · Cited by 3
Balancing Profit and Cultural Heritage: Multi-Objective Dynamic Pricing for Hanfu Using Deep Deterministic Policy Gradient
Zhao, Q. L., Mao, G., & Shen, W. (2025) · IEEE Access · doi:10.1109/access.2025.3526685 · Cited by 3
Advance Multi-Priority, Multi-Appointment Patient Scheduling With Dependent Demand and Lead Times
Moradi, S., Sauré, A., & Patrick, J. (2025) · Production and Operations Management · doi:10.1177/10591478251356474 · Cited by 3
Valutazione economica di un programma di screening anti-HCV in Italia
Cicchetti, A., Ruggeri, M., Coretti, S., Piscaglia, A. C., Ponziani, F. R., Lanati, E., & Gasbarrini, A. (2011) · PharmacoEconomics Italian Research Articles · doi:10.1007/bf03320686 · Cited by 3
A Value-of-Information Framework for Personalizing the Timing of Surveillance Testing
Bansal, A., Heagerty, P. J., Inoue, L. Y. T., Veenstra, D. L., Wolock, C. J., & Basu, A. (2021) · Medical Decision Making · doi:10.1177/0272989x211049213 · Cited by 3
Co-learning of extrusion deposition quality for supporting interconnected additive manufacturing systems
Wei, A.-T., Wang, H., Dickens, T. J., & Chi, H. (2022) · IISE Transactions · doi:10.1080/24725854.2022.2080306 · Cited by 3
A Causal Network-Based Markov Decision Process Model for Intervention Planning
Lee, S., Bain, P. A., Musa, A. J., Baker, C., & Li, J. (2022) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2022.3228643 · Cited by 3
Stochastic control for organ donations: A review
Ren, X., Fu, M. C., & Marcus, S. I. (2023) · Systems & Control Letters · doi:10.1016/j.sysconle.2023.105476 · Cited by 3
A functional Hidden Markov Model to incorporate dynamics into Bayesian optimal stopping problems: Helping physicians manage traumatic brain injuries
Zavadskiy, G., Zantedeschi, D., & Jank, W. (2023) · Decision Support Systems · doi:10.1016/j.dss.2023.114078 · Cited by 3
Optimal acceptance of incompatible kidneys
Ren, X., Fu, M. C., & Marcus, S. I. (2025) · Journal of the Operational Research Society · doi:10.1080/01605682.2024.2444410 · Cited by 3
An assessment method for reliability of distributed control system
Liu, Z., Li, D., Lei, Y., & Si-yan, X. (2015) · IEEE International Conference on Information and Automation (ICIA) · doi:10.1109/icinfa.2015.7279487 · Cited by 3
Mitigating Nonattendance Using Clinic-Resourced Incentives Can Be Mutually Beneficial: A Contingency Management-Inspired Partially Observable Markov Decision Process Model
Bai, Y., & Berg, B. P. (2021) · Value in Health · doi:10.1016/j.jval.2021.03.014 · Cited by 3
Optimal Information Collection Policies in a Markov Decision Process Framework
Cipriano, L. E., Goldhaber‐Fiebert, J. D., Liu, S., & Weber, T. A. (2018) · Medical Decision Making · doi:10.1177/0272989x18793401 · Cited by 3
Toward Valuing Flexibility in Transmission Planning
Tee, C. Y., & Ilić, M. (2016) · Wiley eBooks · doi:10.1002/9781119083016.ch8 · Cited by 3
Optimizing Interpretable Treatment and Screening Policies in Healthcare
McNealey, A. K., Marrero, W. J., Steimle, L. N., & Garcia, G. P. (2023) · Springer eBooks · doi:10.1007/978-3-030-54621-2_866-1 · Cited by 3
A shared decision-making tool to support the decision of patients with Lumbar Disc Herniation on whether to undergo surgery: a pilot study
Corso, L. L., Biasuz, R., Falavigna, A., Rech, M. M., & Berteli, G. O. (2023) · Research Society and Development · doi:10.33448/rsd-v12i3.40722 · Cited by 2
Prompt-to-policy: Leveraging large language models to guide deep reinforcement learning in public health emergencies
Yang, H., Zhang, C., Yang, F., Zhang, X., Zhao, Y., & Sun, H. (2026) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2026.109289 · Cited by 2
Recent advances in artificial intelligence for melanoma: A review of history, models, datasets, applications, and ethical and legal considerations
Parhi, R., Jena, G. K., Garg, A., Dhobi, M., & Mondal, S. (2026) · Journal of Holistic Integrative Pharmacy · doi:10.1016/j.jhip.2026.03.002 · Cited by 2
The Application of Reinforcement Learning to the FlipIt Security Game
Yang, X., Howley, E., & Schukat, M. (2022) · Cyber Research Conference - Ireland (Cyber-RCI) · doi:10.1109/cyber-rci55324.2022.10032686 · Cited by 2
A Decision Framework for AR, Dialogue and Eye Gaze to Enhance Human-Robot Collaboration
Zou, C., Ding, Y. F., Chandan, K., & Zhang, S. (2024) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3610978.3640550 · Cited by 2
State discretization for continuous-state MDPs in infectious disease control
Zhang, S., & Suen, S. (2024) · IISE Transactions on Healthcare Systems Engineering · doi:10.1080/24725579.2024.2428953 · Cited by 2
Optimal timing of organs-at-risk-sparing adaptive radiation therapy for head-and-neck cancer under re-planning resource constraints
Nosrat, F., Dede, C., McCullum, L. B., Garcia, R. V., Mohamed, A., Scott, J. G., Bates, J. E., McDonald, B. A., Wahid, K. A., Naser, M. A., et al. (2025) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2025.100715 · Cited by 2
Self-citation
Innovative Routes for Enhancing Adolescent Marijuana Treatment: Interplay of Peer Influence Across Social Media and Geolocation
Chung, T. A., Pelechrinis, K., Faloutsos, M., Hylek, L., Suffoletto, B. P., & Ewing, S. W. F. (2016) · Current Addiction Reports · doi:10.1007/s40429-016-0095-x · Cited by 2
Validation of Double Transition Model by Analyzing Reward Distributions
Santos, E., Nguyen, H. P. N., Kim, K. J., Hyde, G., & Nyanhongo, C. (2020) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wiiat50758.2020.00088 · Cited by 2
Quickest way to less headache days: an operational research model and its implementation for chronic migraine
Lo, I., & Zhang, P. (2025) · BMC Neurology · doi:10.1186/s12883-025-04124-5 · Cited by 2
Robust Optimal Control for Medical Treatment Decisions
Zhang, Y. (2014) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 2
Determining the optimal strategy for the live-attenuated herpes zoster vaccine in adults
Harvey, M. J., Denton, B. T., Prosser, L. A., & Hutton, D. W. (2018) · Vaccine · doi:10.1016/j.vaccine.2018.06.059 · Cited by 2
Use of a Multidisciplinary Design Optimization Approach to Model Treatment Decisions in Oropharyngeal Cancer
Aarhus, R. T., & Huang, E. (2018) · INCOSE International Symposium · doi:10.1002/j.2334-5837.2018.00537.x · Cited by 2
Using Discrete Time Markov Chains for Control of Idle Character Animation
Streck, A., & Wolbers, T. (2018) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2018.8490450 · Cited by 2
Price Management in Resource Allocation Problem with Approximate Dynamic Programming
Forootani, A., Tipaldi, M., Liuzza, D., & Glielmo, L. (2018) · European Control Conference (ECC) · doi:10.23919/ecc.2018.8550559 · Cited by 2
Developing High-Quality Microsimulation Models Using R in Health Decision Sciences
Eom, H., & Li, Y. (2020) · Winter Simulation Conference · doi:10.1109/wsc48552.2020.9384037 · Cited by 2
Enhancing Energy Trading Between Different Islanded Microgrids A Reinforcement Learning Algorithm Case Study in Northern Kordofan State
ELamin, M., Elhassan, F., & Manzoul, M. A. (2021) · International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE) · doi:10.1109/iccceee49695.2021.9429584 · Cited by 2
An Application of Inverse Reinforcement Learning to Estimate Interference in Drone Swarms
Kim, K. J., Santos, E., Nguyen, H. P. N., & Pieper, S. (2022) · Entropy · doi:10.3390/e24101364 · Cited by 2
Addressing Wicked Problems and Deep Uncertainties in Risk Analysis
Cox, L. A. (2023) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-031-32013-2_7 · Cited by 2
Prescriptive Analytical Models for Dynamic IoT Data Streams: A Review
Shubha, T. V., Kumar, S. M. D., & Venugopal, K. R. (2024) · International Journal of Computing and Digital Systems · doi:10.12785/ijcds/150115 · Cited by 2
Enhancing Robotic Systems for Revolutionizing Healthcare Using Markov Decision Processes
Dahamou, I., & Daoui, C. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-53082-1_23 · Cited by 2
Parallelizing model checking algorithms using multi-core and many-core architectures
Wu, Z. (2017) · doi:10.32657/10356/72377 · Cited by 2
USING FINITE‐HORIZON MARKOV DECISION PROCESSES FOR OPTIMIZING POST‐MAMMOGRAPHY DIAGNOSTIC DECISIONS
Tunç, S., Alagöz, O., Chhatwal, J., & Burnside, E. S. (2018) · Wiley eBooks · doi:10.1002/9781118960158.ch8 · Cited by 2
TREATMENT OPTIMIZATION FOR PATIENTS WITH TYPE 2 DIABETES
Mason, J. E. (2018) · Wiley eBooks · doi:10.1002/9781118960158.ch16 · Cited by 2
The Implications of State Aggregation in Deteriorating Markov Decision Processes with Optimal Threshold PoliciesPreprint
Pollack, M., & Steimle, L. N. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4871846 · Cited by 2
Discrete Event Modeling and Simulation of Large Markov Decision Process : Application to the Leverage Effects in Financial Asset Optimization Processes
Barbieri, E. (2023) · theses.fr (ABES) · doi:10.70675/94042ff8zd2bcz4e4cza593z12abe2190d76 · Cited by 2
A Dual-Critic Deep Deterministic Policy Gradient Approach for Task Offloading in Edge-Fog-Cloud Environment
Ebrahim, M. A., Ebrahim, G. A., & Mohamed, H. K. (2023) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca59364.2023.10401692 · Cited by 1
基于DQN的低轨卫星负载均衡路由算法
Yang, 陈. C., Jing, 左. Z., Yang, 曹. C., Xiaofeng, 彭. P., Liang, 谷. G., Qingyang, 谭. T., & Peifu, 熊. X. (2025) · Acta Optica Sinica · doi:10.3788/aos250706 · Cited by 1
Domain independent heuristics for online stochastic contingent planning
Blumenthal, O., & Shani, G. (2024) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-024-09947-5 · Cited by 1
Cost-effectiveness of personalized policies for implementing organ-at-risk sparing adaptive radiation therapy in head and neck cancer
Hosseinian, S., Suarez-Aguirre, D., Dede, C., Garcia, R. V., McCullum, L. B., Hemmati, M., Karagoz, A., Mohamed, A., Lai, S. Y., Hutcheson, K. A., et al. (2025) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2025.100772 · Cited by 1
Self-citation
Frontiers in Operations: Bounding Counterfactual Outcomes of Health Insurance Delay-and-Deny Practices
Haugh, M. B., & Singal, R. (2026) · Manufacturing & Service Operations Management · doi:10.1287/msom.2025.0030 · Cited by 3
Generation and evaluation of adaptive explanations based on dynamic partner-modeling and non-stationary decision making
Robrecht, A. S., Kowalski, C., & Kopp, S. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2026.1558674 · Cited by 1
Multi-USV coordination control method of path planning and tracking based on MADDPG for target capture
Zhu, Q., Hu, C., & Lu, C. (2026) · Measurement and Control · doi:10.1177/00202940251379174 · Cited by 1
UAV-Assisted Security-Aware Vehicular Edge Computing: A TD3-Enhanced Scheme
Ren, T., Cui, J., Cao, X., & Ren, Y. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3709174 · Cited by 1
A decision-theoretic planning approach for clinical practice guideline modelling
Acosta, D., & Garcia-Gomez, J. M. (2014) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi.2014.6864338 · Cited by 1
Application of Markov Decision Processes (MDPs) in Petroleum Industry
Shi, W., & Guo, J. G. (2014) · GSTF Journal of Engineering Technology · doi:10.5176/2251-3701_2.4.93 · Cited by 1

Authors: Alagoz, O., Hsu, H., Schaefer, A. J., & Roberts, M. S.
Affiliations: Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI,
University of Pittsburgh (US)
ORCID: Hsu, Heather
Schaefer, Andrew J.
Roberts, Mark S.
Venue: Medical Decision Making
Publisher: SAGE Publications
Published: 2009-12-31
Volume: 30
Issue: 4
Pages: 474-483
ISSN: 0272-989X (ISSN-L)
1552-681X
DOI: 10.1177/0272989x09353194
PMC: PMC3060044
OpenAlex: W2036336974
Cited by: 266
References: 34
Citations by year: 2026: 15 · 2025: 18 · 2024: 22 · 2023: 21 · 2022: 28 · 2021: 24 · 2020: 16 · 2019: 12 · 2018: 26 · 2017: 16 · 2016: 14 · 2015: 15 · 2014: 8 · 2013: 5 · 2012: 16
FWCI: 17.22
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Medical Decision Making (publishedVersion)
PubMed (publishedVersion)
PubMed Central (acceptedVersion)
RePEc: Research Papers in Economics (submittedVersion)
Funders: National Science Foundation
U.S. National Library of Medicine (R21LM008273)
National Center for Research Resources (UL1RR025011)
Division of Civil, Mechanical and Manufacturing Innovation
Topics: Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
Social Sciences › Decision Sciences › Management Science and Operations Research › Complex Systems and Decision Making
Keywords: Markov decision process, sequential decision making under uncertainty, living donor liver transplantation, Markov model, decision analysis, computational efficiency, life expectancy
MeSH terms: Decision Making, Humans, Markov Chains, Liver Transplantation, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Markov Decision Processes: A Tool for Sequential Decision Making under Uncertainty

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks