The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index.
Markov Decision Processes: A Tool for Sequential Decision Making under Uncertainty
Open accessAbstract
We provide a tutorial on the construction and evaluation of Markov decision processes (MDPs), which are powerful analytical tools used for sequential decision making under uncertainty that have been widely used in many industrial and manufacturing applications but are underutilized in medical decision making (MDM). We demonstrate the use of an MDP to solve a sequential clinical treatment problem under uncertainty. Markov decision processes generalize standard Markov models in that a decision process is embedded in the model and multiple decisions are made over time. Furthermore, they have significant advantages over standard decision analysis. We compare MDPs to standard Markov-based simulation models by solving the problem of the optimal timing of living-donor liver transplantation using both methods. Both models result in the same optimal transplantation policy and the same total life expectancies for the same patient and living donor. The computation time for solving the MDP model is significantly smaller than that for solving the Markov model. We briefly describe the growing literature of MDPs applied to medical decisions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, despite such applicability of sequential decisionmaking techniques like MDPs to medical decision-making, there have been relatively few applications in healthcare [18,19].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“Markov decision processes (MDPs) are one efficient technique for determining such optimal sequential decisions (termed a 'policy') in dynamic and uncertain environments [18,19], and have been explored in medical decision-making problems in recent years [18,20].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“Other methods such as Markov decision processes generalize STMs by allowing embedding of sequential decisions, and thus, multiple decisions can be made in multiple time periods [39,40].”
From State-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3 (2012) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Cost-Effectiveness in Health and Medicine
(1996) · doi:10.1093/oso/9780195108248.001.0001 · Cited by 7,193
Puterman, M. L. (1994) · Wiley series in probability and statistics · doi:10.1002/9780470316887 · Cited by 6,057
Approximate Dynamic Programming
Powell, W. B. (2011) · Wiley series in probability and statistics · doi:10.1002/9781118029176 · Cited by 2,359
Approximate Dynamic Programming
Powell, W. B. (2007) · Wiley series in probability and statistics · doi:10.1002/9780470182963 · Cited by 1,582
Dynamic Programming and Stochastic Control
Bertsekas, D. P., & White, C. C. (1977) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1977.4309612 · Cited by 1,211
The Markov process in medical prognosis
Beck, J. R., & Pauker, S. G. (1983) · Medical Decision Making · doi:10.1177/0272989x8300300403 · Cited by 957
Operations research : applications and algorithms
Winston, W. L. (1994) · Cited by 638
Dynamic programming: Models and applications
Denardo, E. V. (1982) · CERN Document Server (European Organization for Nuclear Research) · Cited by 411
Roberts, M. S., Angus, D. C., Bryce, C. L., Valenta, Z., & Weissfeld, L. A. (2004) · Liver Transplantation · doi:10.1002/lt.20137 · Cited by 380
Self-citation
Research works printed but not linked: 5
- Bellman RE, 1957, Dynamic Programming
- Faissol D., 2006, Swann J. Timing of Testing and Treatment of Hepatitis C and Other Diseases
- Kreke JE, 2007, Decisions for Patients with Pneumonia-Related Sepsis [PhD dissertation]
- Kurt M., At what lipid ratios should a patient with type 2 diabetes initiate statins?
- 2006, Pro T Suite [computer program]. Version release 1
Cited by
Sorted by citations
Items 201–264 of 264
Learning Stroke Treatment Progression Models for an MDP Clinical Decision Support System
Coroian, D. C., & Hauser, K. K. (2015) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611974010.76 · Cited by 1
Work-in-Progress: Hierarchical Control of a Catoptric Surface
Chamberlain, R. D., Ahrens, C., Gill, C. D., & Mitchell, S. A. (2018) · ACM International Conference on Embedded Software (EMSOFT) · doi:10.1109/emsoft.2018.8537230 · Cited by 1
A MARKOV DECISION PROCESS APPROACH FOR MENTAL HEALTH
Göçgün, Y. (2020) · Konya Journal of Engineering Sciences · doi:10.36306/konjes.745756 · Cited by 1
WMat algorithm based on Q-Learning algorithm in taxi-v2 game
Esmaeily, F., & Keyvanpour, M. R. (2020) · International Conference on Smart Cities, Internet of Things and Applications (SCIoT) · doi:10.1109/sciot50840.2020.9250211 · Cited by 1
Graham, E. L. (2022) · Systems and Information Engineering Design Symposium · doi:10.1109/sieds55548.2022.9799386 · Cited by 1
Modelling Uncertainty Effects in Decision Taking in Buyer-Supplier Relations
Kamann, D. F. (2023) · Journal of Economics Management and Trade · doi:10.9734/jemt/2023/v29i71101 · Cited by 1
Sensitivity Analysis for Stopping Criteria with Application to Organ Transplantations
Ren, X., Fu, M. C., & Marcus, S. I. (2023) · Winter Simulation Conference · doi:10.1109/wsc60868.2023.10408603 · Cited by 1
Rasmussen, D. A., Bursell, M. G., & Burkhart, F. (2025) · Genetics · doi:10.1093/genetics/iyaf244 · Cited by 1
Modeling Search Behaviors during the Acquisition of Expertise in a Sequential Decision-Making Task
Moënne‐Loccoz, C., Vergara, R. C., López, V., Mery, D., & Cosmelli, D. J. (2017) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2017.00080 · Cited by 1
Optimal treatment policies for pelvic organ prolapse in women
Zhuo, Y., Solak, S., Harmanli, O., & Jones, K. A. (2021) · Decision Sciences · doi:10.1111/deci.12521 · Cited by 1
Robust and Interpretable Sequential Decision-Making for Healthcare
Grand-Clément, J. (2021) · doi:10.7916/d8-maqq-mp30 · Cited by 1
Solving Optimization Problems with Blackwell Approachability
Grand-Clément, J., & Kroer, C. (2023) · Mathematics of Operations Research · doi:10.1287/moor.2023.1376 · Cited by 2
Abdollahian, M. (2015) · Digital Commons - University of South Florida (University of South Florida) · Cited by 1
Convergence Rates of Posterior Distributions in Markov Decision ProcessPreprint
Li, Z., & Laber, E. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.09083 · Cited by 1
Demiray, O., Güneş, E. D., & Örmeci, E. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.04357 · Cited by 1
Lizotte, D. J. (2017) · Wiley eBooks · doi:10.1002/9781118445112.stat08000 · Cited by 1
The Implications of State Aggregation on the Utility of Estimated Markov Decision Process ModelsPreprint
Pollack, M., & Steimle, L. N. (2022) · doi:10.31219/osf.io/dxykc · Cited by 1
Cox, L. A. (2012) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-1-4614-6058-9_2
An Evacuation Decision Making Model for Firefighters in Ad Hoc Robot Network
Li, D. M., Zou, J., Yue, K., Guan, H., & Wang, J. C. (2013) · Advanced materials research · doi:10.4028/www.scientific.net/amr.756-759.504
Ruggeri, M., Coretti, S., Bidino, R. D., Marcellusi, A., Mennini, F. S., & Cicchetti, A. (2012) · PharmacoEconomics Italian Research Articles · doi:10.1007/bf03337458
An Evacuation Decision Making Model for Firefighters in Ad Hoc Robot Network
Li, D., Zou, J., Yue, K., Guan, H., & Wang, J. (2012) · International Conference on Computer and Information Application · doi:10.2991/iccia.2012.121
Hunink, M. G. M., Weinstein, M. C., Wittenberg, E., Drummond, M., Pliskin, J. S., Wong, J. B., & Glasziou, P. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139506779.013
MDP Modeling for the Prediction of Agent Movement in Limited Space
Jin, H., Kim, S., Jung, C., & Lee, M. (2015) · 한국경영과학회지 · doi:10.7737/jkorms.2015.40.3.063
Approximate dynamic programming algorithms for production-planning problems
Liu, N. (2013) · Holmes Museum Of Anthropology (Wichita State University)
Robust Healthcare Financing Systems
Xiao, T. (2015)
MODELING CHRONIC HEPATITIS C DURING RAPID THERAPEUTIC ADVANCE
Liu, S. (2018) · Wiley eBooks · doi:10.1002/9781118960158.ch6
Ruggeri, M., Coretti, S., Bidino, R. D., Marcellusi, A., Mennini, F. S., & Cicchetti, A. (1998) · International Journal of Early Childhood · doi:10.1007/bf03325363
A Data Dissemination Method Based on Region Type Correlations for Mobile Opportunistic Networks
Liu, L., & Chen, D. (2018) · Wireless Communications and Mobile Computing · doi:10.1155/2018/1576286
Modeling Ductal Carcinoma in Situ Management.
Goodarzi, S. H. (2018) · NCSU Libraries Repository (North Carolina State University Libraries)
Hierarchical control of a catoptric surface: work-in-progress
Chamberlain, R. D., Ahrens, C., Gill, C. D., & Mitchell, S. A. (2018) · Embedded Software · doi:10.5555/3283535.3283542
Hayward, R. A., Schell, G. J., Robinson, J. G., Sussman, J. B., & Lavieri, M. S. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/608216
A code to evaluate investment projects in energy system with a SAM and a Energy-SAM
Rao, M. N., Ferrarese, C., & Sabetta, M. (2019) · ENEA Open Archive (National Agency for New Technologies, Energy and Sustainable Economic Development)
A conceptual framework for modeling longitudinal healthcare encounter data
Balasubramanian, H., Murphy, N., & Rossi, M. (2016) · Winter Simulation Conference · doi:10.1109/wsc.2016.7822257
Utilizing Clinical Trial Data to Assess Timing of Surgical Treatment for Emphysema Patients
Alimohammadi, M., Chaovalitwongse, W. A., Vesselle, H. J., & Zhang, S. (2022) · Medical Decision Making · doi:10.1177/0272989x221132256
Learning by Successful or Unsuccessful Experiences?
Kim, K. J., & Santos, E. (2023) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/21695067231192528
Player Exploration Patterns in Interactive Molecular Docking with Electrostatic Visual Cues
Liu, L., Adamson, T., Tapia, L., & Jacobson, B. (2023) · ACM SIGGRAPH Conference on Motion, Interaction and Games (MIG) · doi:10.1145/3623264.3624463
Trusted Task Offloading and Resource Allocation Strategy in MEC Environment
Tong, Z., Wang, J., & Li, K. (2024) · Studies in big data · doi:10.1007/978-981-97-2644-8_5
The loss outweighs the gain: Myopic risk ignorance in sequential decision making
Zhiqin, C. H. E. N., Jiatao, M. A., & ZHANG, X. (2024) · Advances in Psychological Science · doi:10.3724/sp.j.1042.2024.01980
Does the selection of fox for their reactions to humans affect the decision-making during learning?
Андреевна, М. И., & Kharlamova, A. V. (2024) · Журнал высшей нервной деятельности им И П Павлова · doi:10.31857/s0044467724030061
Eteman, V. (2024) · Akademisyen Kitabevi eBooks · doi:10.37609/akya.3349
Does Long-Term Selection for Reactions to Humans Affect Decision-Making during Learning in Foxes?
Андреевна, М. И., & Kharlamova, A. V. (2024) · Neuroscience and Behavioral Physiology · doi:10.1007/s11055-024-01738-z
Decision Intelligence for Healthcare Decarbonization
Nadarajah, S., & Dziemian, S. (2025) · Foundations and Trends® in Technology Information and Operations Management · doi:10.1561/0200000116-4
Zhang, H., Song, Y., Chen, G., & Yu, P. (2025) · Elsevier eBooks · doi:10.1016/b978-0-44-336492-1.00015-0
Duan, Y., Ye, M., & Dai, P. (2025) · Kalpa publications in computing · doi:10.29007/42w6
Finkel, A., Fougea, G., & Roux, S. L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5451114
Optimal Control for Remote Patient Monitoring with Multidimensional Health States
Chandak, S., Thapa, I., Bambos, N., & Scheinker, D. (2025) · International Conference on Communications · doi:10.1109/icc52391.2025.11161852
Optimizing personalized COVID-19 treatment strategies using finite-horizon MDP
Akraoui, B. E., Es-sabery, F., Albahar, M. A., Daoui, C., & Larach, A. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07858-6
Launching Under Pressure: How Resource Scarcity Affects Design Decisions and Resource Management
Letting, C., & Menold, J. (2025) · Journal of Mechanical Design · doi:10.1115/1.4070399
Vovchenko, I. V., Zyablovsky, A. A., Pukhov, A. A., & Andrianov, E. S. (2026) · SciPost Physics Core · doi:10.21468/scipostphyscore.9.3.056
Golsefidi, A. H., Hüttel, F. B., Samaranayake, S., & Pereira, F. C. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131433
Reinforcement Learning-Based Personalized Mood Stabilizer Dosage Optimization
de Filippis, R., & Foysal, A. A. (2026) · OALib · doi:10.4236/oalib.1114925
Relatively Robust Timing of Medical Tests Under Ambiguous Prevalence Dynamics
Weber, T. A. (2026) · International Research Conference on Smart Computing and Systems Engineering (SCSE) · doi:10.1109/scse70081.2026.11499885
Cochran, A. L., David, J. C., & Zayas‐Cabán, G. (2026) · Stochastic Systems · doi:10.1287/stsy.2023.0034
Verghese, P., & Thiyagarajan, D. R. (2026) · International Academic Journal of Science and Engineering · doi:10.71086/iajse/v13i1/iajse1337
Li, J., & Wang, Y. (2026) · Expert Systems · doi:10.1111/exsy.70353
Artificial intelligence applications in clinical decision-making and mental healthcare consultation
Bennett, C. C. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-33151-0.00017-4
Automatic therapy planning based on concept drift
Ramírez-Cruz, M. N., Sucar, L. E., Morales, E. F., & Rivas, J. J. (2026) · Frontiers in Rehabilitation Sciences · doi:10.3389/fresc.2026.1874544
Mhammad, S., Molodyakov, S. A., & Владимирович, У. Л. (2026) · International Conference on Neural Networks and Neurotechnologies (NeuroNT) · doi:10.1109/neuront71829.2026.11651407
Zhao, X. U., Ma, Y., Chang, L. I. U., He, B., & Hu, J. (2026) · Aerospace · doi:10.3390/aerospace13090812
Dong, Y., 何正文, Zheng, W., & Fang, C. (2026) · Computers & Industrial Engineering · doi:10.1016/j.cie.2026.112437
Stochastic Processes for Adaptive Healthcare Systems
Sachan, S. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-9705-4.ch001
Translational Evidence-Based Interventions in Osteoimmunology
Chiappelli, F. (2011) · Springer eBooks · doi:10.1007/978-1-4419-9495-0_3
Metadata
- Authors
- Alagoz, O., Hsu, H., Schaefer, A. J., & Roberts, M. S.
- Affiliations
- Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI,
- University of Pittsburgh (US)
- Venue
- Medical Decision Making
- Publisher
- SAGE Publications
- Published
- 2009-12-31
- Volume
- 30
- Issue
- 4
- Pages
- 474-483
- ISSN
- 0272-989X (ISSN-L)
- 1552-681X
- PMC
- PMC3060044
- OpenAlex
- W2036336974
- Cited by
- 266
- References
- 34
- Citations by year
- 2026: 15 · 2025: 18 · 2024: 22 · 2023: 21 · 2022: 28 · 2021: 24 · 2020: 16 · 2019: 12 · 2018: 26 · 2017: 16 · 2016: 14 · 2015: 15 · 2014: 8 · 2013: 5 · 2012: 16
- FWCI
- 17.22
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Decision Making (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Science Foundation
- U.S. National Library of Medicine (R21LM008273)
- National Center for Research Resources (UL1RR025011)
- Division of Civil, Mechanical and Manufacturing Innovation
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Social Sciences › Decision Sciences › Management Science and Operations Research › Complex Systems and Decision Making
- Keywords
- Markov decision process, sequential decision making under uncertainty, living donor liver transplantation, Markov model, decision analysis, computational efficiency, life expectancy
- MeSH terms
- Decision Making, Humans, Markov Chains, Liver Transplantation, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
