The DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
Modeling Medical Treatment Using Markov Decision Processes
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Currently, most treatment decisions in the medical domain are made via ad-hoc or heuristic approaches, but there is a growing body of evidence that such complex treatment decisions are better handled through modeling rather than intuition alone [18,21].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
In the citing paper
“Markov decision processes (MDPs) are one efficient technique for determining such optimal sequential decisions (termed a 'policy') in dynamic and uncertain environments [18,19], and have been explored in medical decision-making problems in recent years [18,20].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
In the citing paper
“As noted elsewhere, the collection of such 'natural history' disease data is fraught with many challenges, ethical and otherwise, particularly if such untreated conditions pose significant health risks [18].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
In the citing paper
“However, despite such applicability of sequential decisionmaking techniques like MDPs to medical decision-making, there have been relatively few applications in healthcare [18,19].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
In the citing paper
“The work presented here adds to a growing body of evidence that such complex treatment decisions may be better handled through modeling than intuition alone [18,21].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
In the citing paper
“MDPs (and their partially observable cousins) directly address many of the challenges faced in clinical decision-making [17,18].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Methods for the Economic Evaluation of Health Care Programmes
Drummond, M. E., Sculpher, M. J., Torrance, G. W., O’Brien, B. J., & Stoddart, G. L. (2005) · doi:10.1093/oso/9780198529446.001.0001 · Cited by 13,648
(1995) · Cited by 10,558
Availability: A heuristic for judging frequency and probability
Tversky, A., & Kahneman, D. (1973) · Cognitive Psychology · doi:10.1016/0010-0285(73)90033-9 · Cited by 9,984 · cited 1×
“However, there is a growing feeling among medical practitioners that some treatment decisions are too complicated to solve accurately using intuition alone [1, 2].”
BackgroundMarkov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Cost-Effectiveness in Health and Medicine
(1996) · doi:10.1093/oso/9780195108248.001.0001 · Cited by 7,193
Time Discounting and Time Preference: A Critical Review
Frederick, S., Loewenstein, G., & O’Donoghue, T. (2002) · Journal of Economic Literature · doi:10.1257/002205102320161311 · Cited by 6,106
Puterman, M. L. (1994) · Wiley series in probability and statistics · doi:10.1002/9780470316887 · Cited by 6,057
Flagged first, then by citations
2 of the 53 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Roberts, M.S. and D.C. Angus (2002). The optimal timing of liver transplantation: Final report R01 HS09694. University of Pittsburgh, Pittsburgh, PA.
Cited by
Sorted by citations
Items 101–153 of 153
A PopPBPK-RL approach for precision dosing of benazepril in renal impaired patients
Vigueras, G., Muñoz-Gil, L., Reinisch, V., & Pinto, J. T. (2024) · Journal of Pharmacokinetics and Pharmacodynamics · doi:10.1007/s10928-024-09953-4 · Cited by 3
Simulation Optimization of Spatiotemporal Dynamics in 3D Geometries
Yao, B., Leonelli, F. M., & Yang, H. Y. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2024.3524132 · Cited by 3
A Causal Network-Based Markov Decision Process Model for Intervention Planning
Lee, S., Bain, P. A., Musa, A. J., Baker, C., & Li, J. (2022) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2022.3228643 · Cited by 3
Prediction of hypotension during haemodialysis through data analytics and machine learning
Masum, S., Hopgood, A. A., Lewis, R. J., & Sangala, N. C. (2024) · Journal of Kidney Care · doi:10.12968/jokc.2024.9.5.215 · Cited by 3
Age‐dependent optimal policies for hepatitis C virus treatment
Mondschein, S., Yankovic, N., & Matus, O. (2019) · International Transactions in Operational Research · doi:10.1111/itor.12705 · Cited by 3
Zavadskiy, G., Zantedeschi, D., & Jank, W. (2023) · Decision Support Systems · doi:10.1016/j.dss.2023.114078 · Cited by 3
Optimal Information Collection Policies in a Markov Decision Process Framework
Cipriano, L. E., Goldhaber‐Fiebert, J. D., Liu, S., & Weber, T. A. (2018) · Medical Decision Making · doi:10.1177/0272989x18793401 · Cited by 3
Zeng, Z., Fan, Z., Xie, X., Swartz, C. H., DePriest, P. D., & Li, J. (2019) · Flexible Services and Manufacturing Journal · doi:10.1007/s10696-019-09347-6 · Cited by 3
Optimizing Interpretable Treatment and Screening Policies in Healthcare
McNealey, A. K., Marrero, W. J., Steimle, L. N., & Garcia, G. P. (2023) · Springer eBooks · doi:10.1007/978-3-030-54621-2_866-1 · Cited by 3
Can Deep Reinforcement Learning solve the portfolio allocation problem ?
Benhamou, E. (2023) · theses.fr (ABES) · doi:10.70675/e3e5604az2ed2z4e10za31ez0ae46086cc91 · Cited by 3
State discretization for continuous-state MDPs in infectious disease control
Zhang, S., & Suen, S. (2024) · IISE Transactions on Healthcare Systems Engineering · doi:10.1080/24725579.2024.2428953 · Cited by 2
Lo, I., & Zhang, P. (2025) · BMC Neurology · doi:10.1186/s12883-025-04124-5 · Cited by 2
Medical treatment analysis using probabilistic model checking
Debbi, H., Bourahla, M., & Debbi, A. E. (2013) · International Journal of Biomedical Engineering and Technology · doi:10.1504/ijbet.2013.057674 · Cited by 2
Robust Optimal Control for Medical Treatment Decisions
Zhang, Y. (2014) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 2
Outcome And State Transition Modeling For Adaptive Interdisciplinary Pain Management
LeBoulluec, A. K., Zeng, L., Chen, V. C. P., Rosenberger, J. M., & Gatchel, R. J. (2013) · UTA ResearchCommons (University of Texas Arlington) · Cited by 2
Determining the optimal strategy for the live-attenuated herpes zoster vaccine in adults
Harvey, M. J., Denton, B. T., Prosser, L. A., & Hutton, D. W. (2018) · Vaccine · doi:10.1016/j.vaccine.2018.06.059 · Cited by 2
Artificial Intelligence Approaches for the COVID-19 Pandemic
Srinivas, P., Chakkaravarthy, D. M., & Bhattacharyya, D. (2021) · EAI/Springer Innovations in Communication and Computing · doi:10.1007/978-3-030-72752-9_13 · Cited by 2
Addressing Wicked Problems and Deep Uncertainties in Risk Analysis
Cox, L. A. (2023) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-031-32013-2_7 · Cited by 2
On Minimizing Total Discounted Cost in MDPs Subject to Reachability Constraints
Savas, Y., Verginis, C. K., Hibbard, M., & Topcu, U. (2024) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2024.3384834 · Cited by 1
A Review on Application of Reinforcement Learning in Healthcare
Dhawale, C. A., & Dhawale, K. (2023) · Advances in information security, privacy, and ethics book series · doi:10.4018/978-1-6684-6646-9.ch008 · Cited by 1
Shinohara, A., Takumoto, Y., Tauchi, J., Morishita, K., Kawasaki, T., & Akazawa, M. (2025) · Scientific Reports · doi:10.1038/s41598-024-80549-1 · Cited by 1
MDPax: GPU-accelerated MDP solvers in Python with JAX
Farrington, J., Wong, W. K., Li, K., & Utley, M. (2025) · The Journal of Open Source Software · doi:10.21105/joss.08897 · Cited by 1
Finding the Maximum Safe Treatment Interval for Patients With Chronic Conditions
DeRoos, L. J., Lavieri, M. S., & Stein, J. D. (2024) · Production and Operations Management · doi:10.1177/10591478241281077 · Cited by 1
Deep reinforcement learning in healthcare and biomedical application
Thakre, B. P., Yadav, U., & Bondre, S. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-30146-9.00008-3 · Cited by 2
Kim, Y., Choi, J., Chong, Y., Jiang, X., & Yu, H. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3132924 · Cited by 1
Understanding user experiences and expectations toward patient appointment systems
Huai, Y. (2023) · Applied and Computational Engineering · doi:10.54254/2755-2721/6/20230708 · Cited by 1
An Approach for Dynamic Optimization of Prevention Program Implementation in Stochastic Environments
Kang, Y., & Prabhu, V. V. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-19656-0_37 · Cited by 1
Deb, N., Dalal, A., & Basak, G. K. (2020) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2011.13960 · Cited by 1
Markov Decision Processes: Application to Treatment Planning
Baucum, M., & Khojandi, A. (2022) · Springer eBooks · doi:10.1007/978-3-030-54621-2_844-1 · Cited by 1
Cox, L. A. (2012) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-1-4614-6058-9_2
Modelling traditional Chinese medicine therapy planning with POMDP
Feng, Q. B., Zhou, X., Huang, H., Zhang, X., & Zhang, R. (2013) · International Journal of Functional Informatics and Personalised Medicine · doi:10.1504/ijfipm.2013.057405
Staged Multi-armed Bandits.Preprint
Tekin, C., & van der Schaar, M. (2015) · arXiv (Cornell University)
MDP Modeling for the Prediction of Agent Movement in Limited Space
Jin, H., Kim, S., Jung, C., & Lee, M. (2015) · 한국경영과학회지 · doi:10.7737/jkorms.2015.40.3.063
Optimal Timing of Statin Initiation for Patients with Type 2 Diabetes
Mason, J. E. (2009) · NCSU Libraries Repository (North Carolina State University Libraries)
Jimenez, E. (2015) · Journal of International Crisis and Risk Communication Research
Modeling Multi-level Incentives in Health Care: A Multiscale Decision Theory Approach
Zhang, H. (2016) · VTechWorks (Virginia Tech)
Essays on the management of appointments for chronic conditions
Nenova, Z. D. (2017) · D-Scholarship@Pitt (University of Pittsburgh)
Zaleski, J., & Peruvemba, R. (2019) · Advances in public policy and administration (APPA) book series · doi:10.4018/978-1-5225-7591-7.ch006
Hayward, R. A., Schell, G. J., Robinson, J. G., Sussman, J. B., & Lavieri, M. S. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/608216
Reis, W. A. S. (2019) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.45.2019.tde-02092019-212258
Sequential Learning-based IaaS Composition
Mistry, S., Fattah, S. M. M., & Bouguettaya, A. (2021) · ACM Transactions on the Web · doi:10.1145/3452332
(2013) · Elsevier eBooks · doi:10.1016/b978-0-12-407795-9.00027-x
Optimal Time Intervals Between Personal Mammogram Test Decisions
Sütçü, M. (2023) · International Journal of Advanced Natural Sciences and Engineering Researches · doi:10.59287/ijanser.650
Supervised reinforcement learning for recommending treatment strategies in sepsis
Sun, H., Chen, Q., & Yujing, Y. (2023) · International Conference on Information Science, Electrical and Automation Engineering (ISEAE) · doi:10.1117/12.2689783
A Graphical Model of Nocturnal Therapy of Juvenile Diabetics
Deja, R., & Deja, G. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.10.003
Reinforcement Learning for Hospital Outbreak Simulations
Kim, D., Gonzalez, G. V., & Juárez, J. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-95838-0_19
Alnemer, M. S., Kotliar, K. E., Neuhaus, V., Pape, H.-C., & Ciritsis, B. D. (2024) · Figshare · doi:10.6084/m9.figshare.26631937
Vidéosurveillance d’une salle de traitement en radiothérapie
Sotelo, M. P. (2011) · theses.fr (ABES) · doi:10.70675/11a95410zce9ez4343zba60zf0c443add7b4
Relatively Robust Timing of Medical Tests Under Ambiguous Prevalence Dynamics
Weber, T. A. (2026) · International Research Conference on Smart Computing and Systems Engineering (SCSE) · doi:10.1109/scse70081.2026.11499885
Karampelesis, P., Denazis, S., Koufopavlou, O., & Zacharaki, E. I. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1760912
Stochastic Processes for Adaptive Healthcare Systems
Sachan, S. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-9705-4.ch001
Reifsnider, O. S. (2011) · TigerPrints (Clemson University)
Metadata
- Authors
- Schaefer, A. J., Bailey, M. D., Shechter, S. M., & Roberts, M. S.
- Affiliations
- University of Pittsburgh (US)
- Venue
- International Series in Operations Research & Management Science
- Publisher
- Springer US
- Published
- 2005
- Pages
- 593-612
- ISSN
- 0884-8289 (ISSN-L)
- OpenAlex
- W58990062
- Cited by
- 155
- References
- 53
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 11 · 2023: 13 · 2022: 6 · 2021: 13 · 2020: 8 · 2019: 17 · 2018: 15 · 2017: 9 · 2016: 5 · 2015: 12 · 2014: 3 · 2013: 6 · 2012: 5
- FWCI
- 8.16
- Type
- book chapter
- Language
- en
- Open access
- closed
- Licence
- Locations
- International series in management science/operations research/International series in operations research & management science (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Social Sciences › Decision Sciences › Management Science and Operations Research › Complex Systems and Decision Making
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Keywords
- Markov decision process, treatment decision-making, sequential decision making, stochastic dynamic programming, treatment planning, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
