The DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
Modeling Medical Treatment Using Markov Decision Processes
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Currently, most treatment decisions in the medical domain are made via ad-hoc or heuristic approaches, but there is a growing body of evidence that such complex treatment decisions are better handled through modeling rather than intuition alone [18,21].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
In the citing paper
“Markov decision processes (MDPs) are one efficient technique for determining such optimal sequential decisions (termed a 'policy') in dynamic and uncertain environments [18,19], and have been explored in medical decision-making problems in recent years [18,20].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
In the citing paper
“As noted elsewhere, the collection of such 'natural history' disease data is fraught with many challenges, ethical and otherwise, particularly if such untreated conditions pose significant health risks [18].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
In the citing paper
“However, despite such applicability of sequential decisionmaking techniques like MDPs to medical decision-making, there have been relatively few applications in healthcare [18,19].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
In the citing paper
“The work presented here adds to a growing body of evidence that such complex treatment decisions may be better handled through modeling than intuition alone [18,21].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
In the citing paper
“MDPs (and their partially observable cousins) directly address many of the challenges faced in clinical decision-making [17,18].”
From Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach (2013) · cites this paper as [18]
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Methods for the Economic Evaluation of Health Care Programmes
Drummond, M. E., Sculpher, M. J., Torrance, G. W., O’Brien, B. J., & Stoddart, G. L. (2005) · doi:10.1093/oso/9780198529446.001.0001 · Cited by 13,648
(1995) · Cited by 10,558
Availability: A heuristic for judging frequency and probability
Tversky, A., & Kahneman, D. (1973) · Cognitive Psychology · doi:10.1016/0010-0285(73)90033-9 · Cited by 9,984 · cited 1×
“However, there is a growing feeling among medical practitioners that some treatment decisions are too complicated to solve accurately using intuition alone [1, 2].”
BackgroundMarkov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Cost-Effectiveness in Health and Medicine
(1996) · doi:10.1093/oso/9780195108248.001.0001 · Cited by 7,193
Time Discounting and Time Preference: A Critical Review
Frederick, S., Loewenstein, G., & O’Donoghue, T. (2002) · Journal of Economic Literature · doi:10.1257/002205102320161311 · Cited by 6,106
Puterman, M. L. (1994) · Wiley series in probability and statistics · doi:10.1002/9780470316887 · Cited by 6,057
Methods for the economic evaluation of health care programmes
Wilson, S. (1987) · Journal of Epidemiology & Community Health · doi:10.1136/jech.41.4.355-a · Cited by 4,312
Dynamic Programming and Markov Processes.
Freimer, M. L., & Howard, R. A. (1961) · Journal of the American Statistical Association · doi:10.2307/2282268 · Cited by 3,388
Shapley, L. S. (1953) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.39.10.1095 · Cited by 2,549
A Note on Measurement of Utility
Samuelson, P. A. (1937) · The Review of Economic Studies · doi:10.2307/2967612 · Cited by 2,140
Small Area Variations in Health Care Delivery
Wennberg, J. E., & Gittelsohn, A. M. (1973) · Science · doi:10.1126/science.182.4117.1102 · Cited by 1,891
Availability: A heuristic for judging frequency and probability
Tversky, A., & Kahneman, D. J. (1982) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809477.012 · Cited by 1,251
Multiattribute Utility Function for a Comprehensive Health Status Classification System
Torrance, G. W., Feeny, D. H., Furlong, W. J., Barr, R. D., ZHANG, Y., & Wang, Q. (1996) · Medical Care · doi:10.1097/00005650-199607000-00004 · Cited by 900
The Linear Programming Approach to Approximate Dynamic Programming
de Farias, D. P., & Roy, B. V. (2003) · Operations Research · doi:10.1287/opre.51.6.850.24925 · Cited by 700
Applying a New Device in the Optimization of Exponential Queuing Systems
Lippman, S. A. (1975) · Operations Research · doi:10.1287/opre.23.4.687 · Cited by 698
Time and Decision: Economic and Psychological Perspectives on Intertemporal Choice
Güth, W. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.04.003 · Cited by 601
Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey
Arapostathis, A., Borkar, V. S., Fernández-Gaucherand, E., Ghosh, M. K., & Marcus, S. I. (1993) · SIAM Journal on Control and Optimization · doi:10.1137/0331018 · Cited by 579
Handbook of Markov Decision Processes
Feinberg, E. A., & Shwartz, A. (2002) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-1-4615-0805-2 · Cited by 551
Geographic Variation in the Use of Breast-Conserving Treatment for Breast Cancer
Nattinger, A. B., Gottlieb, M. S., Veum, J. A., Yahnke, D. P., & Goodwin, J. S. (1992) · New England Journal of Medicine · doi:10.1056/nejm199204233261702 · Cited by 542
A survey of algorithmic methods for partially observed Markov decision processes
Lovejoy, W. S. (1991) · Annals of Operations Research · doi:10.1007/bf02055574 · Cited by 527
Social preferences for health states: An empirical evaluation of three measurement techniques
Torrance, G. W. (1976) · Socio-Economic Planning Sciences · doi:10.1016/0038-0121(76)90036-7 · Cited by 519
Modeling Myopic Decisions: Evidence for Hyperbolic Delay-Discounting within Subjects and Amounts
Kirby, K. N., & Maraković, N. N. (1995) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1995.1086 · Cited by 448
Regional Variation across the United States in the Management of Acute Myocardial Infarction
Pilote, L., Califf, R. M., Sapp, S. K., Miller, D. P., Mark, D. B., Weaver, W. D., Gore, J. M., Armstrong, P. W., Ohman, E. M., & Topol, E. J. (1995) · New England Journal of Medicine · doi:10.1056/nejm199508313330907 · Cited by 334
Time and Decision: Economic and Psychological Perspectives of Intertemporal Choice
Loewenstein, G. F., Read, D., & Baumeister, R. F. (2003) · Cited by 302
Discount Functions and the Measurement of Patients' Values
Christensen-Szalanski, J. J. J. (1984) · Medical decision making · doi:10.1177/0272989x8400400108 · Cited by 238
Regional Variation across the United States in the Management of Acute Myocardial Infarction
Viskin, S., Kitzis, I., & Belhassen, B. (1996) · New England Journal of Medicine · doi:10.1056/nejm199601183340314 · Cited by 238
Developing and Implementing Computerized Protocols for Standardization of Clinical Decisions
Morris, A. H. (2000) · Annals of Internal Medicine · doi:10.7326/0003-4819-132-5-200003070-00007 · Cited by 219
Technical Note—An Equivalence Between Continuous and Discrete Time Markov Decision Processes
Serfozo, R. F. (1979) · Operations Research · doi:10.1287/opre.27.3.616 · Cited by 212
Markov-Renewal Programming. I: Formulation, Finite Return Models
Jewell, W. S. (1963) · Operations Research · doi:10.1287/opre.11.6.938 · Cited by 181
Sufficient statistics in the optimum control of stochastic systems
Striebel, C. T. (1965) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(65)90027-2 · Cited by 180
Planning treatment of ischemic heart disease with partially observable Markov decision processes
Hauskrecht, M., & Fraser, H. (2000) · Artificial Intelligence in Medicine · doi:10.1016/s0933-3657(99)00042-1 · Cited by 176
An Equivalence between Continuous and Discrete Time Markov Decision Processes.
Serfozo, R. F. (1976) · Defense Technical Information Center (DTIC) · Cited by 159
Ahn, J., & Hornberger, J. C. (1996) · Management Science · doi:10.1287/mnsc.42.5.629 · Cited by 89
Markov-Renewal Programming. II: Infinite Return Models, Example
Jewell, W. S. (1963) · Operations Research · doi:10.1287/opre.11.6.949 · Cited by 89
Neuro-Dynamic Programming: Overview and Recent Trends
Roy, B. V. (2002) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-1-4615-0805-2_14 · Cited by 62
Solution Procedures for Partially Observed Markov Decision Processes
White, C. C., & Scherer, W. T. (1989) · Operations Research · doi:10.1287/opre.37.5.791 · Cited by 60
Optimal Control of a Birth and Death Epidemic Process
Lefèvre, C. (1981) · Operations Research · doi:10.1287/opre.29.5.971 · Cited by 58
Deciding when to intervene: a Markov decision process approach
Magni, P., Quaglini, S., Marchetti, M., & Barosi, G. (2000) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(00)00099-x · Cited by 55
Comparison of Some Suboptimal Control Policies in Medical Drug Therapy
Hu, C., Lovejoy, W. S., & Shafer, S. L. (1996) · Operations Research · doi:10.1287/opre.44.5.696 · Cited by 52
Elstein, A. S., Schwartz, A., & NENDAZ, M. R. (2002) · Springer international handbooks of education · doi:10.1007/978-94-010-0462-6_9 · Cited by 45
Decision making in health care
Kernick, D. P. (2003) · Focus on Alternative and Complementary Therapies · doi:10.1111/j.2042-7166.2003.tb05803.x · Cited by 39
Toward Electronic Medical Records That Improve Care
Tierney, W. M., Overhage, J. M., & McDonald, C. J. (1995) · Annals of Internal Medicine · doi:10.7326/0003-4819-122-9-199505010-00011 · Cited by 33
Roberts, M. S., & Smith, K. (2020) · CRC Press eBooks · doi:10.1201/9780429491368-2 · Cited by 14
Self-citationGillies, A. (2019) · CRC Press eBooks · doi:10.1201/9781315377315-16 · Cited by 9
(1992) · Lecture notes in economics and mathematical systems · doi:10.1007/978-3-642-48914-3 · Cited by 7
Pflug, G. C., & Dieter, U. (1992) · Springer eBooks · Cited by 2
Pflug, G., & Laxenburg, I. W. O. C. I. M. I. S. A. O. 1. (1992) · Cited by 1
Research works printed but not linked: 1
- Roberts, M.S. and D.C. Angus (2002). The optimal timing of liver transplantation: Final report R01 HS09694. University of Pittsburgh, Pittsburgh, PA.
Cited by
Sorted by citations
Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0213-5 · Cited by 1,290
Reinforcement Learning in Healthcare: A Survey
Yu, C., Liu, J., Nemati, S., & Yin, G. (2021) · ACM Computing Surveys · doi:10.1145/3477600 · Cited by 783
Bennett, C. C., & Hauser, K. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.12.003 · Cited by 378
“Markov decision processes (MDPs) are one efficient technique for determining such optimal sequential decisions (termed a “policy”) in dynamic and uncertain environments [18,19], and have been explored in medical decision-making problems in recent years [18,20].”
InfluentialMarkov decision processes: a tool for sequential decision making under uncertainty
Alagoz, O., Hsu, H., Schaefer, A. J., & Roberts, M. S. (2010) · Medical Decision Making · doi:10.1177/0272989x09353194 · Cited by 266
Self-citationData, M. C. (2020) · OAPEN (The OAPEN Foundation) · doi:10.1007/978-3-319-43742-2 · Cited by 273
Metadata
- Authors
- Schaefer, A. J., Bailey, M. D., Shechter, S. M., & Roberts, M. S.
- Affiliations
- University of Pittsburgh (US)
- Venue
- International Series in Operations Research & Management Science
- Publisher
- Springer US
- Published
- 2005
- Pages
- 593-612
- ISSN
- 0884-8289 (ISSN-L)
- OpenAlex
- W58990062
- Cited by
- 155
- References
- 53
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 11 · 2023: 13 · 2022: 6 · 2021: 13 · 2020: 8 · 2019: 17 · 2018: 15 · 2017: 9 · 2016: 5 · 2015: 12 · 2014: 3 · 2013: 6 · 2012: 5
- FWCI
- 8.16
- Type
- book chapter
- Language
- en
- Open access
- closed
- Licence
- Locations
- International series in management science/operations research/International series in operations research & management science (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Social Sciences › Decision Sciences › Management Science and Operations Research › Complex Systems and Decision Making
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Keywords
- Markov decision process, treatment decision-making, sequential decision making, stochastic dynamic programming, treatment planning, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
