Markov decision processes: Discrete stochastic dynamic programming
Abstract
From the Publisher: The past decade has seen considerable theoretical and applied research on Markov decision processes, as well as the growing use of these models in ecology, economics, communications engineering, and other fields where outcomes are uncertain and sequential decision-making processes are needed. A timely response to this increased activity, Martin L. Puterman's new work provides a uniquely up-to-date, unified, and rigorous treatment of the theoretical, computational, and applied research on Markov decision process models. It discusses all major research directions in the field, highlights many significant applications of Markov decision processes models, and explores numerous important topics that have previously been neglected or given cursory coverage in the literature. Markov Decision Processes focuses primarily on infinite horizon discrete time models and models with discrete time spaces while also examining models with arbitrary state spaces, finite horizon models, and continuous-time discrete state models. The book is organized around optimality criteria, using a common framework centered on the optimality (Bellman) equation for presenting results. The results are presented in a theorem-proof format and elaborated on through both discussion and examples, including results that are not available in any other book. A two-state Markov decision process model, presented in Chapter 3, is analyzed repeatedly throughout the book and demonstrates many results an…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
Rudin, W. (1973) · Cited by 13,708 · see citing works
Introduction to Stochastic Dynamic Programming.
Berry, D. A., & Ross, S. M. (1986) · Journal of the American Statistical Association · doi:10.2307/2289262 · Cited by 1,159 · see citing works
White, C. C., & White, D. J. (1989) · European Journal of Operational Research · doi:10.1016/0377-2217(89)90348-2 · Cited by 422 · see citing works
Dynamic programming: Models and applications
Denardo, E. V. (1982) · CERN Document Server (European Organization for Nuclear Research) · Cited by 409 · see citing works
Finding Optimal (s, S) Policies Is About As Simple As Evaluating a Single Policy
Zheng, Y., & Federgruen, A. (1991) · Operations Research · doi:10.1287/opre.39.4.654 · Cited by 317 · see citing works
Optimization over Time. Dynamic Programming and Stochastic Control. Volume 1
Thomas, L. C., & Whittle, P. (1983) · Journal of the Operational Research Society · doi:10.1057/jors.1983.56 · Cited by 162 · see citing works
Dynamic Programming and Its Application
Puterman, M. L. (1979) · Academic Press, Inc. eBooks · Cited by 57 · see citing works
Self-citationOn Homogeneous Markov Models with Continuous Time and Finite or Countable State Space
Yushkevic, A. A., & Fainberg, E. A. (1979) · Theory of Probability and Its Applications · doi:10.1137/1124014 · Cited by 34 · see citing works
Finite Dynamic Programming: An Approach to Finite Markov Decision Processes
White, D. J. (1978) · Andalas University Repository (Andalas University) · Cited by 24 · see citing works
Handbooks in operations research and management science volume 2: Stochastic models
Pintelon, L. M. (1991) · European Journal of Operational Research · doi:10.1016/0377-2217(91)90047-y · Cited by 13 · see citing works
Markov: A methodology for the solution of infinite time horizon markov decision processes
Williams, B. K. (1988) · Applied Stochastic Models and Data Analysis · doi:10.1002/asm.3150040405 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · see citing works
Machine learning: Algorithms, real-world applications and research directions
Sarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00592-x · Cited by 5,583 · see citing works
“The problem to be solved in reinforcement learning (RL) is defined as a Markov Decision Process (MDP) [86], i.”
InfluentialPlanning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · see citing works
Bayesian Networks and Decision Graphs
Jensen, F. V., & Nielsen, T. D. (2007) · Information science and statistics · doi:10.1007/978-0-387-68282-2 · Cited by 4,157 · see citing works
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · see citing works
Sarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00815-1 · Cited by 2,707 · see citing works
Theory and Applications of Robust Optimization
Brown, D. B., Caramanis, C., & Bertsimas, D. (2010) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,524 · see citing works
Deep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342 · see citing works
A Comprehensive Survey of Multiagent Reinforcement Learning
Busoniu, L., Babuska, R., & De Schutter, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.913919 · Cited by 2,235 · see citing works
Applications of Deep Reinforcement Learning in Communications and Networking: A Survey
Luong, N. C., Hoang, D. T., Gong, S., Niyato, D. T., Wang, P., Liang, Y., & Kim, D. I. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2916583 · Cited by 2,111 · see citing works
Generative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942 · see citing works
Mobile cloud computing: A survey
Fernando, N., Loke, S. W., & Rahayu, W. J. (2012) · Future Generation Computer Systems · doi:10.1016/j.future.2012.05.023 · Cited by 1,663 · see citing works
Handbook of Constraint Programming
Rossi, F., Beek, P. V., & Walsh, T. (2006) · Foundations of artificial intelligence · doi:10.1016/s1574-6526(06)x8001-x · Cited by 1,787 · see citing works
Transfer Learning for Reinforcement Learning Domains: A Survey
Taylor, M. E., & Stone, P. (2009) · doi:10.5555/1577069.1755839 · Cited by 1,521 · see citing works
Nichols, J. M., & WILLIAMS, B. (2006) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2006.08.007 · Cited by 1,319 · see citing works
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · RePEc: Research Papers in Economics · Cited by 1,310 · see citing works
Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0213-5 · Cited by 1,290 · see citing works
Artificial Intelligence: A New Synthesis
(1998) · Elsevier eBooks · doi:10.1016/c2009-0-27773-7 · Cited by 1,207 · see citing works
Foundations of Machine Learning
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018) · doi:10.7551/mitpress/10290.001.0001 · Cited by 1,122 · see citing works
(2015) · Journal of Machine Learning Research · Cited by 1,119 · see citing works
Algorithms for reinforcement learning
Szepesvári, C. (2010) · Synthesis lectures on artificial intelligence and machine learning · doi:10.1007/978-3-031-01551-9 · Cited by 308 · see citing works
Action understanding as inverse planning
Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009) · Cognition · doi:10.1016/j.cognition.2009.07.005 · Cited by 920 · see citing works
Nash Q-Learning for General-Sum Stochastic Games
Hu, J., Wellman, M. P., & Boutilier, C. (2003) · Cited by 909 · see citing works
The misinterpretation of noncanonical sentences
Ferreira, F. (2003) · Cognitive Psychology · doi:10.1016/s0010-0285(03)00005-7 · Cited by 876 · see citing works
Opponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844 · see citing works
Deep Direct Reinforcement Learning for Financial Signal Representation and Trading
Deng, Y., Bao, F., Kong, Y., Ren, Z., & Dai, Q. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2522401 · Cited by 822 · see citing works
Near-Optimal Reinforcement Learning in Polynomial Time
Kearns, M. J., & Singh, S. P. (2002) · Machine Learning · doi:10.1023/a:1017984413808 · Cited by 795 · see citing works
SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces
Kurniawati, H., Hsu, D., & Lee, W. S. (2008) · Robotics: Science and Systems · doi:10.15607/rss.2008.iv.009 · Cited by 776 · see citing works
Model-checking algorithms for continuous-time markov chains
Christel Baier, A., Haverkort, B. R. H. M., Hermanns, H., & Katoen, J.-P. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1205180 · Cited by 762 · see citing works
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762 · see citing works
Multi-agent Reinforcement Learning: An Overview
Buşoniu, L., Babuška, R., & Schutter, B. D. (2010) · Studies in computational intelligence · doi:10.1007/978-3-642-14435-6_7 · Cited by 749 · see citing works
The Option-Critic Architecture
Bacon, P., Harb, J., & Precup, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10916 · Cited by 219 · see citing works
Near-optimal Regret Bounds for Reinforcement Learning
Jaksch, T., Ortner, R., & Auer, P. L. (2010) · Cited by 716 · see citing works
A Distributional Perspective on Reinforcement LearningPreprint
Bellemare, M. G., Dabney, W., & Munos, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06887 · Cited by 724 · see citing works
Robust Control of Markov Decision Processes with Uncertain Transition Matrices
Nilim, A., & Ghaoui, L. E. (2005) · Operations Research · doi:10.1287/opre.1050.0216 · Cited by 702 · see citing works
A survey and critique of multiagent deep reinforcement learning
Hernández-Leal, P., Kartal, B., & Taylor, M. E. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09421-1 · Cited by 747 · see citing works
Reinforcement learning algorithms: A brief survey
Shakya, A. K., Pillai, G. N., & Chakrabarty, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120495 · Cited by 700 · see citing works
Noisy Networks for ExplorationPreprint
Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.10295 · Cited by 664 · see citing works
Scripting: higher level programming for the 21st Century
Ousterhout, J. K. (1998) · Computer · doi:10.1109/2.660187 · Cited by 648 · see citing works
Doya, K. (2008) · Nature Neuroscience · doi:10.1038/nn2077 · Cited by 642 · see citing works
Q-Learning Algorithms: A Comprehensive Classification and Applications
Jang, B., Kim, M., Harerimana, G., & Kim, J. W. (2019) · IEEE Access · doi:10.1109/access.2019.2941229 · Cited by 643 · see citing works
Reinforcement Learning and Markov Decision Processes
van Otterlo, M., & Wiering, M. (2012) · Adaptation, learning, and optimization · doi:10.1007/978-3-642-27645-3_1 · Cited by 631 · see citing works
A survey of inverse reinforcement learning: Challenges, methods and progress
Arora, S., & Doshi, P. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103500 · Cited by 619 · see citing works
Reinforcement Learning with Hierarchies of Machines
Parr, R., & Russell, S. (1997) · Cited by 618 · see citing works
Ratliff, N., Bagnell, J. A., & Zinkevich, M. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143936 · Cited by 617 · see citing works
Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms
Singh, S. P., Jaakkola, T., Littman, M. L., & Szepesvári, C. (2000) · Machine Learning · doi:10.1023/a:1007678930559 · Cited by 616 · see citing works
Advances in Energy Harvesting Communications: Past, Present, and Future Challenges
Ku, M.-L., Li, W., Chen, Y., & Ray Liu, K. J. (2016) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2015.2497324 · Cited by 606 · see citing works
Distributional Reinforcement Learning With Quantile Regression
Dabney, W., Rowland, M., Bellemare, M., & Munos, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11791 · Cited by 612 · see citing works
Decision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580 · see citing works
Iyengar, G. (2005) · Mathematics of Operations Research · doi:10.1287/moor.1040.0129 · Cited by 580 · see citing works
Natural actor–critic algorithms
Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M., & Lee, M. (2009) · Automatica · doi:10.1016/j.automatica.2009.07.008 · Cited by 563 · see citing works
Value-Function Approximations for Partially Observable Markov Decision Processes
Hauskrecht, M. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.678 · Cited by 534 · see citing works
Model-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000086 · Cited by 668 · see citing works
Peng, H., & Shen, X. (2020) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3036962 · Cited by 525 · see citing works
A comprehensive survey on model compression and acceleration
Choudhary, T., Mishra, V. K., Goswami, A., & Jagannathan, S. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09816-7 · Cited by 519 · see citing works
Perseus: Randomized point-based value iteration for POMDPs
Spaan, M. T. J., & Vlassis, N. A. (2005) · UvA-DARE (University of Amsterdam) · Cited by 526 · see citing works
Security and Game Theory: Algorithms, Deployed Systems, Lessons Learned
Tambe, M. (2011) · Cited by 509 · see citing works
Frameworks and Results in Distributionally Robust Optimization
Rahimian, H., & Mehrotra, S. (2022) · Open Journal of Mathematical Optimization · doi:10.48550/arxiv.1908.05659 · Cited by 504 · see citing works
Two formal analyses of attack graphs
Jha, S., Sheyner, O., & Wing, J. M. (2003) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csfw.2002.1021806 · Cited by 502 · see citing works
Online Planning Algorithms for POMDPs
Ross, S., Pineau, J., Paquet, S., & Chaib-draa, B. (2008) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2567 · Cited by 503 · see citing works
Value-Decomposition Networks For Cooperative Multi-Agent LearningPreprint
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. (2017) · arXiv (Cornell University) · doi:10.4230/lipics.cp.2023.25 · Cited by 489 · see citing works
Agent-Oriented Supply-Chain Management
Fox, M. S., Barbuceanu, M., & Teigen, R. (2001) · Springer eBooks · doi:10.1007/978-1-4615-1599-9_5 · Cited by 482 · see citing works
On the Sample Complexity of Reinforcement Learning
Kakade, S. M. (2003) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 510 · see citing works
Relative Entropy Policy Search
Peters, J., Mülling, K., & Altün, Y. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7727 · Cited by 467 · see citing works
Reinforcement learning for control: Performance, stability, and deep approximators
Buşoniu, L., de Bruin, T., Tolić, D., Kober, J., & Palunko, I. (2018) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2018.09.005 · Cited by 468 · see citing works
Opponency revisited: competition and cooperation between dopamine and serotonin
Boureau, Y.-L., & Dayan, P. (2011) · Neuropsychopharmacology · doi:10.1038/npp.2010.151 · Cited by 456 · see citing works
Cooperative Heterogeneous Multi-Robot Systems
Rizk, Y., Awad, M., & Tunstel, E. W. (2019) · ACM Computing Surveys · doi:10.1145/3303848 · Cited by 460 · see citing works
A stochastic control strategy for hybrid electric vehicles
Lin, C.-C., Peng, H., & Grizzle, J. W. (2004) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.2004.1384056 · Cited by 450 · see citing works
Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103535 · Cited by 445 · see citing works
Applications of reinforcement learning in energy systems
Perera, A. T. D., & Kamalaruban, P. (2021) · Renewable & Sustainable Energy Reviews · doi:10.1016/j.rser.2020.110618 · Cited by 388 · see citing works
An analysis of model-based Interval Estimation for Markov Decision Processes
Strehl, A. L., & Littman, M. L. (2008) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2007.08.009 · Cited by 428 · see citing works
A survey of point-based POMDP solvers
Shani, G., Pineau, J., & Kaplow, R. (2012) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-012-9200-2 · Cited by 427 · see citing works
Risk-averse dynamic programming for Markov decision processes
Ruszczyński, A. (2010) · Mathematical Programming · doi:10.1007/s10107-010-0393-3 · Cited by 418 · see citing works
System-level power optimization
Luca Benini, C., & Micheli, G. D. (2000) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/335043.335044 · Cited by 415 · see citing works
Fully Decentralized Multi-Agent Reinforcement Learning with Networked AgentsPreprint
Zhang, K., Yang, Z., Liu, H., Zhang, T., & Başar, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.08757 · Cited by 426 · see citing works
State of the art on automatic road extraction for GIS update: a novel classification
Mena, J. B. (2003) · Pattern Recognition Letters · doi:10.1016/s0167-8655(03)00164-8 · Cited by 411 · see citing works
Autonomous experimentation systems for materials development: A community perspective
Stach, E. A., DeCost, B., Kusne, A. G., Hattrick-Simpers, J. R., Brown, K. A., Reyes, K. G., Schrier, J., Billinge, S. J. L., Buonassisi, T., Foster, I. T., et al. (2021) · Matter · doi:10.1016/j.matt.2021.06.036 · Cited by 412 · see citing works
Value-function reinforcement learning in Markov games
Littman, M. L. (2001) · Cognitive Systems Research · doi:10.1016/s1389-0417(01)00015-8 · Cited by 396 · see citing works
An MDP-Based Vertical Handoff Decision Algorithm for Heterogeneous Wireless Networks
Stevens‐Navarro, E., Lin, Y., & Wong, V. W. S. (2008) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2007.907072 · Cited by 395 · see citing works
Reinforcement learning in robotic applications: a comprehensive survey
Singh, B., Kumar, R., & Singh, V. P. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09997-9 · Cited by 396 · see citing works
Markov Decision Processes with Applications to Finance
Bäuerle, N., & Rieder, U. (2011) · Universitext · doi:10.1007/978-3-642-18324-9 · Cited by 395 · see citing works
Lin, J., Rao, Y., Lu, J., & Zhou, J. (2017) · neural information processing systems · Cited by 394 · see citing works
Lee, J. H., Shin, J., & Realff, M. J. (2017) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2017.10.008 · Cited by 385 · see citing works
Average reward reinforcement learning: Foundations, algorithms, and empirical results
Mahadevan, S. (1996) · Machine Learning · doi:10.1007/bf00114727 · Cited by 386 · see citing works
Fuzzy inference system learning by reinforcement methods
Jouffe, L. (1998) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/5326.704563 · Cited by 385 · see citing works
Planning, Learning and Coordination in Multiagent Decision Processes
Boutilier, C. E. (1996) · Cited by 395 · see citing works
Serotonin in Affective Control
Dayan, P., & Huys, Q. J. M. (2009) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.051508.135607 · Cited by 381 · see citing works
SPUDD: Stochastic Planning using Decision DiagramsPreprint
Hoey, J., St‐Aubin, R., Hu, A. J., & Boutilier, C. E. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.6704 · Cited by 387 · see citing works
Opportunistic Spectrum Access via Periodic Channel Sensing
Zhao, Q., Geirhofer, S., Tong, L., & Sadler, B. M. (2008) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2007.907867 · Cited by 372 · see citing works
FOR LEAST-SQUARES POLICY ITERATION
Schwab, D. (2016) · Cited by 372 · see citing works
Greenwald, A., & Hall, K. (2003) · Cited by 384 · see citing works
Online Learning for Offloading and Autoscaling in Energy Harvesting Mobile Edge Computing
Xu, J., Chen, L., & Ren, S. (2017) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2017.2725277 · Cited by 368 · see citing works
Efficient Solution Algorithms for Factored MDPs
Guestrin, C., Koller, D., Parr, R., & Venkataraman, S. (2003) · Cited by 384 · see citing works
Efficient Reductions for Imitation Learning
Ross, S., & Bagnell, D. (2018) · doi:10.1184/r1/6554699 · Cited by 366 · see citing works
Kernel-Based Reinforcement Learning
Ormoneit, D., & Sen, S. (2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_100235 · Cited by 381 · see citing works
Optimality of Myopic Sensing in Multichannel Opportunistic Access
Ahmad, S. H. A., Liu, M., Javidi, T., Zhao, Q. Q., & Krishnamachari, B. (2009) · IEEE Transactions on Information Theory · doi:10.1109/tit.2009.2025561 · Cited by 364 · see citing works
Even-Dar, E., & Mansour, Y. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44581-1_39 · Cited by 364 · see citing works
Stochastic dynamic programming with factored representations
Boutilier, C., Dearden, R., & Goldszmidt, M. (2000) · Artificial Intelligence · doi:10.1016/s0004-3702(00)00033-3 · Cited by 361 · see citing works
Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation
Dayan, P., & Berridge, K. C. (2014) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0277-8 · Cited by 357 · see citing works
Exploiting Structure in Policy Construction
Boutilier, C. E., Dearden, R., & Goldszmidt, M. (1995) · Cited by 350 · see citing works
- Authors
- Hazeghi, K., & Puterman, M. L.
- ORCID
- Puterman, Martin L.
- Venue
- Journal of the American Statistical Association
- Publisher
- JSTOR
- Published
- 1995-03
- Volume
- 90
- Issue
- 429
- Pages
- 392
- ISSN
- 0162-1459 (ISSN-L)
- OpenAlex
- W2119567691
- Cited by
- 8,037
- References
- 12
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 86 · 2023: 145 · 2022: 233 · 2021: 616 · 2020: 674 · 2019: 595 · 2018: 480 · 2017: 459 · 2016: 450 · 2015: 507 · 2014: 451 · 2013: 420 · 2012: 392
- Type
- book review
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- Markov decision process, stochastic dynamic programming, discrete-time models, infinite horizon model, finite horizon model, Bellman equation, optimality criteria, average reward criterion, constrained models, sequential decision making under uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
