The DOI matches its Crossref record.
Dynamic Programming and Markov Processes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
Bayesian Networks and Decision Graphs
Jensen, F. V., & Nielsen, T. D. (2007) · Information Science and Statistics · doi:10.1007/978-0-387-68282-2 · Cited by 4,129
A Survey of Collaborative Filtering Techniques
Su, X., & Khoshgoftaar, T. M. (2009) · Advances in Artificial Intelligence · doi:10.1155/2009/421425 · Cited by 3,602
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
Markov games as a framework for multi-agent reinforcement learning
Littman, M. L. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50027-1 · Cited by 2,614
Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem
Vamvoudakis, K. G., & Lewis, F. L. (2010) · Automatica · doi:10.1016/j.automatica.2010.02.018 · Cited by 1,703
Optimal Operation of Multireservoir Systems: State-of-the-Art Review
Labadie, J. W. (2004) · Journal of Water Resources Planning and Management · doi:10.1061/(asce)0733-9496(2004)130:2(93 · Cited by 1,576
Self-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564
Reinforcement learning and adaptive dynamic programming for feedback control
Lewis, F. L., & Vrabie, D. (2009) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2009.933854 · Cited by 1,558
The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
Smallwood, R. D., & Sondik, E. J. (1973) · Operations Research · doi:10.1287/opre.21.5.1071 · Cited by 1,479
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,435
The Complexity of Markov Decision Processes
Papadimitriou, C. H., & Tsitsiklis, J. N. (1987) · Mathematics of Operations Research · doi:10.1287/moor.12.3.441 · Cited by 1,329
On the Theory of Infinitely Repeated Games with Discounting
Abreu, D. (1988) · Econometrica · doi:10.2307/1911077 · Cited by 1,252
Toward a Theory of Discounted Repeated Games with Imperfect Monitoring
Abreu, D., Pearce, D., & Stacchetti, E. (1990) · Econometrica · doi:10.2307/2938299 · Cited by 1,203
Coordinated multi-robot exploration
Burgard, W., Moors, M., Stachniss, C., & Schneider, F. E. (2005) · IEEE Transactions on Robotics · doi:10.1109/tro.2004.839232 · Cited by 1,177
Discrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof
Al-Tamimi, A., Lewis, F. L., & Abu-Khalaf, M. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.926614 · Cited by 1,075
Decision-Theoretic Planning: Structural Assumptions and Computational Leverage
Boutilier, C., Dean, T., & Hanks, S. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.575 · Cited by 1,074
Algorithms for Reinforcement Learning
Szepesvári, C. (2010) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01551-9 · Cited by 1,049
Learning metric-topological maps for indoor mobile robot navigation
Thrun, S. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(97)00078-7 · Cited by 1,042
Adaptive optimal control for continuous-time linear systems based on policy iteration
Vrabie, D., Pastravanu, O., Abu-Khalaf, M., & Lewis, F. L. (2009) · Automatica · doi:10.1016/j.automatica.2008.08.017 · Cited by 949
Information Science and Statistics
Andersen, E. B. (1986) · COMPSTAT · doi:10.1007/978-3-642-46890-2_1 · Cited by 899
Optimal and Autonomous Control Using Reinforcement Learning: A Survey
Kiumarsi, B., Vamvoudakis, K. G., Modares, H., & Lewis, F. L. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2773458 · Cited by 891
Optimal control of Markov processes with incomplete state information
Åström, K. J. (1965) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(65)90154-x · Cited by 854
Experiences with an interactive museum tour-guide robot
Burgard, W., Cremers, A. B., Fox, D., Hähnel, D., Lakemeyer, G., Schulz, D., Steiner, W., & Thrun, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00070-3 · Cited by 818
Multiagent learning using a variable learning rate
Bowling, M., & Veloso, M. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(02)00121-2 · Cited by 775
Collaborative multi-robot exploration
Burgard, W., Moors, M., Fox, D., Simmons, R., & Thrun, S. (2002) · Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065) · doi:10.1109/robot.2000.844100 · Cited by 762
Reinforcement learning in the brain
Niv, Y. (2009) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2008.12.005 · Cited by 717
MINERVA: a second-generation museum tour-guide robot
Thrun, S., Bennewitz, M., Burgard, W., Cremers, A. B., Dellaert, F., Fox, D., Hahnel, D., Rosenberg, C., Roy, N., Schulte, J., et al. (2003) · Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C) · doi:10.1109/robot.1999.770401 · Cited by 701
Modares, H., & Lewis, F. L. (2014) · Automatica · doi:10.1016/j.automatica.2014.05.011 · Cited by 701
A survey of maintenance models: The control and surveillance of deteriorating systems
Pierskalla, W. P., & Voelker, J. A. (1976) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800230302 · Cited by 678
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., ODoherty, J., & Pezzulo, G. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.022 · Cited by 674
Optimal cartel equilibria with imperfect monitoring
Abreu, D., Pearce, D., & Stacchetti, E. (1986) · Journal of Economic Theory · doi:10.1016/0022-0531(86)90028-1 · Cited by 665
Reinforcement learning algorithms: A brief survey
Shakya, A. K., Pillai, G., & Chakrabarty, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120495 · Cited by 656
Vrabie, D., & Lewis, F. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.03.008 · Cited by 634
Reinforcement Learning and Markov Decision Processes
van Otterlo, M., & Wiering, M. (2012) · Adaptation, Learning, and Optimization · doi:10.1007/978-3-642-27645-3_1 · Cited by 628
Al-Tamimi, A., Lewis, F. L., & Abu-Khalaf, M. (2007) · Automatica · doi:10.1016/j.automatica.2006.09.019 · Cited by 611
Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning
Luo, S. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106208 · Cited by 579
The Jigsaw continuous sensing engine for mobile phone applications
Lu, H., Yang, J., Liu, Z., Lane, N. D., Choudhury, T., & Campbell, A. T. (2010) · Proceedings of the 8th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/1869983.1869992 · Cited by 560
Survey of Model-Based Reinforcement Learning: Applications on Robotics
Polydoros, A. S., & Nalpantidis, L. (2017) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-017-0468-y · Cited by 534
Control Techniques for Complex Networks
Meyn, S. (2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511804410 · Cited by 528
Value-Function Approximations for Partially Observable Markov Decision Processes
Hauskrecht, M. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.678 · Cited by 527
The complexity of stochastic games
Condon, A. (1992) · Information and Computation · doi:10.1016/0890-5401(92)90048-k · Cited by 517
Probabilistic Algorithms and the Interactive Museum Tour-Guide Robot Minerva
Thrun, S., Beetz, M., Bennewitz, M., Burgard, W., Cremers, A. B., Dellaert, F., Fox, D., Hähnel, D., Rosenberg, C., Roy, N., et al. (2000) · The International Journal of Robotics Research · doi:10.1177/02783640022067922 · Cited by 516
Risk-Sensitive Markov Decision Processes
Howard, R. A., & Matheson, J. E. (1972) · Management Science · doi:10.1287/mnsc.18.7.356 · Cited by 501
Self-citationMaintenance Policies for Stochastically Failing Equipment: A Survey
McCall, J. J. (1965) · Management Science · doi:10.1287/mnsc.11.5.493 · Cited by 465
Contraction Mappings in the Theory Underlying Dynamic Programming
Denardo, E. V. (1967) · SIAM Review · doi:10.1137/1009030 · Cited by 456
On the Complexity of Solving Markov Decision Problems
Littman, M. L., Dean, T., & Kaelbling, L. P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1302.4971 · Cited by 424
A survey of point-based POMDP solvers
Shani, G., Pineau, J., & Kaplow, R. (2013) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-012-9200-2 · Cited by 422
Risk-averse dynamic programming for Markov decision processes
Ruszczyński, A. (2010) · Mathematical Programming · doi:10.1007/s10107-010-0393-3 · Cited by 415
White, C. C., & White, D. J. (1989) · European Journal of Operational Research · doi:10.1016/0377-2217(89)90348-2 · Cited by 415
Robust Adaptive Dynamic Programming and Feedback Stabilization of Nonlinear Systems
Jiang, Y., & Jiang, Z.-P. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2294968 · Cited by 414
Control Strategies for an Idealized Public Transportation System
Osuna, E. E., & Newell, G. F. (1972) · Transportation Science · doi:10.1287/trsc.6.1.52 · Cited by 409
Towards a mathematical theory of cortical micro-circuits
George, D., & Hawkins, J. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000532 · Cited by 404
Markov Decision Processes with Applications to Finance
Bäuerle, N., & Rieder, U. (2011) · Universitext · doi:10.1007/978-3-642-18324-9 · Cited by 398
Advances in Decision Analysis: From Foundations to Applications
(2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511611308 · Cited by 398
Value-function reinforcement learning in Markov games
Littman, M. L. (2001) · Cognitive Systems Research · doi:10.1016/s1389-0417(01)00015-8 · Cited by 397
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/1532443041827907 · Cited by 388
Average reward reinforcement learning: Foundations, algorithms, and empirical results
Mahadevan, S. (1996) · Machine Learning · doi:10.1007/bf00114727 · Cited by 385
Stochastic dynamic programming with factored representations
Boutilier, C., Dearden, R., & Goldszmidt, M. (2000) · Artificial Intelligence · doi:10.1016/s0004-3702(00)00033-3 · Cited by 361
Reinforcement Learning: A Tutorial Survey and Recent Advances
Gosavi, A. (2009) · INFORMS Journal on Computing · doi:10.1287/ijoc.1080.0305 · Cited by 326
Automated Verification Techniques for Probabilistic Systems
Forejt, V., Kwiatkowska, M., Norman, G., & Parker, D. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-21455-4_3 · Cited by 319
A unified framework for stochastic optimization
Powell, W. B. (2019) · European Journal of Operational Research · doi:10.1016/j.ejor.2018.07.014 · Cited by 314
Chapter 51 Structural estimation of markov decision processes
Rust, J. (1994) · Handbook of Econometrics · doi:10.1016/s1573-4412(05)80020-0 · Cited by 309
A Reinforcement Learning Method for Maximizing Undiscounted Rewards
Schwartz, A. (1993) · Machine Learning Proceedings 1993 · doi:10.1016/b978-1-55860-307-3.50045-9 · Cited by 308
Bounded-parameter Markov decision processes
Givan, R., Leach, S., & Dean, T. (2000) · Artificial Intelligence · doi:10.1016/s0004-3702(00)00047-3 · Cited by 308
Equivalence notions and model minimization in Markov decision processes
Givan, R., Dean, T., & Greig, M. (2003) · Artificial Intelligence · doi:10.1016/s0004-3702(02)00376-4 · Cited by 308
A˚ström, K. J., & Kumar, P. R. (2014) · Automatica · doi:10.1016/j.automatica.2013.10.012 · Cited by 304
Galvani, A. P., Reluga, T. C., & Chapman, G. B. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0606774104 · Cited by 282
Reinforcement learning design for cancer clinical trials
Zhao, Y., Kosorok, M. R., & Zeng, D. (2009) · Statistics in Medicine · doi:10.1002/sim.3720 · Cited by 271
A review on deep reinforcement learning for fluid mechanics
Garnier, P., Viquerat, J., Rabault, J., Larcher, A., Kuhnle, A., & Hachem, E. (2021) · Computers & Fluids · doi:10.1016/j.compfluid.2021.104973 · Cited by 270
Minimizing the average cost of paging and registration: A timer-based method
Rose, C. (1996) · Wireless Networks · doi:10.1007/bf01225634 · Cited by 263
Adaptive-critic-based neural networks for aircraft optimal control
Balakrishnan, S. N., & Biega, V. (1996) · Journal of Guidance, Control, and Dynamics · doi:10.2514/3.21715 · Cited by 261
On Nonterminating Stochastic Games
Hoffman, A. J., & Karp, R. M. (1966) · Management Science · doi:10.1287/mnsc.12.5.359 · Cited by 258
Dynamic multi-objective scheduling for flexible job shop by deep reinforcement learning
Luo, S., Zhang, L., & Fan, Y. (2021) · Computers & Industrial Engineering · doi:10.1016/j.cie.2021.107489 · Cited by 255
Finding approximate POMDP solutions through belief compression
Roy, N., Gordon, G., & Thrun, S. (2005) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1496 · Cited by 254
Active inference and agency: optimal control without cost functions
Friston, K., Samothrakis, S., & Montague, R. (2012) · Biological Cybernetics · doi:10.1007/s00422-012-0512-8 · Cited by 241
An object-oriented representation for efficient reinforcement learning
Diuk, C., Cohen, A., & Littman, M. L. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390187 · Cited by 226
Modeling Medical Treatment Using Markov Decision Processes
Schaefer, A. J., Bailey, M. D., Shechter, S. M., & Roberts, M. S. (2005) · International Series in Operations Research & Management Science · doi:10.1007/1-4020-8066-2_23 · Cited by 155
Markov decision processes with delays and asynchronous cost collection
Katsikopoulos, K. V., & Engelbrecht, S. E. (2003) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2003.809799 · Cited by 129
An Investigation of Strategies in Baseball
Lindsey, G. R. (1963) · Operations Research · doi:10.1287/opre.11.4.477 · Cited by 86
A multifaceted vision of the human-AI collaboration: a comprehensive review
Puerta-Beldarrain, M., Gómez-Carmona, O., Sánchez-Corcuera, R., Casado-Mansilla, D., López-de-Ipiña, D., & Chen, L. (2025) · IEEE Access · doi:10.1109/access.2025.3536095 · Cited by 73
Low Earth Orbit Satellite Based Communication Systems—Research Opportunities
Gavish, B. (1997) · European Journal of Operational Research · doi:10.1016/s0377-2217(96)00390-6 · Cited by 28
Rational thoughts in neural codes
Wu, Z., Kwon, M., Daptardar, S., Schrater, P., & Pitkow, X. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912336117 · Cited by 27
Time-critical dynamic decision modeling in medicine
Xiang, Y., & Poh, K.-L. (2002) · Computers in Biology and Medicine · doi:10.1016/s0010-4825(01)00036-1 · Cited by 22
Training-Based Antenna Selection for PER Minimization: A POMDP Approach
Padmanabhan, S., Stephen, R. G., Murthy, C. R., & Coupechoux, M. (2015) · IEEE Transactions on Communications · doi:10.1109/tcomm.2015.2455504 · Cited by 15
Macro, Finance, and Macro Finance: Solving Nonlinear Models in Continuous Time with Machine Learning
Duarte, V. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3012602 · Cited by 9
Metadata
- Authors
- Freimer, M., & Howard, R. A.
- Venue
- Journal of the American Statistical Association
- Publisher
- JSTOR
- Published
- 1961-06
- Volume
- 56
- Issue
- 294
- Pages
- 411
- ISSN
- 0162-1459 (ISSN-L)
- OpenAlex
- W2028145673
- Cited by
- 3,378
- References
- 0
- FWCI
- 14.84
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.