The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
The Optimal Control of Partially Observable Markov Processes over the Infinite Horizon: Discounted Costs
Abstract
This paper treats the discounted cost, optimal control problem for Markov processes with incomplete state information. The optimization approach for these partially observable Markov processes is a generalization of the well-known policy iteration technique for finding optimal stationary policies for completely observable Markov processes. The state space for the problem is the space of state occupancy probability distributions (the unit simplex). The development of the algorithm introduces several new ideas, including the class of finitely transient policies, which are shown to possess piecewise linear cost functions. The paper develops easily implemented approximations to stationary policies based on these finitely transient policies and shows that the concave hull of an approximation can be included in the well-known Howard policy improvement algorithm with subsequent convergence. The paper closes with a detailed example illustrating the application of the algorithm to the two-state partially observable Markov process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
Smallwood, R. D., & Sondik, E. J. (1973) · Operations Research · doi:10.1287/opre.21.5.1071 · Cited by 1,479
Self-citationDiscounted Dynamic Programming
Blackwell, D. (1965) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177700285 · Cited by 918
Iosifescu, M., & Howard, R. A. (1972) · International Statistical Review · doi:10.2307/1402771 · Cited by 897
Contraction Mappings in the Theory Underlying Dynamic Programming
Denardo, E. V. (1967) · SIAM Review · doi:10.1137/1009030 · Cited by 456
The optimal control of partially observable Markov processes
Sondik, E. J. (1971) · Defense Technical Information Center (DTIC) · Cited by 440
Self-citationArbitrary State Markovian Decision Processes
Ross, S. M. (1968) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177698041 · Cited by 102
Cited by
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
Decision-Theoretic Planning: Structural Assumptions and Computational Leverage
Boutilier, C., Dean, T., & Hanks, S. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.575 · Cited by 1,074
Learning policies for partially observable environments: Scaling up
Littman, M. L., Cassandra, A. R., & Kaelbling, L. P. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50052-9 · Cited by 650
Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey
Arapostathis, A., Borkar, V. S., Fernández-Gaucherand, E., Ghosh, M. K., & Marcus, S. I. (1993) · SIAM Journal on Control and Optimization · doi:10.1137/0331018 · Cited by 592
A survey of algorithmic methods for partially observed Markov decision processes
Lovejoy, W. S. (1991) · Annals of Operations Research · doi:10.1007/bf02055574 · Cited by 532
Value-Function Approximations for Partially Observable Markov Decision Processes
Hauskrecht, M. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.678 · Cited by 527
Probabilistic Algorithms and the Interactive Museum Tour-Guide Robot Minerva
Thrun, S., Beetz, M., Bennewitz, M., Burgard, W., Cremers, A. B., Dellaert, F., Fox, D., Hähnel, D., Rosenberg, C., Roy, N., et al. (2000) · The International Journal of Robotics Research · doi:10.1177/02783640022067922 · Cited by 516
Online Planning Algorithms for POMDPs
Ross, S., Pineau, J., Paquet, S., & Chaib-draa, B. (2008) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2567 · Cited by 496
A Survey of Multi-Objective Sequential Decision-Making
Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3987 · Cited by 461
A survey of point-based POMDP solvers
Shani, G., Pineau, J., & Kaplow, R. (2013) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-012-9200-2 · Cited by 422
Opportunistic Spectrum Access via Periodic Channel Sensing
Zhao, Q., Geirhofer, S., Tong, L., & Sadler, B. M. (2008) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2007.907867 · Cited by 372
Stochastic dynamic programming with factored representations
Boutilier, C., Dearden, R., & Goldszmidt, M. (2000) · Artificial Intelligence · doi:10.1016/s0004-3702(00)00033-3 · Cited by 361
Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes
Cassandra, A. R., Littman, M. L., & Zhang, N. L. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1302.1525 · Cited by 361
Liu, K., & Zhao, Q. (2010) · IEEE Transactions on Information Theory · doi:10.1109/tit.2010.2068950 · Cited by 347
Learning Without State-Estimation in Partially Observable Markovian Decision Processes
Singh, S. P., Jaakkola, T., & Jordan, M. I. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50042-8 · Cited by 327
Reinforcement Learning: A Tutorial Survey and Recent Advances
Gosavi, A. (2009) · INFORMS Journal on Computing · doi:10.1287/ijoc.1080.0305 · Cited by 326
A unified framework for stochastic optimization
Powell, W. B. (2019) · European Journal of Operational Research · doi:10.1016/j.ejor.2018.07.014 · Cited by 314
Computationally Feasible Bounds for Partially Observed Markov Decision Processes
Lovejoy, W. S. (1991) · Operations Research · doi:10.1287/opre.39.1.162 · Cited by 307
Markov decision processes with delays and asynchronous cost collection
Katsikopoulos, K. V., & Engelbrecht, S. E. (2003) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2003.809799 · Cited by 129
Khalvati, K., Kiani, R., & Rao, R. P. N. (2021) · Nature Communications · doi:10.1038/s41467-021-25419-4 · Cited by 64
Training-Based Antenna Selection for PER Minimization: A POMDP Approach
Padmanabhan, S., Stephen, R. G., Murthy, C. R., & Coupechoux, M. (2015) · IEEE Transactions on Communications · doi:10.1109/tcomm.2015.2455504 · Cited by 15
Learning State-Variable Relationships in POMCP: A Framework for Mobile Robots
Zuccotto, M., Piccinelli, M., Castellini, A., Marchesini, E., & Farinelli, A. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.819107 · Cited by 2
“Partially Observable Markov Decision Processes (POMDPs) (Sondik, 1978; Kaelbling et al., 1998) are a powerful framework for planning under uncertainty.”
Background
Metadata
- Authors
- Sondik, E. J.
- Venue
- Operations Research
- Published
- 1978-04
- Volume
- 26
- Issue
- 2
- Pages
- 282-304
- ISSN
- 0030-364X (ISSN-L)
- 1526-5463
- OpenAlex
- W2044375425
- Cited by
- 536
- References
- 6
- FWCI
- 4.43
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.