The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Online Planning Algorithms for POMDPs
Open accessAbstract
Partially Observable Markov Decision Processes (POMDPs) provide a rich framework for sequential decision-making under uncertainty in stochastic domains. However, solving a POMDP is often intractable except for small problems due to their complexity. Here, we focus on online approaches that alleviate the computational complexity by computing good local policies at each decision step during the execution. Online algorithms generally consist of a lookahead search to find the best action to execute at each time step in an environment. Our objectives here are to survey the various existing online POMDP methods, analyze their properties and discuss their advantages and disadvantages; and to thoroughly evaluate these online approaches in different environments under various metrics (return, error bound reduction, lower bound improvement). Our experimental results indicate that state-of-the-art online heuristic search methods can handle large POMDP domains efficiently.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
Principles of Artificial Intelligence
Nilsson, N. J. (1982) · doi:10.1007/978-3-662-09438-9 · Cited by 3,457 · cited 1×
“Washington (1997) proposed a slightly different approach inspired by the AO∗ algorithm (Nilsson, 1980), where the search is only conducted in the best solution graph.”
MethodThe Optimal Control of Partially Observable Markov Processes over a Finite Horizon
Smallwood, R. D., & Sondik, E. J. (1973) · Operations Research · doi:10.1287/opre.21.5.1071 · Cited by 1,479
The Complexity of Markov Decision Processes
Papadimitriou, C. H., & Tsitsiklis, J. N. (1987) · Mathematics of Operations Research · doi:10.1287/moor.12.3.441 · Cited by 1,329
Learning to act using real-time dynamic programming
Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(94)00011-o · Cited by 1,104
Point-based value iteration: an anytime algorithm for POMDPs
Pineau, J., Gordon, G., & Thrun, S. (2003) · Cited by 918
Self-citationOptimal control of Markov processes with incomplete state information
Åström, K. J. (1965) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(65)90154-x · Cited by 854
Principles of artificial intelligence
McDermott, J. (1980) · Artificial Intelligence · doi:10.1016/0004-3702(80)90026-0 · Cited by 814
Learning policies for partially observable environments: Scaling up
Littman, M. L., Cassandra, A. R., & Kaelbling, L. P. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50052-9 · Cited by 650
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L. (2005) · Cited by 588
Sondik, E. J. (1978) · Operations Research · doi:10.1287/opre.26.2.282 · Cited by 536
Value-Function Approximations for Partially Observable Markov Decision Processes
Hauskrecht, M. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.678 · Cited by 527
Perseus: Randomized point-based value iteration for POMDPs
Spaan, M. T. J., & Vlassis, N. A. (2005) · UvA-DARE (University of Amsterdam) · Cited by 526
A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes
Kearns, M., Mansour, Y., & Ng, A. Y. (2002) · Machine Learning · doi:10.1023/a:1017932429737 · Cited by 488
Tractable Inference for Complex Stochastic Processes
Boyen, X., & Koller, D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.7362 · Cited by 471
The optimal control of partially observable Markov processes
Sondik, E. J. (1971) · Defense Technical Information Center (DTIC) · Cited by 440
(2004) · doi:10.5555/1036843.1036906 · Cited by 417
LAO∗: A heuristic search algorithm that finds solutions with loops
Hansen, E. A., & Zilberstein, S. (2001) · Artificial Intelligence · doi:10.1016/s0004-3702(01)00106-0 · Cited by 388
Anytime Point-Based Approximations for Large POMDPs
Pineau, J., Gordon, G., & Thrun, S. (2006) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2078 · Cited by 370
Self-citationA Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms
Monahan, G. E., & MONAHANt, G. E. (2007) · Cited by 368
Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes
Cassandra, A. R., Littman, M. L., & Zhang, N. L. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1302.1525 · Cited by 361
Algorithms for Sequential Decision Making
Littman, M. L. (1996) · Cited by 341
Computationally Feasible Bounds for Partially Observed Markov Decision Processes
Lovejoy, W. S. (1991) · Operations Research · doi:10.1287/opre.39.1.162 · Cited by 307
Madani, O., Hanks, S., & Condon, A. (1999) · National Conference on Artificial Intelligence · Cited by 305
Exploiting structure to efficiently solve large scale partially observable markov decision processes
Poupart, P. (2005) · Cited by 247
Solving POMDPs by Searching in Policy Space
Hansen, E. A. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.7380 · Cited by 187
Planning treatment of ischemic heart disease with partially observable Markov decision processes
Hauskrecht, M., & Fraser, H. (2000) · Artificial Intelligence in Medicine · doi:10.1016/s0933-3657(99)00042-1 · Cited by 173
Bounded Finite State Controllers
Poupart, P., & Boutilier, C. E. (2003) · Cited by 167
A point-based POMDP algorithm for robot planning
Spaan, M. T. J., & Spaan, N. (2004) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2004.1307420 · Cited by 145
Speeding Up the Convergence of Value Iteration in Partially Observable Markov Decision Processes
Zhang, N. L., & Zhang, W. (2001) · Journal of Artificial Intelligence Research · doi:10.1613/jair.761 · Cited by 140
Point-Based POMDP Algorithms: Improved Analysis and ImplementationPreprint
Smith, T., & Simmons, R. G. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.1412 · Cited by 115
Koenig, S. (2001) · doi:10.1609/aimag.v22i4.1596 · Cited by 106
An online POMDP algorithm for complex multiagent environments
Paquet, S., Tobin, L., & Chaib-draa, B. (2005) · Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems · doi:10.1145/1082473.1082620 · Cited by 92
Self-citationAEMS: an anytime online search algorithm for approximate policy refinement in large POMDPs
Ross, S., & Chaib-draa, B. (2007) · International Joint Conference on Artificial Intelligence · Cited by 87
Self-citationParallel Rollout for Online Solution of Partially Observable Markov Decision Processes
Chang, H. S., Givan, R. L., & Chong, E. K. P. (2004) · Discrete Event Dynamic Systems · doi:10.1023/b:disc.0000028199.78776.c4 · Cited by 77
Approximate Planning for Factored POMDPs using Belief State SimplificationPreprint
McAllester, D., & Singh, S. P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.6719 · Cited by 71
Tractable planning under uncertainty: exploiting structure
Pineau, J., Gordon, G. J., & Thrun, S. (2004) · Cited by 57
Self-citationStochastic local search for POMDP controllers
Braziunas, D., & Boutilier, C. E. (2004) · TSpace (University of Toronto) · Cited by 52
Markovian Decision Processes with Probabilistic Observation of States
Satia, J., & Lave, R. E. (1973) · Management Science · doi:10.1287/mnsc.20.1.1 · Cited by 47
BI-POMDP: Bounded, incremental partially-observable Markov-model planning
Washington, R. M. (1997) · Lecture notes in computer science · doi:10.1007/3-540-63912-8_105 · Cited by 41
Theoretical Analysis of Heuristic Search Methods for Online POMDPs.
Ross, S., Pineau, J., & Chaib-draa, B. (2008) · PubMed · Cited by 28
Self-citationRollout algorithms for stochastic scheduling problems
Bertsekas, D. P., & Castañón, D. A. (2002) · Proceedings of the IEEE Conference on Decision & Control, including the Symposium on Adaptive Processes · doi:10.1109/cdc.1998.758655 · Cited by 27
A fast point-based algorithm for POMDPs
Vlassis, N. A., & Spaan, M. T. J. (2004) · UvA-DARE (University of Amsterdam) · Cited by 26
Paquet, E. (2006) · Cited by 22
Rollout Algorithms for Stochastic Scheduling Problems
Bertsekas, D. P., & Castanon, D. A. (1998) · doi:10.21236/ada459559 · Cited by 20
Adaptation for Changing Stochastic Environments through Online POMDP Policy Learning
Shani, G., Brafman, R. I., & Shimony, S. E. (2005) · Cited by 14
Cited by
A survey of uncertainty in deep neural networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10562-9 · Cited by 1,406
Algorithms for Reinforcement Learning
Szepesvári, C. (2010) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01551-9 · Cited by 1,049
Adaptive Submodularity: Theory and Applications in Active Learning and Stochastic Optimization
Golovin, D., & Krause, A. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1003.3967 · Cited by 461
Bennett, C. C., & Hauser, K. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.12.003 · Cited by 378
“Here, we present an online approach where the system continually plans, executes, observes, and re-plans the optimal treatment strategy from any given timepoint [33].”
Intention-aware online POMDP planning for autonomous driving in a crowd
Bai, H., Cai, S., Ye, N., Hsu, D., & Lee, W. S. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7139219 · Cited by 321
Metadata
- Authors
- Ross, S., Pineau, J., Paquet, S., & Chaib-draa, B.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2008-07-29
- Volume
- 32
- Issue
- 2
- Pages
- 663-704
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W2144913588
- Cited by
- 496
- References
- 48
- FWCI
- 43.99
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.