The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Optimal and Approximate Q-value Functions for Decentralized POMDPs
Open accessAbstract
Decision-theoretic planning is a popular approach to sequential decision making problems, because it treats uncertainty in sensing and acting in a principled way. In single-agent frameworks like MDPs and POMDPs, planning can be carried out by resorting to Q-value functions: an optimal Q-value function Q* is computed in a recursive manner by dynamic programming, and then an optimal policy is extracted from Q*. In this paper we study whether similar Q-value functions can be defined for decentralized POMDP models (Dec-POMDPs), and how policies can be extracted from such value functions. We define two forms of the optimal Q-value function for Dec-POMDPs: one that gives a normative description as the Q-value function of an optimal pure joint policy and another one that is sequentially rational and thus gives a recipe for computation. This computation, however, is infeasible for all but the smallest problems. Therefore, we analyze various approximate Q-value functions that allow for efficient computation. We describe how they relate, and we prove that they all provide an upper bound to the optimal Q-value function Q*. Finally, unifying some previous approaches for solving Dec-POMDPs, we describe a family of algorithms for extracting policies from such Q-value functions, and perform an experimental evaluation on existing test problems, including a new firefighting benchmark problem.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
The Complexity of Decentralized Control of Markov Decision Processes
Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002) · Mathematics of Operations Research · doi:10.1287/moor.27.4.819.297 · Cited by 1,192
Decision-Theoretic Planning: Structural Assumptions and Computational Leverage
Boutilier, C., Dean, T., & Hanks, S. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.575 · Cited by 1,074
Point-based value iteration: an anytime algorithm for POMDPs
Pineau, J., Gordon, G., & Thrun, S. (2003) · Cited by 918
Learning policies for partially observable environments: Scaling up
Littman, M. L., Cassandra, A. R., & Kaelbling, L. P. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50052-9 · Cited by 650
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L. (2005) · Cited by 588
Value-Function Approximations for Partially Observable Markov Decision Processes
Hauskrecht, M. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.678 · Cited by 527
Perseus: Randomized point-based value iteration for POMDPs
Spaan, M. T. J., & Vlassis, N. A. (2005) · UvA-DARE (University of Amsterdam) · Cited by 526
Self-citationDynamic programming for partially observable stochastic games
Hansen, E. A., Bernstein, D. S., & Zilberstein, S. (2004) · Cited by 422
Planning, Learning and Coordination in Multiagent Decision Processes
Boutilier, C. E. (1996) · Cited by 392
The Communicative Multiagent Team Decision Problem: Analyzing Teamwork Theories and Models
Pynadath, D. V., & Tambe, M. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1024 · Cited by 376
Taming Decentralized POMDPs: Towards Efficient Policy Computation for Multiagent Settings
Nair, R., Nair, R., Tambe, M., Tambe, M., Marsella, S., Yokoo, M., Pynadath, D. V., & Marsella, S. (2003) · Cited by 330
Kitano, H., Tadokoro, S., Noda, I., Matsubara, H., Takahashi, T., Shinjou, A., & Shimada, S. (2003) · IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028) · doi:10.1109/icsmc.1999.816643 · Cited by 306
Decentralized Control of Cooperative Systems: Categorization and Complexity Analysis
Goldman, C. V., & Zilberstein, S. (2004) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1427 · Cited by 232
Solving transition independent decentralized Markov decision processes
Becker, R., Shlomo Zilberstein, B., Lesser, V., & Goldman, C. V. (2004) · doi:10.5555/1622487.1622500 · Cited by 226
Optimizing information exchange in cooperative multi-agent systems
Goldman, C. V., & Zilberstein, S. (2003) · Proceedings of the second international joint conference on Autonomous agents and multiagent systems · doi:10.1145/860575.860598 · Cited by 212
Networked distributed POMDPs: a synthesis of distributed constraint optimization and POMDPs
Nair, R., Varakantham, P. R., Tambe, M., & Yokoo, M. (2005) · Cited by 206
The Optimal Control of Partially Observable Markov Decision Processes.
Sondik, E. J. (1971) · Medical Entomology and Zoology · Cited by 170
Fast algorithms for finding randomized strategies in game trees
Koller, D., Megiddo, N., & von Stengel, B. (1994) · Proceedings of the twenty-sixth annual ACM symposium on Theory of computing - STOC '94 · doi:10.1145/195058.195451 · Cited by 170
MAA*: A Heuristic Search Algorithm for Solving Decentralized POMDPsPreprint
Szer, D., Charpillet, F., & Zilberstein, S. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.1359 · Cited by 167
Approximate Solutions for Partially Observable Stochastic Games with Common Payoffs
Emery, R., Gordon, G., Schneider, J., & Thrun, S. (2004) · doi:10.1109/aamas.2004.67 · Cited by 167
Bounded policy iteration for decentralized POMDPs
Bernstein, D. S., Hansen, E. A., & Zilberstein, S. (2005) · Cited by 143
Improved Memory-Bounded Dynamic Programming for Decentralized POMDPsPreprint
Seuken, S., & Zilberstein, S. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.5295 · Cited by 96
An online POMDP algorithm for complex multiagent environments
Paquet, S., Tobin, L., & Chaib-draa, B. (2005) · Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems · doi:10.1145/1082473.1082620 · Cited by 92
Decentralized Markov Decision Processes with Event-Driven Interactions
Becker, R., Zilberstein, S., & Lesser, V. (2004) · doi:10.1109/aamas.2004.100 · Cited by 88
Reasoning about joint beliefs for execution-time communication decisions
Roth, M., Simmons, R. G., & Veloso, M. (2005) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/1082473.1082593 · Cited by 87
Exploiting locality of interaction in factored Dec-POMDPs
Oliehoek, F. A., Spaan, M. T. J., Whiteson, S., & Vlassis, N. A. (2008) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/xqlq5000 · Cited by 81
Self-citationApplications of Markov Decision Processes in Communication Networks
Altman, E. (2002) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-1-4615-0805-2_16 · Cited by 74
Interaction-driven Markov games for decentralized multiagent planning under uncertainty
Spaan, M. T. J., & Melo, F. S. (2008) · doi:10.5555/1402383.1402458 · Cited by 61
Self-citationANALYZING MYOPIC APPROACHES FOR MULTI‐AGENT COMMUNICATION
Becker, R., Carlin, A., Lesser, V., & Zilberstein, S. (2009) · Computational Intelligence · doi:10.1111/j.1467-8640.2008.01329.x · Cited by 59
Decentralized control of finite state Markov processes
Hsu, K., & Marcus, S. I. (1982) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1982.1102924 · Cited by 55
Decentralized planning under uncertainty for teams of communicating agents
Spaan, M. T. J., Gordon, G. J., & Vlassis, N. A. (2006) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/1160633.1160678 · Cited by 54
Self-citationDecentralized control of a multiple access broadcast channel: performance bounds
Ooi, J. M., & Wornell, G. W. (2002) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.1996.574318 · Cited by 48
Exploiting factored representations for decentralized execution in multiagent teams
Roth, M., Simmons, R. G., & Veloso, M. (2007) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/1329125.1329213 · Cited by 46
Learning to communicate in a decentralized environment
Goldman, C. V., Allen, M., & Zilberstein, S. (2006) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-006-0008-9 · Cited by 45
An Optimal Best-First Search Algorithm for Solving Infinite Horizon DEC-POMDPs
Szer, D., & Charpillet, F. (2005) · Lecture notes in computer science · doi:10.1007/11564096_38 · Cited by 41
Mixed Integer Linear Programming For Exact Finite-Horizon Planning In Decentralized PomdpsPreprint
Aras, R., Dutech, A., & Charpillet, F. (2007) · arXiv (Cornell University) · doi:10.48550/arxiv.0707.2506 · Cited by 30
An Iterative Algorithm for Solving Constrained Decentralized Markov Decision Processes
Beynier, A., & Mouaddib, A. (2006) · Cited by 28
Marecki, J., & Tambe, M. (2007) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/1329125.1329389 · Cited by 20
Complexity analysis and optimal algorithms for decentralized decision making
Bernstein, D. S., & Zilberstein, S. (2005) · Cited by 11
Heuristic anytime approaches to stochastic decision processes
Fernández, J. L., Sanz, R., Simmons, R. G., & Diéguez, A. R. (2006) · Journal of Heuristics · doi:10.1007/s10732-006-4834-3 · Cited by 9
Dec-POMDPs and extensive form games: equivalence of models and algorithms
Oliehoek, F. A., & Vlassis, N. A. (2006) · UvA-DARE (University of Amsterdam) · Cited by 7
Self-citationA hierarchical model for decentralized fighting of large scale urban fires
Oliehoek, F. A., & Visser, A. (2006) · UvA-DARE (University of Amsterdam) · Cited by 5
Self-citation
Cited by
Multi-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957
A survey and critique of multiagent deep reinforcement learning
Hernandez-Leal, P., Kartal, B., & Taylor, M. E. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09421-1 · Cited by 671
The StarCraft Multi-Agent Challenge
Samvelyan, M., Rashid, T., Schroeder de Witt, C., Farquhar, G., Nardelli, N., Rudner, T. G. J., Hung, C.-M., Torr, P. H. S., Foerster, J., & Whiteson, S. (2019) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/lvzz5205 · Cited by 453
Multi-agent reinforcement learning as a rehearsal for decentralized planning
Kraemer, L., & Banerjee, B. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.01.031 · Cited by 343
Partially Observable Markov Decision Processes
Spaan, M. T. J. (2012) · Adaptation, Learning, and Optimization · doi:10.1007/978-3-642-27645-3_12 · Cited by 315
Self-citation
Metadata
- Authors
- Oliehoek, F. A., Spaan, M. T. J., & Vlassis, N.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2008-05-28
- Volume
- 32
- Pages
- 289-353
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W2166533447
- Cited by
- 265
- References
- 43
- FWCI
- 4.77
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.