The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Optimal and Approximate Q-value Functions for Decentralized POMDPs
Open accessAbstract
Decision-theoretic planning is a popular approach to sequential decision making problems, because it treats uncertainty in sensing and acting in a principled way. In single-agent frameworks like MDPs and POMDPs, planning can be carried out by resorting to Q-value functions: an optimal Q-value function Q* is computed in a recursive manner by dynamic programming, and then an optimal policy is extracted from Q*. In this paper we study whether similar Q-value functions can be defined for decentralized POMDP models (Dec-POMDPs), and how policies can be extracted from such value functions. We define two forms of the optimal Q-value function for Dec-POMDPs: one that gives a normative description as the Q-value function of an optimal pure joint policy and another one that is sequentially rational and thus gives a recipe for computation. This computation, however, is infeasible for all but the smallest problems. Therefore, we analyze various approximate Q-value functions that allow for efficient computation. We describe how they relate, and we prove that they all provide an upper bound to the optimal Q-value function Q*. Finally, unifying some previous approaches for solving Dec-POMDPs, we describe a family of algorithms for extracting policies from such Q-value functions, and perform an experimental evaluation on existing test problems, including a new firefighting benchmark problem.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
The Complexity of Decentralized Control of Markov Decision Processes
Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002) · Mathematics of Operations Research · doi:10.1287/moor.27.4.819.297 · Cited by 1,192
Decision-Theoretic Planning: Structural Assumptions and Computational Leverage
Boutilier, C., Dean, T., & Hanks, S. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.575 · Cited by 1,074
Point-based value iteration: an anytime algorithm for POMDPs
Pineau, J., Gordon, G., & Thrun, S. (2003) · Cited by 918
Learning policies for partially observable environments: Scaling up
Littman, M. L., Cassandra, A. R., & Kaelbling, L. P. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50052-9 · Cited by 650
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L. (2005) · Cited by 588
Value-Function Approximations for Partially Observable Markov Decision Processes
Hauskrecht, M. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.678 · Cited by 527
Perseus: Randomized point-based value iteration for POMDPs
Spaan, M. T. J., & Vlassis, N. A. (2005) · UvA-DARE (University of Amsterdam) · Cited by 526
Self-citationDynamic programming for partially observable stochastic games
Hansen, E. A., Bernstein, D. S., & Zilberstein, S. (2004) · Cited by 422
Planning, Learning and Coordination in Multiagent Decision Processes
Boutilier, C. E. (1996) · Cited by 392
Cited by
Multi-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957
A survey and critique of multiagent deep reinforcement learning
Hernandez-Leal, P., Kartal, B., & Taylor, M. E. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09421-1 · Cited by 671
The StarCraft Multi-Agent Challenge
Samvelyan, M., Rashid, T., Schroeder de Witt, C., Farquhar, G., Nardelli, N., Rudner, T. G. J., Hung, C.-M., Torr, P. H. S., Foerster, J., & Whiteson, S. (2019) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/lvzz5205 · Cited by 453
Multi-agent reinforcement learning as a rehearsal for decentralized planning
Kraemer, L., & Banerjee, B. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.01.031 · Cited by 343
Partially Observable Markov Decision Processes
Spaan, M. T. J. (2012) · Adaptation, Learning, and Optimization · doi:10.1007/978-3-642-27645-3_12 · Cited by 315
Self-citation
Metadata
- Authors
- Oliehoek, F. A., Spaan, M. T. J., & Vlassis, N.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2008-05-28
- Volume
- 32
- Pages
- 289-353
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W2166533447
- Cited by
- 265
- References
- 43
- FWCI
- 4.77
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Optimal and Approximate Q-value Functions for Decentralized POMDPs. https://app.cochat.ai/papers/10.1613/jair.2447
- The paper
- Oliehoek, F. A., Spaan, M. T. J., & Vlassis, N. (2008). Optimal and Approximate Q-value Functions for Decentralized POMDPs. Journal of Artificial Intelligence Research. https://doi.org/10.1613/jair.2447
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.