Policy Iteration for Decentralized Control of Markov Decision Processes
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Coordination of distributed agents is required for problems arising in many areas, including multi-robot systems, networking and e-commerce. As a formal framework for such problems, we use the decentralized partially observable Markov decision process (DEC-POMDP). Though much work has been done on optimal dynamic programming algorithms for the single-agent version of the problem, optimal algorithms for the multiagent case have been elusive. The main contribution of this paper is an optimal policy iteration algorithm for solving DEC-POMDPs. The algorithm uses stochastic finite-state controllers to represent policies. The solution can include a correlation device, which allows agents to correlate their actions without communicating. This approach alternates between expanding the controller and performing value-preserving transformations, which modify the controller without sacrificing value. We present two efficient value-preserving transformations: one can reduce the size of the controller and the other can improve its value while keeping the size fixed. Empirical results demonstrate the usefulness of value-preserving transformations in increasing value while keeping controller size to a minimum. To broaden the applicability of the approach, we also present a heuristic version of the policy iteration algorithm, which sacrifices convergence to optimality. This algorithm further reduces the size of the controllers at each step by assuming that probability distributions over the other agents' actions are known. While this assumption may not hold in general, it helps produce higher quality solutions in our test problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
Subjectivity and Correlation in Randomized Strategies
Aumann, R. J. (1974) · Journal of Mathematical Economics · doi:10.1016/0304-4068(74)90037-8 · Cited by 1,630 · cited 2×
“The importance of correlation has been recognized in the game theory community (Aumann, 1974), but there has been little work on algorithms for finding correlated policies.”
BackgroundThe Complexity of Decentralized Control of Markov Decision Processes
Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002) · Mathematics of Operations Research · doi:10.1287/moor.27.4.819.297 · Cited by 1,192
Self-citationOptimal control of Markov processes with incomplete state information
Åström, K. J. (1965) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(65)90154-x · Cited by 854 · cited 1×
“It was shown by Astrom (1965) that a belief state constitutes a sufficient statistic for the agent’s observation history, and it is possible to define an MDP over belief states as follows.”
Background
Cited by
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
Zhang, K., Yang, Z., & Başar, T. (2021) · Studies in Systems, Decision and Control · doi:10.1007/978-3-030-60990-0_12 · Cited by 1,264
Cooperative Heterogeneous Multi-Robot Systems: A Survey
Rizk, Y., Awad, M., & Tunstel, E. W. (2020) · ACM Computing Surveys · doi:10.1145/3303848 · Cited by 449
Decision Making in Multiagent Systems: A Survey
Rizk, Y., Awad, M., & Tunstel, E. W. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2018.2840971 · Cited by 219
“learning [48]–[50], policy search [51], and linear programming [52], [53] require models of the state transition and reward functions.”
BackgroundMemory-Limited Partially Observable Stochastic Control and Its Mean-Field Control Approach
Tottori, T., & Kobayashi, T. J. (2022) · Entropy · doi:10.3390/e24111599 · Cited by 11
“Actually, the finite-state controller, whose idea is similar with ML-POSC, plays a key role in extending POMDP from a single-agent system to a multiagent system [43], [44], [45], [46], [21], [47].”
BackgroundDecentralized Stochastic Control with Finite-Dimensional Memories: A Memory Limitation Approach
Tottori, T., & Kobayashi, T. J. (2023) · Entropy · doi:10.3390/e25050791 · Cited by 7
Metadata
- Authors
- Bernstein, D. S., Amato, C., Hansen, E. A., & Zilberstein, S.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2009-03-01
- Volume
- 34
- Pages
- 89-132
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W2103541323
- Cited by
- 88
- References
- 4
- FWCI
- 9.40
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.