The DOI matches its DataCite record.
Asynchronous Credit Assignment for Multi-Agent Reinforcement Learning
Open accessAbstract
Credit assignment is a critical problem in multi-agent reinforcement learning (MARL), aiming to identify agents' marginal contributions for optimizing cooperative policies. Current credit assignment methods typically assume synchronous decision-making among agents. However, many real-world scenarios require agents to act asynchronously without waiting for others. This asynchrony introduces conditional dependencies between actions, which pose great challenges to current methods. To address this issue, we propose an asynchronous credit assignment framework, incorporating a Virtual Synchrony Proxy (VSP) mechanism and a Multiplicative Value Decomposition (MVD) algorithm. VSP enables physically asynchronous actions to be virtually synchronized during credit assignment. We theoretically prove that VSP preserves both task equilibrium and algorithm convergence. Furthermore, MVD leverages multiplicative interactions to effectively model dependencies among asynchronous actions, offering theoretical advantages in handling asynchronous tasks. Extensive experiments show that our framework consistently outperforms state-of-the-art MARL methods on challenging tasks while providing improved interpretability for asynchronous cooperation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 34
- Christopher Amato, George Konidaris, Leslie P Kaelbling, and Jonathan P How. Modeling and planning with macro-actions in decentralized pomdps. Journal of Artificial Intelligence Research, 64:817-859, 2019.
- Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: An evolutionary computation perspective. In Proceedings of the genetic and evolutionary computation conference companion, pages 314-315, 2019.
- Yuxin Chen, Hejun Wu, Yongheng Liang, and Guoming Lai. Varlenmarl: A framework of variable-length time-step multi-agent reinforcement learn- ing for cooperative charging in sensor networks. In 2021 18th Annual IEEE International Conference on Sensing, Communication, and Networking, pages 1-9. IEEE, 2021.
- Carlos F Daganzo and Yan- feng Ouyang. Public transportation systems: Principles of system design, operations planning and real-time con- trol. World Scientific, 2019.
- Shlomi Dolev, Chryssis Georgiou, Ioan- nis Marcoullis, and Elad M. Schiller. Practically-self- stabilizing virtual synchrony. Journal of Computer and System Sciences, 96:50-73, 2018.
- Foerster et al., 2018] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelli- gence, volume 32, 2018.
- David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on Learning Representations, 2017.
- Trevor J Hastie. Generalized additive mod- els. In Statistical models in S, pages 249-307. Routledge, 2017.
- Siddhant M. Jayakumar, Woj- ciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations, 2020.
- Jiahui Li, Kun Kuang, Baoxiang Wang, Fu- rui Liu, Long Chen, Changjie Fan, Fei Wu, and Jun Xiao. Deconfounded value decomposition for multi-agent rein- forcement learning. In International Conference on Ma- chine Learning, pages 12843-12856. PMLR, 2022.
- Xinran Li, Zifan Liu, Shibo Chen, and Jun Zhang. Individual contributions as intrinsic exploration scaffolds for multi-agent reinforcement learning. In Inter- national Conference on Machine Learning, 2024.
- Yongheng Liang, Hejun Wu, and Haitao Wang. ASM-PPO: Asynchronous and scalable multi-agent ppo for cooperative charging. In Proceedings of the 21st International Conference on Autonomous Agents and Mul- tiagent Systems, pages 798-806, 2022.
- Yongheng Liang, Hejun Wu, and Haitao Wang. Asynchronous multi-agent reinforcement learning for collaborative partial charging in wireless rechargeable sensor networks. IEEE Transactions on Mobile Comput- ing, 2023.
- Liu et al., 2023] Zichuan Liu, Yuanyang Zhu, and Chunlin Chen. NA 2 Q: Neural attention additive model for inter- pretable multi-agent q-learning. In International Confer- ence on Machine Learning, pages 22539-22558. PMLR, 2023.
- Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q- value functions for decentralized pomdps. Journal of Ar- tificial Intelligence Research, 32:289-353, 2008.
- Frans A Oliehoek, Christopher Am- ato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
- Afshin Oroojlooy and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. Applied Intelligence, 53(11):13677-13722, 2023.
- Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Fo- erster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
- David E. Rumelhart and James L. McClelland. A General Framework for Parallel Distributed Processing, pages 45-76. 1987.
- Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 2186-2188, 2019.
- Son et al., 2019] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learn- ing to factorize with transformation for cooperative multi- agent reinforcement learning. In International conference on machine learning, pages 5887-5896. PMLR, 2019.
- Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep net- works. In International conference on machine learning, pages 3319-3328. PMLR, 2017.
- Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zam- baldi, Max Jaderberg, Marc Lanctot, Nicolas Son- nerat, Joel Z. Leibo, Karl Tuyls, and Thore Grae- pel. Value-decomposition networks for cooperative multi- agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, page 2085-2087, 2018.
- Theocharous and Kaelbling, 2003] Georgios Theocharous and Leslie Kaelbling. Approximate planning in pomdps with macro-actions. Advances in neural information pro- cessing systems, 16, 2003.
- Wang and Sun, 2021] Jiawei Wang and Lijun Sun. Re- ducing bus bunching with asynchronous multi-agent rein- forcement learning. In Proceedings of the Thirtieth Inter- national Joint Conference on Artificial Intelligence,, pages 426-433, 2021.
- Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Duel- ing network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003. PMLR, 2016.
- Wang et al., 2020a] Jianhong Wang, Yuan Zhang, Tae- Kyun Kim, and Yunjie Gu. Shapley q-value: A local re- ward approach to solve global reward games. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 7285-7292, 2020.
- Wang et al., 2021] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex duel- ing multi-agent q-learning. In International Conference on Learning Representations, 2021.
- Jianhong Wang, Yuan Zhang, Yunjie Gu, and Tae-Kyun Kim. Shaq: Incorporating shapley value theory into multi-agent q-learning. Advances in Neural Information Processing Systems, 35:5941-5954, 2022.
- Yuchen Xiao, Weihao Tan, and Christo- pher Amato. Asynchronous actor-critic for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35:4385-4400, 2022.
- Yang et al., 2020a] Yaodong Yang, Jianye Hao, Guangy- ong Chen, Hongyao Tang, Yingfeng Chen, Yujing Hu, Changjie Fan, and Zhongyu Wei. Q-value path decompo- sition for deep multiagent reinforcement learning. In Inter- national Conference on Machine Learning, pages 10706- 10715. PMLR, 2020.
- Chao Yu, Akash Velu, Eugene Vinitsky, Ji- axuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Sys- tems, 35:24611-24624, 2022.
- Meng Zhou, Ziyu Liu, Pengwei Sui, Yix- uan Li, and Yuk Ying Chung. Learning implicit credit as- signment for cooperative multi-agent reinforcement learn- ing. Advances in neural information processing systems, 33:11853-11864, 2020.
- Xiaobo Zhou, Zhihui Ke, and Tie Qiu. Recommendation-driven multi-cell cooperative caching: A multi-agent reinforcement learning approach. IEEE Transactions on Mobile Computing, 2023.
Cited by
Sorted by citations
HiveMind: Contribution-Guided Online Prompt Optimization of LLM Multi-Agent Systems
Xia, Y., Wang, T., Zhang, S., Weng, Z., Cao, B., & Liew, S. C. (2025) · arXiv · doi:10.48550/arxiv.2512.06432
Metadata
- Authors
- Liang, Y., Wu, H., Wang, H., & Cai, H.
- Affiliations
- Sun Yat-sen University (CN)
- Shantou University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 170-178
- arXiv
- 2408.03692
- OpenAlex
- W4414360686
- Cited by
- 6
- References
- 0
- Citations by year
- 2026: 5 · 2025: 1
- FWCI
- 5.69
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62272497)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Keywords
- credit assignment, multi-agent reinforcement learning, cooperative policy, action dependencies, convergence analysis, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
