Dynamic Resource Allocation in Systems-of-Systems Using a Heuristic-Based Interpretable Deep Reinforcement Learning
The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Abstract
Systems-of-systems (SoS) often include multiple agents that interact in both cooperative and competitive modes. Moreover, they involve multiple resources, including energy, information, and bandwidth. If these resources are limited, agents need to decide how to share resources cooperatively to reach the system-level goal, while performing the tasks assigned to them autonomously. This paper takes a step toward addressing these challenges by proposing a dynamic two-tier learning framework, based on deep reinforcement learning that enables dynamic resource allocation while acknowledging the autonomy of systems constituents. The two-tier learning framework that decouples the learning process of the SoS constituents from that of the resource manager ensures that the autonomy and learning of the SoS constituents are not compromised as a result of interventions executed by the resource manager. We apply the proposed two-tier learning framework on a customized OpenAI Gym environment and compare the results of the proposed framework to baseline methods of resource allocation to show the superior performance of the two-tier learning scheme across a different set of SoS key parameters. We then use the results of this experiment and apply our heuristic inference method to interpret the decisions of the resource manager for a range of environment and agent parameters.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The RL manager serves as the governing entity within our framework and is trained using deep reinforcement learning, essentially using a two-layer governance scheme similar to [22,78].”
From Adaptive Information Modulation: Designing Governance Mechanisms for Multi-Agent Artificial Intelligence Systems (2026) · cites this paper as [22]
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Exploration and Exploitation in Organizational Learning
March, J. G. (1991) · Organization Science · doi:10.1287/orsc.2.1.71 · Cited by 21,942
Cognitive radio: making software radios more personal
Mitola, J., & Maguire, G. Q. (1999) · IEEE Personal Communications · doi:10.1109/98.788210 · Cited by 9,135
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Five Rules for the Evolution of Cooperation
Nowak, M. A. (2006) · Science · doi:10.1126/science.1133755 · Cited by 5,894
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
(1999) · Cited by 5,147
A survey of decision tree classifier methodology
Safavian, S. R., & Landgrebe, D. (1991) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.97458 · Cited by 3,808
The Design of CMOS Radio-Frequency Integrated Circuits
Lee, T. H. (2003) · Cambridge University Press eBooks · doi:10.1017/cbo9780511817281 · Cited by 3,497
Markov games as a framework for multi-agent reinforcement learning
Littman, M. L. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50027-1 · Cited by 2,614
A Strategic Model of Social and Economic Networks
Jackson, M. O., & Wolinsky, A. (1996) · Journal of Economic Theory · doi:10.1006/jeth.1996.0108 · Cited by 2,588
Shapley, L. S. (1953) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.39.10.1095 · Cited by 2,382
XAI—Explainable artificial intelligence
Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., & Yang, G.-Z. (2019) · Science Robotics · doi:10.1126/scirobotics.aay7120 · Cited by 2,053
Counterfactual Multi-Agent Policy Gradients
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11794 · Cited by 1,873
Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents
Tan, M. (1993) · Machine Learning Proceedings 1993 · doi:10.1016/b978-1-55860-307-3.50049-6 · Cited by 1,810
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning
(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6 · Cited by 1,377
An Introduction to Multi-Agent Systems
Balaji, P. G., & Srinivasan, D. (2010) · Studies in Computational Intelligence · doi:10.1007/978-3-642-14435-6_1 · Cited by 1,376
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
Zhang, K., Yang, Z., & Başar, T. (2021) · Studies in Systems, Decision and Control · doi:10.1007/978-3-030-60990-0_12 · Cited by 1,264
Towards a System of Systems Concepts
Ackoff, R. L. (1971) · Management Science · doi:10.1287/mnsc.17.11.661 · Cited by 1,149
Multiagent cooperation and competition with deep reinforcement learning
Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., & Vicente, R. (2017) · PLOS ONE · doi:10.1371/journal.pone.0172395 · Cited by 877
Cooperative Multi-agent Control Using Deep Reinforcement Learning
Gupta, J. K., Egorov, M., & Kochenderfer, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-71682-4_5 · Cited by 874
Multi-Agent Reinforcement Learning-Based Resource Allocation for UAV Networks
Cui, J., Liu, Y., & Nallanathan, A. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2935201 · Cited by 593
System of Systems - the meaning of of
Boardman, J., & Sauser, B. (2006) · 2006 IEEE/SMC International Conference on System of Systems Engineering · doi:10.1109/sysose.2006.1652284 · Cited by 506
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Yu, T., Quillen, D., He, Z., Julian, R., Narayan, A., Shively, H., Bellathur, A., Hausman, K., Finn, C., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1910.10897 · Cited by 450
Emergence of grounded compositional language in multi-agent populations
Mordatch, I., & Abbeel, P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11492 · Cited by 440
A crash course in implementation theory
Jackson, M. O. (2001) · Social Choice and Welfare · doi:10.1007/s003550100152 · Cited by 372
Explainability in deep reinforcement learning
Heuillet, A., Couthouis, F., & Díaz-Rodríguez, N. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106685 · Cited by 319
Roughgarden, T. (2010) · Communications of the ACM · doi:10.1145/1785414.1785439 · Cited by 288
Mean Field Multi-Agent Reinforcement Learning
Yang, Y., Luo, R., Li, M., Zhou, M. Q., Zhang, W., & Wang, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05438 · Cited by 286
Dahmann, J. S., & Baldwin, K. J. (2008) · 2008 2nd Annual IEEE Systems Conference · doi:10.1109/systems.2008.4518994 · Cited by 279
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262
Heuristics as adaptive decision strategies in management: HEURISTICS IN MANAGEMENT
Artinger, F., Petersen, M., Gigerenzer, G., & Weibler, J. (2015) · Journal of Organizational Behavior · doi:10.1002/job.1950 · Cited by 255
An Introductory Survey on Attention Mechanisms in NLP Problems
Hu, D. (2020) · Advances in Intelligent Systems and Computing · doi:10.1007/978-3-030-29513-4_31 · Cited by 247
Intelligent Resource Scheduling for 5G Radio Access Network Slicing
Yan, M., Feng, G., Zhou, J., Sun, Y., & Liang, Y.-C. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2922668 · Cited by 225
Kopetz, H. (2011) · Real-Time Systems Series · doi:10.1007/978-1-4419-8237-7_13 · Cited by 204
Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
Naderializadeh, N., Sydir, J. J., Simsek, M., & Nikopour, H. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3051163 · Cited by 187
Deng, S., Xiang, Z., Zhao, P., Taheri, J., Gao, H., Yin, J., & Zomaya, A. Y. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2974875 · Cited by 167
System of systems engineering - New challenges for the 21st century
Jamshidi, M. (2008) · IEEE Aerospace and Electronic Systems Magazine · doi:10.1109/maes.2008.4523909 · Cited by 163
Ferreira, P. V. R., Paffenroth, R., Wyglinski, A. M., Hackett, T. M., Bilen, S. G., Reinhart, R. C., & Mortensen, D. J. (2018) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2018.2832820 · Cited by 136
Network offloading policies for cloud robotics: a learning-based approach
Chinchali, S., Sharma, A., Harrison, J., Elhafsi, A., Kang, D., Pergament, E., Cidon, E., Katti, S., & Pavone, M. (2021) · Autonomous Robots · doi:10.1007/s10514-021-09987-4 · Cited by 117
Brown, O., Eremenko, P., & Collopy, P. (2009) · AIAA SPACE 2009 Conference & Exposition · doi:10.2514/6.2009-6540 · Cited by 111
AL-Tam, F., Correia, N., & Rodríguez, J. (2020) · IEEE Access · doi:10.1109/access.2020.3000893 · Cited by 107
The Logic of Heuristic Decision Making
Simon, H. A. (1977) · Boston studies in the philosophy of science · doi:10.1007/978-94-010-9521-1_10 · Cited by 104
SDRL: Interpretable and Data-Efficient Deep Reinforcement Learning Leveraging Symbolic Planning
Lyu, D., Yang, F., Liu, B., & Gustafson, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012970 · Cited by 96
Disciplinary Convergence in Systems Engineering Research
(2017) · Springer eBooks · doi:10.1007/978-3-319-62217-0 · Cited by 71
Learning Resource Allocation and Pricing for Cloud Profit Maximization
Du, B., Wu, C., & Huang, Z. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017570 · Cited by 63
Explainable Reinforcement Learning through a Causal Lens
Madumal, P., Miller, T., Sonenberg, L., & Vetere, F. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5631 · Cited by 58
Model-Based or Model-Free, a Review of Approaches in Reinforcement Learning
Huang, Q. (2020) · 2020 International Conference on Computing and Data Science (CDS) · doi:10.1109/cds49703.2020.00051 · Cited by 52
A Reinforcement Learning-Based Resource Allocation Scheme for Cloud Robotics
Liu, H., Liu, S., & Zheng, K. (2018) · IEEE Access · doi:10.1109/access.2018.2814606 · Cited by 50
Manufuture and Sustainable Manufacturing
Westkämper, E. (2008) · Springer eBooks · doi:10.1007/978-1-84800-267-8_3 · Cited by 44
Fair Topologies: Community Structures and Network Hubs Drive Emergence of Fairness Norms
Mosleh, M., & Heydari, B. (2017) · Scientific Reports · doi:10.1038/s41598-017-01876-0 · Cited by 36
Self-citationLeveraging Task Modularity in Reinforcement Learning for Adaptable Industry 4.0 Automation
Chen, Q., Heydari, B., & Moghaddam, M. (2021) · Journal of Mechanical Design · doi:10.1115/1.4049531 · Cited by 36
Self-citationGuiding the behavior of sociotechnical systems: The role of agent‐based modeling
Heydari, B., & Pennock, M. J. (2018) · Systems Engineering · doi:10.1002/sys.21435 · Cited by 35
Self-citationTowards Better Interpretability in Deep Q-Networks
Annasamy, R. M., & Sycara, K. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014561 · Cited by 33
Catalysts of Cooperation in System of Systems: The Role of Diversity and Network Structure
Gianetto, D. A., & Heydari, B. (2013) · IEEE Systems Journal · doi:10.1109/jsyst.2013.2284959 · Cited by 32
Self-citationOptimal Modularity for Fractionated Spacecraft: The Case of System F6
Mosleh, M., Dalili, K., & Heydari, B. (2014) · Procedia Computer Science · doi:10.1016/j.procs.2014.03.021 · Cited by 28
Self-citationFrom Modular to Distributed Open Architectures: A Unified Decision Framework
Heydari, B., Mosleh, M., & Dalili, K. (2016) · Systems Engineering · doi:10.1002/sys.21348 · Cited by 27
Self-citationRahman, M. H., Yuan, S., Xie, C., & Sha, Z. (2020) · Design Science · doi:10.1017/dsj.2020.12 · Cited by 26
Managing Sociotechnical Complexity in Engineering Design Projects
Hassannezhad, M., Cantamessa, M., Montagna, F., & Clarkson, P. J. (2019) · Journal of Mechanical Design · doi:10.1115/1.4042614 · Cited by 25
System of Systems Simulation driven Urban Air Mobility Vehicle Design
Prakasha, P. S., Ratei, P., Naeem, N., Nagel, B., & Bertram, O. (2021) · AIAA AVIATION Forum · doi:10.2514/6.2021-3200 · Cited by 25
Multi-agent Based Architecture for Acknowledged System of Systems
Agarwal, S., Pape, L. E., Kilicay-Ergin, N. H., et al. (2014) · Procedia Computer Science · doi:10.1016/j.procs.2014.03.001 · Cited by 22
Caputo, C., & Cardin, M. (2021) · Journal of Mechanical Design · doi:10.1115/1.4052299 · Cited by 22
Mission Engineering and Design using Real-Time Strategy Games: An Explainable-AI Approach
Dachowicz, A. P., Mall, K., Balasubramani, P., Maheshwari, A., Panchal, J. H., DeLaurentis, D., & Raz, A. K. (2021) · Journal of Mechanical Design · doi:10.1115/1.4052841 · Cited by 18
Modeling, Control & Fault Management of Microgrids
Moradian, M., Tabatabaei, F. M., & Moradian, S. (2013) · Smart Grid and Renewable Energy · doi:10.4236/sgre.2013.41013 · Cited by 17
Resource allocation through network architecture in systems of systems: A complex networks framework
Mosleh, M., Ludlow, P., & Heydari, B. (2016) · IEEE International Systems Conference (SysCon) · doi:10.1109/syscon.2016.7490629 · Cited by 16
Self-citationMeluso, J., & Austin‐Breneman, J. (2018) · Journal of Mechanical Design · doi:10.1115/1.4039494 · Cited by 15
Raz, A. K., Blasch, E., Cruise, R., et al. (2019) · AIAA SciTech Forum · doi:10.2514/6.2019-0381 · Cited by 15
INTRODUCTION TO SYSTEMS OF SYSTEMS ENGINEERING
Dahmann, J. S., & Henshaw, M. (2016) · Insight · doi:10.1002/inst.12100 · Cited by 14
Interpretable Approximation of a Deep Reinforcement Learning Agent as a Set of If-Then Rules
Nageshrao, S. P., Costa, B. S. J., & Filev, D. (2019) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2019.00041 · Cited by 12
Exploration-Exploitation in Multi-Agent Competition: Convergence with Bounded Rationality
Leonardos, S., Piliouras, G., & Spendlove, K. (2021) · arXiv · doi:10.48550/arxiv.2106.12928 · Cited by 12
A Game Theoretic Approach for Multi-hop Power Line Communications
Saad, W., Han, Z., & Poor, H. V. (2012) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-642-30373-9_38 · Cited by 11
Xiao, Y., & Sha, Z. (2022) · Design Science · doi:10.1017/dsj.2021.27 · Cited by 8
Evaluating Heuristics in Engineering Design: A Reinforcement Learning Approach
ElSayed, K. A., Bilionis, I., & Panchal, J. H. (2021) · doi:10.1115/detc2021-70425 · Cited by 2
Research works printed but not linked: 4
- Maier, 1998, Architecting Principles for Systems-of-Systems, Syst. Eng.: J. Int. Council Syst. Eng., 1, 267, 10.1002/(SICI)1520-6858(1998)1:4<267::AID-SYS3>3.0.CO;2-D · DOI · Paper page
- Nishimura, 2010, Principles of Magnetic Resonance Imaging
- Sukhbaatar, 2016, Learning Multiagent Communication With Backpropagation, 10.48550/arXiv.1605.07736 · arXiv · doi:10.48550/arxiv.1605.07736 · Paper page
- Wooldridge, 2009, An Introduction to Multiagent Systems
Cited by
The Sos Conductor: Orchestrating Resources With Iterative Agent-Based Reinforcement Learning
Chen, Q., & Heydari, B. (2024) · Systems Engineering · doi:10.1002/sys.21747 · Cited by 5
“This approach not only enhances the basic flat autonomy model from our previous work 5 but also introduces a new dimension of dynamism and flexibility.”
BackgroundInfluentialSelf-citationGenerative Network Design for Multi-Agent Coordination: A VAE-Assisted Deep RL Approach
Chen, Q., & Heydari, B. (2025) · Journal of Mechanical Design · doi:10.1115/1.4068796 · Cited by 4
Self-citationChen, Q., Ilami, S., Lore, N., & Heydari, B. (2026) · Journal of Mechanical Design · doi:10.1115/1.4070755 · Cited by 3
Self-citation
Metadata
- Authors
- Chen, Q., & Heydari, B.
- Affiliations
- Northeastern University (US)
- Venue
- Journal of Mechanical Design
- Publisher
- ASME International
- Published
- 2022-08-08
- Volume
- 144
- Issue
- 9
- ISSN
- 0161-8458 (ISSN-L)
- 1050-0472
- 1528-9001
- OpenAlex
- W4286461697
- Cited by
- 12
- References
- 76
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 3
- FWCI
- 0.66
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Mechanical Design (publishedVersion)
- Funders
- Division of Civil, Mechanical and Manufacturing Innovation (1554560)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Systems Engineering Methodologies and Applications
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- systems of systems, dynamic resource allocation, deep reinforcement learning, explainable decision-making, heuristic inference, autonomous agents, resource manager, OpenAI Gym environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.