Prediction-Based Multi-Agent Reinforcement Learning in Inherently Non-Stationary Environments
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Multi-agent reinforcement learning (MARL) is a widely researched technique for decentralised control in complex large-scale autonomous systems. Such systems often operate in environments that are continuously evolving and where agents’ actions are non-deterministic, so called inherently non-stationary environments. When there are inconsistent results for agents acting on such an environment, learning and adapting is challenging. In this article, we propose P-MARL, an approach that integrates prediction and pattern change detection abilities into MARL and thus minimises the effect of non-stationarity in the environment. The environment is modelled as a time-series, with future estimates provided using prediction techniques. Learning is based on the predicted environment behaviour, with agents employing this knowledge to improve their performance in realtime. We illustrate P-MARL’s performance in a real-world smart grid scenario, where the environment is heavily influenced by non-stationary power demand patterns from residential consumers. We evaluate P-MARL in three different situations, where agents’ action decisions are independent, simultaneous, and sequential. Results show that all methods outperform traditional MARL, with sequential P-MARL achieving best results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Kohonen, T. (1990) · Proceedings of the IEEE · doi:10.1109/5.58325 · Cited by 8,414
Forecasting with artificial neural networks:
Zhang, G., Eddy Patuwo, B., & Y. Hu, M. (1998) · International Journal of Forecasting · doi:10.1016/s0169-2070(97)00044-7 · Cited by 4,229 · cited 2×
“…x taken at time i. Widely-used approaches for predicting future values of time-series (from the example, values xt+1, xt+2, ...) include statistical auto-regressive methods [Box and Jenkins 1970], Artificial Neural Networks (ANNs) [Zhang et al. 1998] and fuzzy logic [Brown and Harris 1994].”
MethodPlanning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · cited 1×
“Environments that are not fully observable should be addressed under the POMDP framework [Kaelbling et al. 1998].”
BackgroundA Comprehensive Survey of Multiagent Reinforcement Learning
Busoniu, L., Babuska, R., & De Schutter, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.913919 · Cited by 2,235 · cited 2×
“Centralised solutions to multi-agent optimisation in resource allocation problems are not feasible, as the time needed to solve a problem grows exponentially with the number of agents involved [Busoniu et al. 2008].”
BackgroundMultiagent systems: A survey from a machine learning perspective
Stone, P., & Veloso, M. (2000) · Autonomous Robots · doi:10.1023/a:1008942012299 · Cited by 1,218
Cited by
Reinforcement learning for demand response: A review of algorithms and modeling techniques
Vázquez-Canteli, J. R., & Nagy, Z. (2019) · Applied Energy · doi:10.1016/j.apenergy.2018.11.002 · Cited by 883
Incentive-based demand response for smart grid with reinforcement learning and deep neural network
Lu, R., & Hong, S. H. (2019) · Applied Energy · doi:10.1016/j.apenergy.2018.12.061 · Cited by 394
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224
Coordinated energy management for a cluster of buildings through deep reinforcement learning
Pinto, G., Piscitelli, M. S., Vázquez-Canteli, J. R., Nagy, Z., & Capozzoli, A. (2021) · Energy · doi:10.1016/j.energy.2021.120725 · Cited by 108
Marinescu, A., Taylor, A., Clarke, S., Serban, I., & Marinescu, C. (2019) · International Journal of Energy Research · doi:10.1002/er.4559 · Cited by 9
Self-citation
Metadata
- Authors
- Marinescu, A., Dusparic, I., & Clarke, S.
- Venue
- ACM Transactions on Autonomous and Adaptive Systems
- Published
- 2017-05-25
- Volume
- 12
- Issue
- 2
- Pages
- 1-23
- ISSN
- 1556-4665 (ISSN-L)
- 1556-4703
- OpenAlex
- W2552300487
- Cited by
- 69
- References
- 6
- FWCI
- 3.28
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Prediction-Based Multi-Agent Reinforcement Learning in Inherently Non-Stationary Environments. https://app.cochat.ai/papers/10.1145/3070861
- The paper
- Marinescu, A., Dusparic, I., & Clarke, S. (2017). Prediction-Based Multi-Agent Reinforcement Learning in Inherently Non-Stationary Environments. ACM Transactions on Autonomous and Adaptive Systems. https://doi.org/10.1145/3070861
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.