A Q-Learning Novelty Search Strategy for Evaluating Robustness of Deep Reinforcement Learning in Open-World Environments
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Despite substantial progress in deep reinforcement learning (DRL), a systematic characterization of DRL agents’ robustness to unexpected events in the environment is relatively understudied. Such unexpected events (“novelties”), especially those that are more structural than parametric, may significantly deteriorate the performance of DRL agents, leading them to be unfit for open-world environments and applications. However, not all novelties affect an agent’s performance equally. Unfortunately, even with reasonable and constrained definitions of the problem, the space of all novelties can be (at least) exponential. Hence, an effective search strategy is required to find novelties that can adversely affect the agent. This article presents a formalism for this problem and proposes a deep Q-learning-based novelty search strategy that efficiently and systematically finds candidate (potentially complex) novelties with significant negative impact on a DRL agent. We conduct a detailed set of experiments in a stochastic multiagent game environment (Monopoly) with complex decision-making properties.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021) · Machine Learning · doi:10.1007/s10994-021-05961-4 · Cited by 645
Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data
Hsu, Y.-C., Shen, Y., Jin, H., & Kira, Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01096 · Cited by 499
Efficient Dropout-Resilient Aggregation for Privacy-Preserving Machine Learning
Liu, Z., Guo, J., Lam, K.-Y., & Zhao, J. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3163592 · Cited by 137
Robust Reinforcement Learning: A Review of Foundations and Recent Advances
Moos, J., Hansel, K., Abdulsamad, H., Stark, S., Clever, D., & Peters, J. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010013 · Cited by 127
Continual Learning with Lifelong Vision Transformer
Wang, Z., Liu, L., Duan, Y., Kong, Y., & Tao, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00027 · Cited by 115
An empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2020) · arXiv · doi:10.48550/arxiv.2003.11881 · Cited by 78
Decision Making in Monopoly Using a Hybrid Deep Reinforcement Learning Approach
Bonjour, T., Haliem, M., Alsalem, A. O., Thomas, S., Li, H., Aggarwal, V., Kejriwal, M., & Bhargava, B. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3166555 · Cited by 47
Self-citationA Dataset Perspective on Offline Reinforcement LearningPreprint
Schweighofer, K., Radler, A., Dinu, M.-C., Hofmarcher, M., Patil, V., Bitto-Nemling, A., Eghbal-zadeh, H., & Hochreiter, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.04714 · Cited by 44
A multi-agent simulator for generating novelty in monopoly
Kejriwal, M., & Thomas, S. (2021) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2021.102364 · Cited by 38
Self-citation
Research works printed but not linked: 5
- 10.1609/aaai.v35i17.17766 · DOI
- Siddiqui, 2023, 17 fatalities, 736 crashes: The shocking toll of teslas autopilot
- Kejriwal, 2022, Designing artificial intelligence for open worlds
- Mirzaei, 2022, Fake it until you make it: Towards accurate near-distribution novelty detection, 11th Int. Conf. Learn. Representations
- Bailis, 2014, Learning to play monopoly: A reinforcement learning approach
Cited by
PokerOWL: A Multi-Agent Poker Environment for Benchmarking Open-World Learning
Chiu, M., Nananukul, N., & Kejriwal, M. (2026) · Applied Sciences · doi:10.3390/app16115458
Self-citationMachine Common Sense: A Grand Challenge in Artificial Intelligence
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789272
Self-citationKejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789276
Self-citationCommonsense Knowledge versus Reasoning
Kejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789275
Self-citationKejriwal, M. (2026) · ACM eBooks · doi:10.1145/3789270.3789277
Self-citation
Metadata
- Authors
- Islam, S., Chiu, M.-H., Bonjour, T., de Oliveira, R., Bhargava, B., & Kejriwal, M.
- Affiliations
- Purdue University West Lafayette (US)
- University of Southern California (US)
- Purdue University, West Lafayette, IN, USA
- Venue
- IEEE Intelligent Systems
- Published
- 2025-09
- Volume
- 40
- Issue
- 5
- Pages
- 5-15
- ISSN
- 1541-1672 (ISSN-L)
- 1941-1294
- OpenAlex
- W4402915306
- Cited by
- 13
- References
- 10
- Citations by year
- 2026: 13
- FWCI
- 1.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Intelligent Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- novelty search, deep reinforcement learning, robustness, open-world environments, Q-learning, structural novelty, monopoly
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.