Generalization and Regularization in DQN

“…loss of plasticity in a performant deep reinforcement learning system, Rainbow (Hessel et al., 2018), on a variation of the Arcade Learning Environment (ALE) (Bellemare et al., 2013; Machado et al., 2018) that involves non-stationarities from changing Atari 2600 games and game modes over time.”

“However, recent work [64], [65] suggests that regularization can increase generalization and sample-efficiency in function approximation with over-parameterized models.”

“We are interested in assessing the generalization of our algorithm, because deep RL techniques often fail to generalize to even slight modifications of the training environment [55], [56], [57].”

“Regularization is a common way to mitigate gradient explosion and is often used to address over-estimation (Farebrother et al., 2018; Chen et al., 2021; Liu et al., 2021; Hiraoka et al., 2022; Li et al., 2023).”

“Nonetheless, the interpretation is more complex when Φ ̸ = I , and algorithms to sovle (3) has been widely used in practice (Farebrother et al., 2018; Cobbe et al., 2019).”

“In either case, RL algorithms have shown to respond poorly to real-world deployments due to their failure to generalize over the training data [28], [29].”

“Other works have sought to stabilize the Q -value estimates by constraining parameter updates, e.g., through regularization (Farebrother et al., 2018), conjugate gradient methods (Schulman et al., 2015b), pre-conditioning the gradient updates (Knight & Lerner, 2018; Achiam et al., 2019), or using…”

“However, developing agents that can systematically generalize to out-of-distribution observations remains an open challenge [3, 9, 35].”

“Due to the realization that training and testing RL agents in the same environments can prevent the detection of overfitting, there have been a variety of generalization studies where RL agents are evaluated on different environments (mostly different levels and modes of a single game) than those on which they are tra…”

“Farebrother et al. [2018], Cobbe et al. [2019, 2020] demonstrated the beneficial effects of popular regularization methods from the supervised learning literature.”

“…for SL such as convolutional neural network, weight decay, batch normalization, dropout and self-supervised learning improve deep RL [16, 6, 22, 8, 30, 34, 13], a thorough study should be conducted before introducing a method from different learning frameworks, because we find the contrasting…”

“Generalisation can be improved via regularisation (Farebrother et al., 2018), data augmentation (Tobin et al., 2017; Raileanu et al., 2020; Ball et al., 2021), or dynamics context modelling (Seo et al., 2020; Lee et al., 2020).”

“For instance, NNs were shown to systematically misgeneralize on Atari environments (Farebrother et al., 2018; Delfosse et al., 2024a).”

“In this context, model generalization can be rephrased as the ability of an agent to effectively apply the learned knowledge and policies to novel environments or tasks unseen during training; see, e.g., [5]–[8].”

“PCG has enabled many findings on RL robustness and generalization, including the important, preliminary result that RL methods easily overfit to singleton environments [20, 39, 65].”

“For example, the introduction of different modes of emulated Atari 2600 games allowed for agents to be trained in one environment while being evaluated in slightly ones [28].”

“However, several studies [9], [10], [11], [12], [13], [14] have highlighted the limited generalization ability of RL agents to transfer learned knowledge to novel environments.”

“Building on this understanding, Farebrother et al. [20] also tackled the challenge of overfitting in DRL, focusing on DQN models applied to Atari 2600 games.”

“Each checkpoint in Figures 14, 15, 16, 17, 18, and 19 are evaluated by 10 episodes on unseen test environments.”

“Please note that we are the first to report Rainbow-IQN results on Atari variants, and hence we had only the DQN experiments of Farebrother et al. (2018) as a guide, but no results were reported for the standard 200 million steps training regime, and the two algorithms have many differences which…”

“…for SL such as convolutional neural network, weight decay, batch normalization, dropout and self-supervised learning improve deep RL [16, 6, 22, 8, 30, 34, 13], a thorough study should be conducted before introducing a method from different learning frameworks, because we find the contrasting…”

“Several papers have investigated generalization and transfer in RL [7, 10, 26, 27], demonstrating that learning on new environments results in drops in performance on previously learned environments [7], or re-initialization can help a plateaued agent make further progress [11].”

“…one of the most popular benchmarks for evaluating RL agents, the ALE has also evolved with new extensions, including stochastic transitions [Machado et al., 2018], various game modes and difficulties [Machado et al., 2018, Farebrother et al., 2018], and multi-player support [Terry and Black, 2020].”

“This dataset encompasses training logs from a DQN agent’s experiences across 60 Atari games (Bellemare et al., 2013; Machado et al., 2018), documented over five distinct runs.”

“When these models err it is impossible to know whether the models have not learned useful representations, or they have not learned good policies with respect to those representations.”

“…integration as a multi-armed bandit or the simplest RL problem (unlike deep RL models such as DQN which can suffer from robust overfitting [17], [18]) helps alleviate the issue of overfitting since here we learn to integrate the data indirectly through trial and error with the action and reward…”

“Regularization has been used to improve RL generalization [10,7,20,17,32].”

“We need specific techniques to avoid overfitting of RL algorithms [2].”

“Following Farebrother et al. (2018), we use the same variants investigated in this prior work: B REAKOUT , H ERO , and F REEWAY , which we visualize in Figure 8 (left).”

“In transfer learning for RL, they have studied this problem, mainly with the objective of transferring from one environment to another (Farebrother et al., 2018; Sasso et al., 2022; van Driessel & Francois-Lavet, 2021).”

“…proposals put forth by Bellemare et al. (2013), there have been a number of subsequent methodologies suggested by followup works (Agarwal et al., 2024; Machado et al., 2018; Patterson et al., 2024), as well as paper-specific evaluations, which can sometimes lead to inconsistencies in results.”

“In addition to the use of neural networks, DQN introduces two key innovations: experience replay and target network [47].”

“Generalization of deep Q-network DQN is proposed in [13], where regularization prevents overfitting to training data.”

“However, the downside of these methods is a potential destabilization of the training process, as the added variability can interfere with the agent’s ability to learn an effective policy [11].”

“Farebrother et al. [4] investigated the DQN method and concluded that it tends to overfit the trained task.”

“A lot of emphasis in deep reinforcement learning has lately been placed on an agent’s ability to learn policies that are robust and generalizable [7], [10], [25].”

“Objective function, namely, the loss function is defined by mean square error[10]: The partial derivative of the error function L(”

“The closest research to the one presented in this work is certainly that of Farebrother et al. [2018] and Tyo and Lipton [2020], who also studied the generalization properties of Deep-Q Networks in a model-free DRL context.”

“This is supported by evidence showing that DQN suffers from severe overfitting on training environments for Atari games (Farebrother et al., 2018b).”

“The key idea is to always measure generalization, just as it is in other parts of machine learning (Cobbe et al., 2019; Farebrother et al., 2018; Hastie et al., 2009; Zhang et al., 2018).”

“In this section, different RSs are explored with well-known reinforcement learning-based algorithms [62] to learn user preferences: MAB [63,64], SARSA [65], and Deep-Q Network (DQN) [66].”

“The former issue is a topic of interest within the transfer learning literature (Farebrother et al., 2020; Sabatelli & Geurts, 2021; Sasso et al., 2023; Zhu et al., 2023), while the latter is commonly referred to as plasticity loss (Nikishin et al., 2022; Lyle et al., 2023; Dohare et al., 2024).”

“Importantly, DQN can learn to generalize observed states and actions, making it adaptable to variations in the alignment problem [13].”

“According to [Liu et al. , 2019; Farebrother et al. , 2018], it is claimed that L2 regularization can produce better results than entropy regularization, and L2 regularization can find a good balance point for model’s ability and generalization.”

“discourage overfitting to any particular workload demand [79].”

“Settings of CR-DRQN’s Network Layers and Neurons According to the design of the existing work [40] and the references [28,41,42], the number The numbers of neural network layers and neurons are crucial factors for the network training [39]. of network layers is set to 4–8, considering the input…”

“poorly on novel situations not seen during training [19], [20], [21], [22].”

“…tricks (Ba et al., 2016; Gogianu et al., 2021; Bjorck et al., 2021; Bhatt et al., 2024; Lee et al., 2025a;b; Palenicek et al., 2025; Elsayed et al., 2024), dropout (Srivastava et al., 2014; Wu et al., 2021; Hiraoka et al., 2021), weight decay (Farebrother et al., 2018; Nauman et al., 2024), etc.”

“…and domain shifts remains a key challenge in RL. Deep RL agents trained on raw pixels often overfit to spurious correlations, leading to catastrophic failures under minor changes in color or object placement, particularly in Atari benchmarks (Farebrother et al., 2018; Delfosse et al., 2024a).”

“The choice of the L 2 regularization is motivated by its effectiveness in promoting generalization, and in preserving feature importance (Emmert‐Streib & Dehmer, 2019; Farebrother et al., 2018).”

“…to avoid the overfitting problem in deep reinforcement learning is to apply regularization techniques originally developed for supervised learning such as dropout [42, 11, 17], data augmentation [22, 48, 52, 34, 26], domain randomization [43, 50, 40, 27], or network randomization technique [24].”

“In RL, this manifests into agents learning training trajectories and being unable to generalize well, not only to unseen states but also across environments [16, 39].”

“[7] ● Learning from Scratch: The AI agent starts with minimal knowledge about the game and learns from scratch.”

“Running the CyGIL library for CyGIL-E and CyGIL-S, and the agent training algorithm, the DL laptop is connected to the emulated network using the VMware REST API over an Ethernet connection [8].”

“R ELATED W ORKS Recent work has suggested that lack of generalization is an endemic issue in deep reinforcement learning, with state-of-the-art algorithms tending to overfit to the environment, thus leading to models that simply memorize surface-level aspects of the environment [5], [8], [21].”

“[17] used DQN to learn policies in ALE and evaluated how they generalize to modified version of the games.”

“…singleton MDPs exhibit poor generalization, and that even minor changes to the environment can cause substantial degradation in agent performance (Justesen et al., 2018; Zhang et al., 2018a;b; Packer et al., 2018; Farebrother et al., 2018; Cobbe et al., 2019; Song et al., 2020; Kirk et al., 2021a).”

“To address the fragility of reward-driven agents, current mitigation strategies typically frame robustness as one of their goals (Pinto et al., 2017; Farebrother et al., 2018; Delfosse et al., 2025).”

“This phenomenon is particularly interesting to study because of its practical implications: DRL agents currently suffer from poor generalization [4], exhibit a loss of plasticity [5], and rarely transfer well between tasks [6].”

“Unfortunately, deep neural networks are prone to overfitting [3], which can lead to highly correlated state representations.”

“Although RL algorithms have achieved signifi-cant success in solving complex tasks, they tend to overfit the training environment (Farebrother et al., 2018; Cobbe et al., 2019), leading to performance deterioration in new environments and presenting significant deficiencies in generalization (Kirk et…”

“Despite the numerous advantages of this approach and the significant progress achieved, it is still very challenging for vision-based agents to generalize their abilities to unseen environments [12–15].”

“…policies for a wide variety of difficult tasks, many studies have highlighted the limited ability of RL agents to generalize to variations of such tasks that would appear easy to a human being (Farebrother et al., 2018; Packer et al., 2018; Zhang et al., 2018; Song et al., 2020; Cobbe et al., 2019).”

“Unless there is a generalisation goal, simply adding a dropout layer in an RL agent can potentially hurt the performance of the learning agent due to the introduction of extra variance that is already plentiful in RL [9].”

“A combination of the convolutional neural network (CNN) [6] and Q-learning called deep Q-networks (DQN) [7, 8] is proposed to handle large state-action space.”

“While some work has been done on studying and improving generalization by regular-izing features in RL (Farebrother, Machado, and Bowling 2018), very little work has been done on disentangling factors of variance in RL in an online setting.”

“For example, [24] studied the effect of L 2 norm on DQN on generalization, [25] studied generalization between simulations vs. the real world, [26] studied parameter variations and [27] studied the effect of different random seeds in environment generation.”

“While zero-padding can compensate for reduced observation dimensions [16], increasing dimensionality necessitates retraining the model to accommodate the new input structure, thereby increasing the cost of system scaling.”

“These approaches largely focus on enforcing robustness via novel regularisation and optimisation objectives (Farebrother et al., 2018; Cobbe et al., 2019; Igl et al., 2019), or by artificially expanding the diversity of training distributions through data mixing (Wang et al., 2020) and augmentation…”

“Several works focused on the euclidean-state-space generalization [27–29], Meta-learning [30], as well as task generalization [31,32].”

“Regularization and dropout 17,18 were also applied to neural models to reduce over fi tting to training environments.”

“A popular category of RL research with a higher degree of generalisation involves training agents on a subset of one or more closely-related environments, and evaluating them in the same set, or a different set of similar environments [164,77,39,19,32,38,62,104,26,25,149,94].”

“…have been developed, which all have in common that the agent learns through trials and errors by means of a reward system, and that what is finally learned is a policy , i.e., a function which takes a given state of the agent as an argument and returns an action (or a set of actions) to be taken.”

“This lack of emphasis contributes to some of the central challenges in reinforcement learning, including weak transferability between tasks and the brittleness of policies to small changes in environments or random seeds [32, 8, 14, 27].”

“This line of work evaluates generalization using sticky actions [29] or by using Atari games with different modes [13].”

“Another is to adopt techniques used in supervised learning to avoid overfitting, e.g., dropout, batch normalization, and specific convolutional architectures (Cobbe et al., 2019; Farebrother et al., 2018; Igl et al., 2019).”

“When a trained policy operates in different environmental dynamics than it was trained under, it often underperforms and achieves suboptimal rewards [Farebrother et al., 2018, Packer et al., 2018, Cobbe et al., 2018, Song et al., 2019, Raileanu and Fergus, 2021].”

“Motivated by this, regularization methods also used to imporve the generalization in Deep RL, which consist of using dropout and l 2 -regularisation [11] in Deep Q Network (DQN) [26], using batch normalization in CoinRun environment [7].”

“…the tremendous success of RL in a myriad of applications, the learned policies have difficulty generalizing to new test environments, even when these environments are slightly different from the training ones (Farebrother et al. (2018); Zhang et al. (2018); Cobbe et al. (2019); Song et al. (2020)).”

“Representative algorithms for value-based RL include Q-learning [7], SARSA [8], and the Deep Q-Network (DQN) algorithm combined with deep learning [9].”

“, 2017] which preserves the original loss function, and l2-regularization [Farebrother et al., 2018] to avoid overfitting on the training environment.”

“Other works have sought to stabilize the Q -value estimates by constraining parameter updates, e.g., through regularization (Farebrother et al., 2018), conjugate gradient methods (Schulman et al., 2015b), pre-conditioning the gradient updates (Knight & Lerner, 2018; Achiam et al., 2019), or using…”

“Futhermore, there has been a substantial effort to characterize transfer, generalization, and overfitting in deep reinforcement learning, primarily in terms of performance [10], [14], [44], [51]—motivating the need for new domains and evaluation methodologies.”