An information-theoretic approach to curiosity-driven reinforcement learning

“Information theoretic approaches inspired by some form of rate-distortion theory have been used widely in machine learning, for example for clustering and dimensionality reduction [26,21,6,20].”

“…a vast literature on applications of information gain measures for artificial curiosity, intrinsically-motivated exploration, and other more precise goals (Storck et al., 1995; Schmidhuber, 2005; Still & Precup, 2012; Hernández-Lobato et al., 2014; Mohamed & Rezende, 2015; Houthooft et al., 2016).”

“This resolution of uncertainty is related to artificial curiosity [54, 55] and speaks to the value of information [56].”

“[10] view the problem of exploration from an information-theoretic prospective and maximizes the information that the most recent state-action pair carries about the future.”

“Over the years, a variety of intrinsic motivation methods have been proposed, largely focusing on exploration, based on information gain [30, 64], prediction error [1, 45, 61], novelty search [39, 56], curiosity [50, 51], entropy [21, 38], or empowerment [36, 42].”

“Therefore, random behaviour does not necessarily equate to irrational behaviour – where there is a longer-term motivation, taking random actions may in fact allow an agent to arrive at the optimal strategy (Still & Precup 2012).”

“From the requisite predictive dynamics, which can be linked to an information bottleneck [21], agent-like features, such as action policies that balance exploration and exploitation [22], and curiosity driven reinforcement-learning [23] can be derived (c.”

“Besides these count-based methods, information theory is widely used to denote the uncertainty reduction of visiting a state, such as maximizing information gain [6], empowerment (or called mutual information) [16, 24], etc.”

“Still and Precup (2012) stressed the importance of choosing policies that not only maximize a task objective, but also allow for high predictive power, which should make the world both interesting and exploitable.”

“Thrun (1992) divided exploration methods in two: undirected, which are closely related to a random walk with the advantage of not using specific knowledge (Szepesvári 2010; Still and Precup 2012); and directed, which need domain specific knowledge with the advantage of generally better exploration.”

“This means that the most probable policies or paths are those that resolve uncertainty when navigating the lived world (Berlyne, 1950; Schmidhuber, 2006; Baranes and Oudeyer, 2009; Still and Precup, 2012; Barto et al., 2013; Moulin and Souchay, 2015).”

“6 leads to the optimal policy, π ∗ (Parush et al., 2011; Tishby and Polani, 2011; Still and Precup, 2012): The optimal policy is a softmax function, with an added frequency-dependent bias (persever-ation) term.”

“…to various reinforcementlearning approaches that draw direct inspiration from this line of thinking [Klyubin et al., 2005, Ortega and Braun, 2011, Still and Precup, 2012, Ortega and Braun, 2013, Shafieepoorfard et al., 2016, Tiomkin and Tishby, 2017, Lerch and Sims, 2018, 2019, Abel et al.,…”

“2b; Gottlieb et al., 2013; Nelson, 2005; Oudeyer & Kaplan, 2007; Still & Precup, 2012).”

“…decision-making under partial observability (Åström, 1965; Smallwood and Sondik, 1973; Kaelbling et al., 1998), with related work studying history compression through predictive state representations and information-theoretic sufficient statistics (Littman and Sutton, 2001; Still and Precup, 2012).”

“Curiosity-driven reinforcement learning method can be utilized to encourage exploration in sparse reward setting [33, 34].”

“…of visits to each state [9]. c) Directed exploration based on alternative learning models, where approaches are based, e.g., on concepts of learned system models [10], which allows trajectory planning, or built-in curiosity [11], which encourages an RL agent to explore what has not been seen.”

“Studies on the relationship between physiological arousal and hedonic states have also stated that reducing uncertainty “makes the world enjoyable” [26,27].”

“REPS is an information theoretic policy search approach which is based on the insight that the information loss between two policy steps should be bounded [19], [20], [8].”

“Previous methods to address the dual control problem include curiosity-driven reinforcement learning [1] and the use of the epsilon coefficient in Deep Q Networks [2].”

“First, previous studies have considered capacity-limited agents that trade reward to gain information [8–11], but did so in a way that abstracts away the underlying costs, and therefore cannot disambiguate the effects of limited storage capacity from other energetic or computational limitations.”

“Closely related measures of curiosity used as intrinsic reward signals are prediction uncertainty [11, 12] and improvement [13].”

“Different methods have been suggested to quantify information gain during exploration by authors like Mobin et al. [52], Still and Precup [53], and Storck et al. [54].”

“The common approach in RL to address this dilemma involves using ad hoc novelty-based intrinsic rewards, encouraging agents to explore more when encountering novel situations in their environment 88–91 .”

“, 2013), 3) the expected free energy effectively addresses the explore-exploit dilemma and confers the agent with artificial curiosity (Friston et al., 2017b; Schmidhuber, 2010; Schwartenbeck et al., 2019; Still and Precup, 2012), as opposed to the need to add ad-hoc information bonus terms (Tokic and Palm, 2011), and…”

“…et al., 2004; Klyubin et al., 2005; Todorov, 2007; Klyubin et al., 2008; Still, 2009; Polani, 2009; Ortega & Braun, 2011; van Dijk & Polani, 2011; Still & Precup, 2012; Tishby & Polani, 2011; Sun et al., 2011; Rubin et al., 2012; Ortega & Braun, 2013; Mohamed & Jimenez Rezende, 2015; Houthooft…”

“Many of previous exploratory attempts employ prediction error (Pathak et al. 2017), prediction uncertainty (Pathak, Gandhi, and Gupta 2019), or variants of uncertainty as self-driving motivations in the learning phase (Dean, Tul-siani, and Gupta 2020; Houthooft et al. 2016; Still and Precup 2012).”

“To express this goal in the form of a reward function, we leverage the concept of curiosity [17]–[19], which is an intrinsic reward that motivates the policy to explore unknown areas.”

“Still & Precup (2012) calculate the curiosity bonus based on the mutual information between the past and the future states within a time series.”

“Some references to such regularizes, also known as intrinsic motivation, include [5, 27, 40, 11, 20, 9, 10].”

“A. Supplementary Related Work (2012) shows that Boltzmann exploration naturally emerges when you minimize the information cost (mutual information) of a policy while maintaining a target reward level.”

“MI has been extensively studied in the RL literature for curiosity-driven exploration and unsupervised skill and option discovery (Still & Precup, 2012; Campos et al., 2020; Mohamed & Rezende, 2015; Gregor et al., 2016; Eysenbach et al., 2019).”

“Information theoretic approaches have also been explored in a reinforcement learning context but do not leverage the (beliefs about) latent states implied by the generative model; see (Still, 2012; Mohamed & Rezende, 2015).”

“One common answer to this question stems from game-theoretic approaches of language evolution that emphasize utility maximization (Chaabouni, Strub, et al., 2021; Steels & Belpaeme, 2005; Still & Precup, 2012).”

“Many efforts have been made to enhance sample efficiency, including improving the exploration ability of the agent (Still & Precup, 2012; Burda et al., 2018; Haarnoja et al., 2018; Ladosz et al., 2022; Yang et al., 2024; Jiang et al., 2025), scaling up compute by reusing data from the re-play…”

“To fulfill that, some researchers enhance the sample efficiency by improving the exploration ability of the agent (Asmuth et al., 2012; Still & Precup, 2012; Burda et al., 2018; Haarnoja et al., 2018; Ladosz et al., 2022; Yang et al., 2024; Jiang et al., 2025), reducing value estimation bias (Van…”

“The authors in [55] apply the idea of maximizing predictive power to the exploration problem in reinforcement learning.”

“Others consider “interesting” (Still & Precup, 2012; Schmidhuber, 1997) to mean a high dependency between present and future states and actions (i.e. “interesting” things are predictable (Still & Precup, 2012) or controllable (Mohamed & Jimenez Rezende, 2015)).”

“On the environmental side, one can move beyond simple low-dimensional sensory landscapes and consider more complex environments in which exploration-exploitation tradeoffs become salient [18, 37, 45, 54, 55].”

“The difference to traditional RL is that instead of rewards being offered from particular states of the environment, they are calculated in relation to the novelty of the learned information (Still & Precup, 2012).”

“…1, 5), uncertainty (Pathak et al., 2017, pp. 1-2), compression progress (Graziano et al., 2011, p. 44), competence (Oddi et al., 2020, pp. 2417-2418), and information gain (Bellemare et al., 2016, p. 4; Houthooft et al., 2016, pp. 2-3; Still and Precup, 2012, p. 139; Frank et al., 2014, pp. 5-6).”

“…quantify the knowledge of the agent of the environment to design the intrinsic reward model, provide a higher intrinsic reward to the observation with insufficient knowledge to encourage the agent to explore it, and obtain better navigation strategies after fully exploring the environment[11, 12].”

“…as -greedy, Thompson sampling (Osband & Van Roy, 2015; Osband et al., 2016; Azizzadenesheli et al., 2018), noisy networks (Fortunato et al., 2018), random network exploration bonuses (Burda et al., 2019) and curiosity-based approaches (Still & Precup, 2012; Frank et al., 2014; Zhelo et al., 2018).”

“…under different names: reward shaping (Ng et al., 1999), optimism in the face of uncertainty (Kearns & Singh, 2002), intrinsic motivation (Chentanez et al., 2005), curiosity-driven RL (Still & Precup, 2012), prediction gain (Bellemare et al., 2016), or entropy-regularized MDPs (Neu et al., 2017).”

“…(Schmidhuber, 1991; Chentanez et al., 2004), such as forecasting complexity (Grass-berger, 1986), statistical complexity (Crutchfield and Young, 1989), predictive information (Bialek et al., 2001; Still and Precup, 2012), and light cone complexity (Shalizi et al., 2004; Aaronson et al., 2014).”

“To promote more informed exploration, several approaches introduce intrinsic rewards based on prediction errors or state novelty [30], [15] or use information-theoretic measures to encourage information gain [31], [32].”

“In 2012,Precup borrowed the idea of information theory in reinforcement learning, and proposed cross-entropy as an internal motivation, and proposed a curiosity-driven reinforcement learning model [9].”

“[34] defined a curiosity‐based, weighted‐random exploration mechanism.”

“While the argument for this is too long to be made in full in the present paper, we will sketch it briefly here: an agent needs to acquire a certain amount of relevant Shannon information from the environment to carry out a given task at a given performance level [28, 25, 34, 33].”

“The objective function can be extended by the predictive power (Zahedi et al., 2010; Still & Precup, 2012) that measures to what extent an agent can influence the environment by its actions over time.”

“This is trivially true for approaches designed for tabular stationary problems (Chentanez et al., 2005; Still and Precup, 2012; Little and Sommer, 2013; Meuleau and Bourgine, 1999; Barto and Simsek, 2005; Szita and Lorincz, 2008; Lopes et al., 2012; Schossau et al., 2016).”

“There is another approach (Still and Precup 2012) also uses mutual information as an exploration reward.”

“themes, these novelty signals might be prediction error (Singh et al., 2004; Stadie et al., 2015), value error (Simsek and Barto, 2006), learning progress (Schmidhuber, 1991), or mutual information (Still and Precup, 2012; Mohamed and Rezende, 2015). The idea also finds roots in continual learning (Ring, 1997). In Thru…”