An information-theoretic approach to curiosity-driven reinforcement learning

“Information theoretic approaches inspired by some form of rate-distortion theory have been used widely in machine learning, for example for clustering and dimensionality reduction [26,21,6,20].”

“Many approaches measure information gain for exploration (Little & Sommer, 2014; Still & Precup, 2012; Storck et al., 1995).”

“This resolution of uncertainty is closely related to satisfying artificial curiosity (Schmidhuber, 1991; Still & Precup, 2012) and speaks to the value of information (Howard, 1966).”

“In particular, it addresses how resolving uncertainty “makes the world interesting and exploitable” (Still & Precup, 2012, p. 139).”

“This has motivated work on exploration via quantification of uncertainty (Still & Precup, 2012; Houthooft et al., 2016) or prediction improvement measures (Schmidhuber, 1991a; Oudeyer et al., 2007; Lopes et al., 2012; Achiam & Sastry, 2017).”

“The general framework for these types of approaches was presented in [54, 55].”

“Epistemic value: also known as intrinsic value [26,75,76] or the value of information [64,77].”

“Other techniques incentivize exploration of states and actions where prediction of a forward model is highly-uncertain (Houthooft et al., 2016; Still & Precup, 2012).”

“…problem that forms the basis of a large number of applications in machine learning, including rate distortion theory [4], information bottleneck methods [26], population coding [1], curiosity-driven exploration [24, 19], model selection [3], and intrinsically-motivated reinforcement learning [20].”

“In particular, such a mutual information can be cast as the information gain (Schmidhuber, 1991; 2010; Sun et al., 2011; Still and Precup, 2012; Houthooft et al., 2016; Russo and Van Roy, 2016; 2018) for estimating w h , which is induced by observing � ( s h , a h ) in addition to D .”

“Intrinsic Motivation: A family of approaches to intrinsic motivation reward an agent based on prediction error [2, 27, 36, 42], prediction uncertainty [11, 44], or improvement [19, 34] of a forward dynamics model of the environment that gets trained along with the agent’s policy.”

“This resolution of uncertainty is closely related to satisfying artificial curiosity (Schmidhuber, 1991; Still & Precup, 2012) and speaks to the value of information (Howard, 1966), particularly in the context of evincing information necessary to realize rewards or payoffs (see Meder & Nelson,…”

“This resolution of uncertainty is closely related to satisfying artificial curiosity (Schmidhuber, 1991; Still and Precup, 2012) and speaks to the value of information (Howard, 1966).”

“This resolution of uncertainty is closely related to satisfying artificial curiosity ( Schmidhuber, 1991 ; Still and Precup, 2012 ) and the ‘value of information’ ( Howard, 1966 ).”

“…various ways, for example by rewarding surprise (Schmidhuber, 1991b;a; 2010; 2006; Achiam & Sastry, 2017), information gain (Little & Som-mer, 2013; Still & Precup, 2012; Houthooft et al., 2016), curiosity (Pathak et al., 2017; Burda et al., 2019b), empowerment (Klyubin et al., 2005; Rezende &…”

“4), the Infomax principle (Linsker, 1990) and formulations of epistemic foraging in terms of information theory and artificial curiosity (Bialek et al., 2001; Itti and Baldi, 2009; Schmidhuber, 2010; Still and Precup, 2012; Sun et al., 2011; Tishby and Polani, 2010). In the present formulation, internal states come to…”

“Still and Precup (2012) derive a trade-off between exploration and exploitation in an attempt to maximize the predictive power of the agent.”

“Information-theoretic approaches have also been explored in a reinforcement learning context (Still & Precup, 2012; Mohamed & Rezende, 2015; Blau, Ott, & Ramos, 2019).”

“An exploration strategy that uses curiosity as an intrinsic reward effectively addresses the problem [27], [28].”

“This resolution of uncertainty is closely related to satisfying artificial curiosity (Schmidhuber, 1991; Still and Precup, 2012) and speaks to the value of information (Howard, 1966).”

“Please see [148,150] for a treatment of things like empowerment [170], information bottleneck [171] and predictive information [172,173].”

“In the current setting, we could replace information distance (Crooks, 2007; Kim, 2018) by information gain (Lindley, 1956; MacKay, 1992; Still & Precup, 2012).”

“This resolution of uncertainty is closely related to satisfying artificial curiosity (Schmidhuber, 1991; Still and Precup, 2012) and speaks to the value of information (Howard, 1966).”

“In this paper, we explore a different computational rationale for perseveration, based on the notion of policy complexity (McNamee et al., 2016; Parush et al., 2011; Still and Precup, 2012; Tishby and Polani, 2011).”

“Computationally, these ideas can be modeled using tools from information and probability theory (e.g. Tishby & Polani, 2011; Ortega & Braun, 2011; Still & Precup, 2012; Rubin et al., 2012; Ortega & Braun, 2013; Tiomkin & Tishby, 2017), for instance, via constraints on the channel capacity between…”

“Information gain additionally has a substantial history outside the RL framework, going back to (Lindley, 1956; Still & Precup, 2012; Sun et al., 2011).”

“Some research formulates intrinsic motivation as the self-prediction error of states and actions, where the generated curiosity or surprise motivates the agent to explore more efficiently even if the external rewards are very sparse [111]–[114].”

“However, methods that include curiosity 106 or uncertainty in the value function 107,108 could be used to guide more efficient exploration of a new maze configuration and consequently lead to faster learning.”

“Numerous attempts were made to guide the exploration in the beginning, motivated by curiosity [26, 27], intrinsic motivation [23, 4], exploration bonuses [30, 25] mutual information [24] and many of its approximations, for instance with variational autoencoders [23].”

“This means that the most probable policies or paths are those that resolve uncertainty when navigating the lived world (Berlyne, 1950; Schmidhuber, 2006; Baranes and Oudeyer, 2009; Still and Precup, 2012; Barto et al., 2013; Moulin and Souchay, 2015).”

“…both pragmatic (extrinsic) and epistemic (intrinsic) imperatives for good plans that resolve the exploration–exploitation dilemma (Cohen, McClure, & Yu, 2007; Friston et al., 2015; Schmidhuber, 2006; Still & Precup, 2012; Sun, Gomez, & Schmidhuber, 2011; Tschantz, Baltieri, Seth, & Buckley, 2020).”

“Some of the curiosity learning approaches reward an agent based on future prediction error or uncertainty [5, 31, 40].”

“In [63], the authors derived an information-theoretic measure to maximize the predictive power of the agent, while in [64], the authors derived an objective function to maximize the mutual information between actions and future states of the environment (i.”

“The application of rate-distortion theory to action selection has been developed theoretically by a number of different authors (Fox, Pakman, & Tishby, 2016; Grau-Moya, Leibfried, & Vrancx, 2018; Lerch & Sims, 2018; Parush, Tishby, & Bergman, 2011; Still & Precup, 2012; Tishby & Polani, 2011).”

“Mutual information is a critical statistical quantity and has been used in single-agent intrinsically motivated exploration (Rubin et al., 2012; Still & Precup, 2012; Salge et al., 2014; Mohamed & Rezende, 2015; Hyoungseok Kim, 2019) and multi-agent intention sharing and hiding (Strouse et al.,…”

“Section 2 makes use of material from [15,16], Section 3 uses material from [24], and Section 4 uses material form [14,17].”

“Prior work on exploration has used error [9, 10, 1, 11], uncertainty [12, 13, 3], and potential improvement [14] of a prediction model as intrinsic motivation.”

“Veness et al. (2015) study information theoretic methods for policy evaluation, while Still and Precup (2012) and Goyal et al. (2018) both leverage information theory for tackling the exploration problem.”

“Causal states, and the related information-theoretic notion of complexity, minimality, and sufficiency are used to derive task-agnostic policies with an intrinsic exploration-exploitation trade off in [34, 35].”

“…systems [see e.g. Mitter and Newton, 2005, Kappen, 2005, Todorov, 2006, 2009, Theodorou et al., 2010, Theodorou, 2015, Still, 2009, Still and Precup, 2012, van den Broek et al., 2010, Friston, 2010, Peters et al., 2010, Tishby and Polani, 2011, Kappen et al., 2012, Neumann and…”

“…thus emerges as an existential imperative (Barto et al., 2013; Lindley, 1956; MacKay, 1992; Schmidhuber, 2006; Schwartenbeck et al., 2019; Still and Precup, 2012) — to the extent that one could say that to be intelligent is (in part) to be curious, and to balance curiosity against…”

“Such constraints on the channel capacity between past states and future actions have been studied in a number of works (e.g. Tishby and Polani, 2011; Ortega and Braun, 2011; Still and Precup, 2012; Rubin et al., 2012; Ortega and Braun, 2013; Tiomkin and Tishby, 2017; Goyal et al., 2019).”

“Still and Precup (2012) take an information theoretic approach to exploration, that chooses a policy which maximizes the predictive power of the agent’s own behavior and environment rewards.”

“Some representative works include Oudeyer & Kaplan (2008; 2009); Sequeira et al. (2011); Still & Precup (2012); Bellemare et al. (2016); Martius et al. (2013); Schmidhuber (2008); Mohamed & Rezende (2015); Haber et al. (2018b;a) Some of the ideas here, in particular the notion of choosing actions…”

“The trade-off between exploration and exploitation is a long-standing challenge in the field of RL (Oudeyer et al., 2007; Schmidhuber, 1991a,b; Still and Precup, 2012).”

“Another closely related regularization for RL is using information bottleneck (Tishby & Polani, 2011; Still & Precup, 2012; Rubin et al., 2012; Ortega & Braun, 2013; Tiomkin & Tishby, 2017).”

“The MI objectives that we study have also been used as reward bonuses to improve exploration, without impacting the representation, in the form of empowerment [36, 35, 49, 39] and information-theoretic curiosity [62].”

“REPS is an information theoretic policy search approach which is based on the insight that the information loss between two policy steps should be bounded [19], [20], [8].”

“Alternatively, some approaches use information gain as an intrinsic reward (Still & Precup, 2012; Houthooft et al., 2016).”

“Our model uses similar information-theoretic principles as those used in previous approaches (Tishby and Polani, 2011; Still and Precup, 2012), yet in a different setting that enables sequential learning of convergent exploration primitives.”

“Empowerment is similar to curiosity concepts of predictive information that focus on the mutual information between the current and the subsequent state [6, 47, 68, 60, 42, 53].”

“This addresses the explore-exploit dilemma and confers the agent with artificial curiosity (Friston, Lin, et al., 2017; Schmidhuber, 2010; Schwartenbeck et al., 2019; Still & Precup, 2012), as opposed to the need to add ad hoc information bonus terms (Tokic & Palm, 2011).”

“…in robotic systems (Martius and Herrmann 2011), (ii) information-theoretic studies of the sensorimotor loop (Ay et al. 2011; Capdepuy et al. 2011; Still and Precup 2011), and (iii) self-organization in information processing networks (Boedecker et al. 2011; Greshenson 2011; Lizier et al. 2011).”

“Other measures based on Information Gain have been explored (Still and Precup, 2012; Little and Sommer, 2013; Achiam and Sastry, 2017; de Abril and Kanai, 2018; Berseth et al., 2019).”

“Still and Precup [6] have studied reinforcement learning by drawing on ideas from information theory, and information theoretic approaches have been studied by past researchers in stock markets and complex systems [7–11].”

“Alternating between maximization of the expected return at fixed information loss and subsequent information loss minimization at constant expected return (Still and Precup, 2012) yields algorithms that allow for automatic adaptation of .”

“This resolution of uncertainty is closely related to artificial curiosity (Schmidhuber, 1991; Still and Precup, 2012) and speaks to the value of information (Howard, 1966), particularly in the context of evincing information necessary to realise preferred outcomes.”

“Such methods are known as curiosity-driven RL [57], [58].”

“Therefore, methods to direct the learning agent to actions which yields as much new information as possible must be considered, like reverse curriculum learning [36] or curiosity-driven learning [37].”

“Apart from computational advantages, information-theoretic approaches provide a solid framework for describing and studying aspects of intelligent behavior [37], from autonomy [38] and curiosity [39] to bounded rationality [40] and game theory [41].”

“Within reinforcement learning, Bayesian notions of model uncertainty have a long history [Schmidhuber, 1991b,a, Dearden et al., 1999, Still and Precup, 2012, Sun et al., 2011, Maddison et al., 2017, Mihatsch and Neuneier, 2002].”

“Curiosity-driven reinforcement learning [20], [32], [33], for example, encourages agents to explore parts of the state space that yield the highest information gain, as opposed to the best reward.”

“…progress as with γ -progress (Kim et al., 2020); model uncertainty as with ensemble disagreement (Pathak et al., 2019; Sekar et al., 2020); novelty of latent state representations as with RIDE (Raileanu & Rockt ¨ aschel, 2020); and information gain (Still & Precup, 2012; Houthooft et al., 2016).”

“Information theoretic measures have also been used for exploration (Still & Precup, 2012; Mohamed & Rezende, 2015; Houthooft et al., 2016; Gregor et al., 2016).”

“This resolution of uncertainty is related to artificial curiosity (Schmidhuber, 1991; Still & Precup, 2012) and speaks to the value of information (Howard, 1966).”

“This resolution of uncertainty is closely related to satisfying artificial curiosity (Schmidhuber, 1991; Still & Precup, 2012) and speaks to the value of information (Howard, 1966).”

“Curiosity learning can also be seen through an information theoretic lens using an information gain objective like in Still and Precup (2012) or Houthooft et al. (2016).”

“Thus, an important direction for future research will be to develop algorithms for finding capacity-limited decision-rules based on versions of Blahut-Arimoto designed for general output distributions ( e.g. , particle filter-based algorithms [Dauwels, 2005]).”

“This central problem in artificial intelligence has evoked a number of solutions, mainly focusing on novelty-seeking behaviors (Schmidhuber, 1991; Lehman & Stanley, 2011; Still & Precup, 2012; Bellemare et al., 2016; Houthooft et al., 2016; Pathak et al., 2017).”

“A KL penalty similar to ours is used in [35], in settings with known reward and transition functions, to encourage “curiosity”.”

“Our approach is made possible by advances in contrastive representation learning for efficient estimation of the mutual information of sequence data [35].”

“” [ 56 – 58].”

“…reward re-labelling strategies like hindsight experience replay (HER) [5], competitive experience replay (CER) [72] and efficient exploratory techniques like intrinsic motivation [6, 8], curiosity [17, 33, 105] and surprise [3, 95, 125]-based exploration have been successfully demonstrated.”

“Information gain techniques define an auxiliary objective that encourages exploration of unexpected environment behavior or model disagreement and have been applied in discrete [37, 38, 39] and continuous actions spaces [40, 41, 42].”

“Other measures based on information gain have been explored (Still and Precup, 2012; Little and Sommer, 2013; Achiam and Sastry, 2017; de Abril and Kanai, 2018; Still and Precup, 2012), though they have been found to perform similarly to Bayesian Surprise (Little and Sommer, 2013).”

“• Intrinsic Motivation [1], [22], [23], [24], [25]: the intrinsic reward is proportional to how much the new state contains new information.”

“Popular methods focus on exploration based on information gain (Houthooft et al., 2016; Still and Precup, 2011), prediction error (Achiam and Sastry, Fig.”

“…solution to this problem for more complex, and especially nonstationary environments, which characterise many of the environments in which humans and other animals must manage this trade-off (Cohen et al., 2007; Jamieson, Malloy, Nowak, & Bubeck, 2014; Kolter & Ng, 2009; Still & Precup, 2012).”

“Exploration approaches include those that maximize model prediction error or improvement [59, 43, 65, 56], maximize model uncertainty [26, 66, 64, 57, 20], maximize state visitation [4, 19, 69, 25], maximize surprise [59, 1, 67], and employ other novelty-based exploration bonuses [42, 7, 32, 33].”