Novelty or Surprise?

“Theories of motivation, particularly of intrinsic motivation (Deci and Ryan, 1985; Baldassarre and Mirolli, 2013), place novelty and surprise among the primary factors that arouse interest and motivate exploratory or avoidance behavior.”

“…basis of a model of classical conditioning that elaborates the Rescorla-Wagner model (Sutton and Barto, 1990) as well as the reward-prediction-error hypothesis about the phasic activity of dopamine producing neurons in the brain (Barto, 1995; Houk et al., 1995; Schultz et al., 1997; Schultz, 1998).”

“Tribus’ notion of surprisal plays a prominent role in the global brain theory of K. Friston and colleagues which is based on the principle of “free-energy minimization” (Friston et al., 2006; Friston, 2009, 2010).”

“(Berlyne, 1960, p. 19) He gives the example of seeing a man taller than any seen before: it is still possible to place the experience on a familiar scale, or more generally, to locate the experience in the appropriate multidimensional feature space.”

“Another area in which prediction errors appear in a Bayesian framework is in the “predictive coding” architectures of Rao and Ballard (1999) and Friston and Kiebel (2009).”

“Error correction is also the basis of Temporal Difference (TD) learning (Sutton, 1988), where the error incorporates information about the long-term expectation of reward and not just the immediate reward.”

“…hypothesis of the phasic activity of dopamine neurons, which has received a large amount of empirical support and represents one of the most fruitful integrations between computational and empirical research (Ungless, 2004; Wise, 2004; Schultz, 2007; Graybiel, 2008; Glimcher, 2011).”

“This Bayesian account of increases in the rate of animal learning observed in certain experiments (Rescorla, 1971) accomplishes what the Pearce-Hall model (Pearce and Hall, 1980) does via its use of an explicit measure of surprise as the magnitude of a prediction error.”

“…learning rule such as the Widrow-Hoff Least Mean Square learning rule (Widrow and Hoff, 1960) and the well-known error backpropagation algorithm (Rumelhart et al., 1986), where the US corresponds to the “teaching input” or “desired output,” and λ − V is the error guiding learning (although the…”

“Dopamine is a neuromodulator that is well known to play a pivotal role in motivational and reinforcement learning processes (Wise, 2004; Berridge, 2007).”

“Itti and Baldi (2005, 2006, 2009) argue that this definition has key advantages over alternatives in being more principled, more widely applicable, and more able to account for what attracts human visual attention.”

“…hypothesis of the phasic activity of dopamine neurons, which has received a large amount of empirical support and represents one of the most fruitful integrations between computational and empirical research (Ungless, 2004; Wise, 2004; Schultz, 2007; Graybiel, 2008; Glimcher, 2011).”

“Another area in which prediction errors appear in a Bayesian framework is in the “predictive coding” architectures of Rao and Ballard (1999) and Friston and Kiebel (2009).”

“It is well accepted that the hippocampal system is involved in the formation of complex episodic memories and seems to play a critical role for the detection of multiple kinds of novel associations (Wan et al., 1999; Brown and Aggleton, 2001).”

“There are very many clustering methods, and there are many methods for determining when a new cluster should be added (Markou and Singh, 2003).”

“Tribus’ notion of surprisal plays a prominent role in the global brain theory of K. Friston and colleagues which is based on the principle of “free-energy minimization” (Friston et al., 2006; Friston, 2009, 2010).”

“The typical result of these tests is the manifestation of a positive wave of the electric field happening about 200–300 ms after the odd stimulus and named “P300” or “P3” (Friedman et al., 2001).”

“Itti and Baldi (2005, 2006, 2009) argue that this definition has key advantages over alternatives in being more principled, more widely applicable, and more able to account for what attracts human visual attention.”

“…dopamine neuron activity is not triggered only by rewards and reward predictors, but by different kinds of salient stimuli (Horvitz, 2000), such as sudden visual or auditory stimuli that have never been associated with rewards (Steinfels et al., 1983; Ljungberg et al., 1992; Horvitz et al., 1997).”

“…is that phasic dopamine neuron activity is not triggered only by rewards and reward predictors, but by different kinds of salient stimuli (Horvitz, 2000), such as sudden visual or auditory stimuli that have never been associated with rewards (Steinfels et al., 1983; Ljungberg et al.,…”

“…hypothesis of the phasic activity of dopamine neurons, which has received a large amount of empirical support and represents one of the most fruitful integrations between computational and empirical research (Ungless, 2004; Wise, 2004; Schultz, 2007; Graybiel, 2008; Glimcher, 2011).”

“…function of dopamine surprise activations is to encourage the animal to engage in activity to discover which aspects of its own activity may trigger surprising events so that the animal may add new actions to its repertoire (Redgrave et al., 1999; Redgrave and Gurney, 2006; Mirolli et al., 2013).”

“Itti and Baldi (2005, 2006, 2009) argue that this definition has key advantages over alternatives in being more principled, more widely applicable, and more able to account for what attracts human visual attention.”

“Of current interest in statistics and machine learning are Bayesian non-parametric clustering methods (Gershman and Blei, 2012).”

“…inferotemporal cortex (Ranganath and Rainer, 2003), an area involved in object recognition, the perirhinal cortex (Brown and Aggleton, 2001), an area close to the hippocampus and involved in episodic memory, and the prefrontal cortex (Asaad et al., 1998), the highest multimodal associative cortex.”

“Theories of motivation, particularly of intrinsic motivation (Deci and Ryan, 1985; Baldassarre and Mirolli, 2013), place novelty and surprise among the primary factors that arouse interest and motivate exploratory or avoidance behavior.”

“Repetition suppression is stimulus spe-cific and has been observed in various types of experiments, from classification (Sobotka and Ringo, 1994) to delayed-matching-to-sample tests (Li et al., 1993).”

“…function of dopamine surprise activations is to encourage the animal to engage in activity to discover which aspects of its own activity may trigger surprising events so that the animal may add new actions to its repertoire (Redgrave et al., 1999; Redgrave and Gurney, 2006; Mirolli et al., 2013).”

“The novelty bonus idea has recently attracted much attention, and it is fostering a number of neuroimaging studies where the activation of the dopaminergic system is studied while subjects are exposed to novel stimuli (e.g., Bunzeck and Duzel, 2006; Wittmann et al., 2008; Krebs et al., 2009).”

“Models of classical conditioning based on Bayesian methods, including the Kalman filter, have been proposed that go beyond the account provided by the Rescorla-Wagner model (Dayan et al., 2000; Kakade and Dayan, 2002a; Courville et al., 2004, 2006).”

“Models of classical conditioning based on Bayesian methods, including the Kalman filter, have been proposed that go beyond the account provided by the Rescorla-Wagner model (Dayan et al., 2000; Kakade and Dayan, 2002a; Courville et al., 2004, 2006).”

“…basis of a model of classical conditioning that elaborates the Rescorla-Wagner model (Sutton and Barto, 1990) as well as the reward-prediction-error hypothesis about the phasic activity of dopamine producing neurons in the brain (Barto, 1995; Houk et al., 1995; Schultz et al., 1997; Schultz, 1998).”

“It refers to the following processes (Hasselmo and Schnell, 1994; Kumaran and Maguire, 2007; Duncan et al., 2012): (a) familiar aspects of the percept (“lures”) actively recall previous memories on the basis of pattern-completion-like mechanisms, for example, an item recalls other items previously…”

“It is well accepted that the hippocampal system is involved in the formation of complex episodic memories and seems to play a critical role for the detection of multiple kinds of novel associations (Wan et al., 1999; Brown and Aggleton, 2001).”

“A typical observation is that with repetition of a stimulus the neurons responding to it present a progressively decreasing activation, a phenomenon called repetition suppression (Ringo, 1996; Henson and Rugg, 2003).”

“Because we know that dopamine is both necessary and sufficient for appetitive instrumental conditioning (Robinson et al., 2006; Zweifel et al., 2009), it is probably safe to assume that it is phasic dopamine that mediates operant conditioning in sensory reinforcement, just as we assume that it is…”

“Intense investigation has led to the isolation of a P3a component of the wave, also called “novelty P3” (Soltani and Knight, 2000).”

“This is one of the first proposals for how ideas related to what psychologists call intrinsic motivation can be implemented in a machine learning system, and much additional research has been lately done in this area (Baldassarre and Mirolli, 2013).”

“Itti and Baldi (2005, 2006, 2009) argue that this definition has key advantages over alternatives in being more principled, more widely applicable, and more able to account for what attracts human visual attention.”

“For example, Grossberg (1982) proposed that an expectation is a feedback pattern of neural activity derived from signaling across an entire network gated by long-term memory, and that unexpected events trigger a “mismatch-modulated arousal burst,” i.e., what we would call a surprise signal.”

“In fact, not surprisingly in human experiments with novel images, it is the hippocampus that seems to be involved (e.g., Lisman and Grace, 2005), whereas light flashes trigger dopamine activity via the superior colliculus, which directly projects to the dopaminergic neurons (Dommett et al., 2005).”

“…experiments are simple light flashes or sudden sounds, and the dopaminergic responses to lights and tones typically persist after many presentations so that talking about novelty of the stimuli does not seem appropriate (Steinfels et al., 1983; Horvitz et al., 1997; Ungless, 2004).”

“The novelty bonus idea has recently attracted much attention, and it is fostering a number of neuroimaging studies where the activation of the dopaminergic system is studied while subjects are exposed to novel stimuli (e.g., Bunzeck and Duzel, 2006; Wittmann et al., 2008; Krebs et al., 2009).”

“It refers to the following processes (Hasselmo and Schnell, 1994; Kumaran and Maguire, 2007; Duncan et al., 2012): (a) familiar aspects of the percept (“lures”) actively recall previous memories on the basis of pattern-completion-like mechanisms, for example, an item recalls other items previously…”

“This Bayesian account of increases in the rate of animal learning observed in certain experiments (Rescorla, 1971) accomplishes what the Pearce-Hall model (Pearce and Hall, 1980) does via its use of an explicit measure of surprise as the magnitude of a prediction error.”

“This interpretation is consistent with the fact that in experiments dealing with spatial associative novelty, subjects are typically exposed to the associative pairings for many times before their familiarity/novelty discrimination responses are assessed (Duzel et al., 2003; Kohler et al., 2005).”

“Modern neuroscience literature distinguishes between three types of novelty to which the brain responds: stimulus nov-elty, contextual novelty, and associative novelty (Ranganath and Rainer, 2003; Kumaran and Maguire, 2007).”

“A typical observation is that with repetition of a stimulus the neurons responding to it present a progressively decreasing activation, a phenomenon called repetition suppression (Ringo, 1996; Henson and Rugg, 2003).”

“This interpretation is consistent with the fact that in experiments dealing with spatial associative novelty, subjects are typically exposed to the associative pairings for many times before their familiarity/novelty discrimination responses are assessed (Duzel et al., 2003; Kohler et al., 2005).”

“For example, if pressing a bar results in the switching on of a light, an animal will start to press the bar, much as if the bar-press were to lead to a reward such as food (e.g., Kish, 1955; Williams and Lowe, 1972; Glow and Winefield, 1978; Reed et al., 1996).”

“The novelty bonus idea has recently attracted much attention, and it is fostering a number of neuroimaging studies where the activation of the dopaminergic system is studied while subjects are exposed to novel stimuli (e.g., Bunzeck and Duzel, 2006; Wittmann et al., 2008; Krebs et al., 2009).”

“Models of classical conditioning based on Bayesian methods, including the Kalman filter, have been proposed that go beyond the account provided by the Rescorla-Wagner model (Dayan et al., 2000; Kakade and Dayan, 2002a; Courville et al., 2004, 2006).”

“Repetition suppression is stimulus spe-cific and has been observed in various types of experiments, from classification (Sobotka and Ringo, 1994) to delayed-matching-to-sample tests (Li et al., 1993).”

“…experiments are simple light flashes or sudden sounds, and the dopaminergic responses to lights and tones typically persist after many presentations so that talking about novelty of the stimuli does not seem appropriate (Steinfels et al., 1983; Horvitz et al., 1997; Ungless, 2004).”

“TD learning is not restricted to predicting reward; the role of reward can be replaced by other stimulus features, and it can be generalized to networks of interrelated predictions (Sutton and Tanner, 2004).”

“…further details and the important role this model has played in the history of animal learning theory (see Schmajuk, 2008, for a review; see also Lepora et al., 2010, and Mannella et al., 2010, for two models that capture the basic brain mechanisms with which classical conditioning is…”

“Contextual novelty is often studied in oddball experiments where, for example, sequences of a repeating auditory stimulus (e.g., a simple tone) are interleaved with rare odd signals (e.g., a “moo” of a cow) (Ranganath and Paller, 1999).”

“Because we know that dopamine is both necessary and sufficient for appetitive instrumental conditioning (Robinson et al., 2006; Zweifel et al., 2009), it is probably safe to assume that it is phasic dopamine that mediates operant conditioning in sensory reinforcement, just as we assume that it is…”

“Prominent neural network methods for novelty detection, such as methods based on self-organizing feature maps (Kohonen, 1984; Nehmzow et al., 2013), perform this basic process where the memory scan is performed in parallel by the network.”

“…important role this model has played in the history of animal learning theory (see Schmajuk, 2008, for a review; see also Lepora et al., 2010, and Mannella et al., 2010, for two models that capture the basic brain mechanisms with which classical conditioning is implemented in, respectively,…”

“For example, if pressing a bar results in the switching on of a light, an animal will start to press the bar, much as if the bar-press were to lead to a reward such as food (e.g., Kish, 1955; Williams and Lowe, 1972; Glow and Winefield, 1978; Reed et al., 1996).”

“Further support that surprise and not novelty supports sensory reinforcement comes from the evidence that light offsets are more-or-less as good reinforcers as light onsets (Glow, 1970; Russell and Glow, 1974).”

“What is now called Kamin blocking is the failure of an animal to learn to elicit a CR when a CS is presented to an animal as part of a compound that includes another CS that had been previously conditioned to elicit a CR (Moore and Schmajuk, 2008).”

“What is now called Kamin blocking is the failure of an animal to learn to elicit a CR when a CS is presented to an animal as part of a compound that includes another CS that had been previously conditioned to elicit a CR (Moore and Schmajuk, 2008).”

“For example, if pressing a bar results in the switching on of a light, an animal will start to press the bar, much as if the bar-press were to lead to a reward such as food (e.g., Kish, 1955; Williams and Lowe, 1972; Glow and Winefield, 1978; Reed et al., 1996).”

“Further support that surprise and not novelty supports sensory reinforcement comes from the evidence that light offsets are more-or-less as good reinforcers as light onsets (Glow, 1970; Russell and Glow, 1974).”

“…regarded as the drive for novelty-seeking behavior (Wittmann, Daw, Seymour, & Dolan, 2008; Krebs, Schott, Sch¨utze, & D¨uzel, 2009; Schwarten-beck et al., 2013), in which we anticipate uncertainty that can be resolved (e.g., opening a birthday present: see also Barto, Mirolli, & Baldassarre, 2013).”

“, after and before an observation); it is also known as expected Bayesian surprise or information gain [82,83].”

“According to cognitive psychology [4], novelty can be expressed either in terms of capabilities to remember an event or as a degree of surprisal [39] aroused by its observation.”

“, 2013; Kidd and Hayden, 2015), states of curiosity are often associated to a psychological interest for activities or stimuli that are surprising, novel, of intermediate complexity, or characterized by a knowledge gap or by errors in prediction, which are features that can themselves be quantified mathematically (Oud…”

“For example, if we ignore prior preferences, then the expected free energy reduces to information gain [112, 156] or intrinsic motivation [144, 149, 150].”

“7, such as intrinsic motivation, artificial curiosity [136–138] and optimal control [93,95,131].”

“One can illustrate the minimization of expected free energy in terms of curiosity and novelty (Barto et al., 2013; Schmidhuber, 2006) by simulating a rat that has never been exposed to the maze previously.”

“13 This injection can be expressed as a variational density 10 It is instructive to write out the curvature of the Lagrangian in terms of coupling among subsets of states: with a slight abuse of notation:”

“Novelty is the process of experiencing something different from what is normally encountered and is often defined as the degree of difference in an individual’s response to a stimulus [66].”

“Yet, while infants may initially be somewhat restricted to passively observing their environment, observations quickly come under the control of action (perhaps bootstrapping in a useful way on passive observations to first acquire sensorimotor contingencies; Deci and Ryan, 1985; Oudeyer, 2007; Schmidhuber, 2010; Bart…”

“As such, it is sometimes referred to as intrinsic motivation (Ryan and Deci, 1985; Oudeyer and Kaplan, 2007; Schmidhuber, 2010; Barto et al., 2013); Schmidhuber, 2010).”

“[36] investigate three kinds of noveltycomplexity (stimulus, associative, and contextual) usually”

“Note that we are using the IM signal for motivation (i.e., as a drive to select and execute a goal/behavior) rather than for learning (i.e., to update connection weights) as in most models using IMs.”

“This means that the most probable policies or paths are those that resolve uncertainty when navigating the lived world (Berlyne, 1950; Schmidhuber, 2006; Baranes and Oudeyer, 2009; Still and Precup, 2012; Barto et al., 2013; Moulin and Souchay, 2015).”

“Published by Elsevier Inc a b s t r a c t a r t i c l e i n f o”

“(Barto et al., 2013) from stimulus-specific adaptation is to com-”

“Several factors may make an entity cognitively accessible / salient.”

“The theoretical contribution of Barto et al. (2013) argues for the importance of distinguishing between novelty and surprise on the basis of a comprehensive analysis of the computational literature related to the two.”

“For a recent review of novelty, see [5].”

“Novel content offers uncommon and unexpected information (Barto et al., 2013; Shoemaker, 1996), which leads individuals to deviate from their routine style of information processing and to attend to a potential threat depicted in the information (Cappella et al.”

“In novelty-based methods, the agent gets excited on seeing a new information whereas in surprise-based methods, the agent gets higher reward when its prediction of future states goes wrong (for details see [20]).”

“This method is based on the observation that statistically infrequent responses in creative generation tasks tend to be the most unique, and common responses tend to be more routine ideas (Goff & Torrance 2002; Barto, Mirolli & Baldassarre 2013).”

“The novelty of VR indicates that it has not been previously encountered or experienced much by a person (Barto et al., 2013).”

“Further, neurorobotics approaches emphasise the importance of exploratory movements (Barto, Mirolli, & Baldassarre, 2013; Oudeyer & Kaplan, 2007; Parr & Friston, 2017; Schmidhuber, 2010).”

“Here we adopt the distinction between novelty and surprise from Barto et al. 2013: “Detecting novelty requires examining (by one means or another) the contents of memory to determine if the stimulus has or has not previously been experienced and attended to.”

“[14]), but rather goals directed to manipulate and physically change the environment for own benefit.”

“[58] criticizes such a definition by stressing that very distinct and memorable events may have low probabilities of occurring, while not being novel (e.”

“First, stimulus novelty manipulations, probably recognized by most as the archetypal cases of novelty, intend to use stimuli that have never been encountered by the observer before (Barto et al., 2013; Schomaker andMeeter, 2015).”

“Exploration of new features and novelty in an application increases interest and excitement (Barto et al., 2013).”

“…exhibit such organization – i.e., why such features are optimal for a system to survive and thrive in a dynamic environment (e. g. (Barlow, 2012; Barto et al., 2013; Friston et al., 2012, 2018, 2009; Kangassalo et al., 2020; Madhav and Cowan, 2020; Schwartenbeck et al., 2019, 2015; Schwartz,…”

“This has been referred to as ‘intrinsic motivation’ (Barto et al., 2013; Oudeyer & Kaplan, 2007; Schmidhuber, 2010) as well as the drive to seek ‘novelty’ (Schwartenbeck et al., 2019), but more generally involves a drive to learn what will be observed when visiting an unfamiliar state.”

“The novelty of places or spaces is processed by a network of brain areas involved in declarative memory, such as the hippocampus and the mammillary region of the hypothalamus 28,37,142–148 .”

“…to reduce uncertainty in the variables and parameters that de fi ne your model, which, in the active inference formulation, corresponds to inference and learning, respectively (Barto et al., 2013; Lindley, 1956; MacKay, 1992; Schmidhuber, 2006; Schwartenbeck et al., 2019; Still and Precup, 2012).”

“…using a range of stimuli and experimental manipulations, demonstrated that the amplitude of the P3b faithfully reflects visual awareness (Vogel et al., 1998; Fernandez-Duque et al., 2003; Sergent et al., 2005; Del Cul et al., 2007; Gaillard et al., 2009; Lamy et al., 2009; Batterink et al., 2012).”

“…2000; Laurent, 2008; Bianca C Wittmann, Bunzeck, Dolan, & Düzel, 2007; B. C. Wittmann, Daw, Seymour, & Dolan, 2008), and provides a bonus for organizing reward-based exploration (Barto et al., 2013; Brafman & Tennenholtz, 2003; Kakade & Dayan, 2002; Laurent, 2008; Manuel Lopes & Oudeyer, 2012).”

“…to sample outcomes that are relevant to the task at hand, even if they are associated with high uncertainty, which is an emergent behavior under the minimization of expected free energy (Barto et al., 2013; Schwartenbeck et al., 2013, 2019; Friston et al., 2015b, 2017c; Limanowski et al., 2015).”

“This is consistent with ideas that novelty is more related to memory-recall and surprise is more related to predictions [35].”

“In learning contexts, surprising information plays a key role and enhances association strength.(27) In biological terms, environmental surprise (eg, hotter than usual, less food, infection) is often analogous with threat, and the primary purpose of the brain has been posited as one of reducing uncertainty or surprise…”

“The conceptual conflation between novelty and surprise is a problem that has been noted in several fields of inquiry within behavioral and brain sciences (Barto et al., 2013; Modirshanechi et al., 2023).”

“Novelty-based IMs try to detect novel items and direct attention to novel items in exploration.”

“Another possibility is to use intrinsic motivations such as novelty or surprise , i.e., to consider as interesting those effects of actions that respectively have never/rarely been experienced before, or that violate expectations (Barto et al., 2013).”

“This underwrites intrinsic motivation in machine learning and robotics (Barto et al., 2013; Deci & Ryan, 1985; Oudeyer & Kaplan, 2007) and expected Bayesian surprise in visual search (Itti & Baldi, 2009; Sun et al., 2011) and the organization of our visual apparatus (Barlow, 1961, 1974; Linsker,…”

“There is a subtle but important distinction between surprise and novelty [40]: novelty is thought to describe a specific stimulus that is not already present in the memory system [41], whereas surprise is a violation of prediction.”

“, a Bayesian inference process conditioned on a system’s norms, such as its survival and very existence (Barto et al., 2013; Baltieri, 2019)).”

“…creativity and fun (e.g. Schmidhuber, 1991, 2010), as a component of intrinsic motivation networks that aim toward maxi-mising epistemic (or informational) gain (e.g. Baldassarre & Mirolli, 2013; Barto et al., 2013; Friston et al., 2017a; Gottlieb et al., 2013 ; Schwartenbeck et al., 2013).”

“…on novelty assessment and processing focuses on testing-paradigms that employ instances of novelty based either on being “ atypical ” or “ deviant ” from the rest of the presented stimuli, or on not having been presented before within the context of the study (Habib 2000; Barto et al., 2013).”

“These results are also broadly consistent with research demonstrating that novelty enhances attention, memory, learning, and perceived value (Barto et al., 2013; Kagan, 2009; Blut et al., 2023; Sánchez-Fernández and Iniesta-Bonillo, 2007; Leroi-Werelds, 2019; Wells et al., 2010).”