Adding prediction risk to the theory of reward learning

“The first view, which is widely used in financial economics and behavioural ecology, asserts that the brain assigns value to prospects by first computing its statistical moments (such as its expected magnitude, its variance or coefficient of variation, and its skewness) and then aggregating them into a value signa”

“Refs. [14,34, 37 ]). Figure 2. A Model for the integration of feelings, empathy and uncertainty in”

“When rewards are drawn from different probability distributions, comparable learning can be achieved by scaling the update component $*PE to the standard deviation ( of the respective reward distribution by either (438)”

“Differentiating between reducible and irreducible uncertainty requires an estimation of “expected uncertainty” based on previous prediction errors (Preuschoff & Bossaerts, 2007; Yu & Dayan, 2005).”

“In addition, recent behavioral evidence shows that human subjects adjust their learning rate to changing risk, implying that they must somehow be tracking risk (Behrens et al., 2007; Preuschoff and Bossaerts, 2007).”

“Theoretical considerations suggest that error teaching signals that are scaled by variance or standard deviation rather than mean can mediate stable learning that is resistant to the predicted risk of outcomes [30].”

“, 2005) as well as computational models of risk learning (Dayan and Yu, 2006; Preuschoff and Bossaerts, 2007).”

“Optimal algorithms, such as the Kalman filter, developed by engineers in the 1960s form a foundation for modern accounts of learning in cognitive science and neuroscience (Bach and Dolan, 2012; Bland and Schaefer, 2012; Daunizeau et al., 2010; Mathys et al., 2011; Nassar et al., 2010; Payzan-LeNestour and Bossaerts, 2…”

“This is because, in its numerator, c3 (the precision weight of ε3) contains an equivalent to a dynamic learning rate (Preuschoff and Bossaerts, 2007) for updating cue-outcome contingencies (see Equation A.”

“In this study, we have used regression analyses approaches that also attempt to decompose the same behavioral performance on the IGT into simpler constructs of moment-to-moment observer state, with focus on reward and risk processing, and prediction error [Hollander et al., 2005; McClure et al., 2003; O’Doherty et al.…”

“There, uncertainty is monolithic: it is the expected magnitude of the prediction error [6].”

“, 2009) and have been linked with variables such as arousal, anxiety, risk preference, or global learning rates (Nassar et al., 2012; Preuschoff and Bossaerts, 2007).”

“As mentioned above, the form of these update equations is similar to those of RW learning, providing a Bayesian perspective on reinforcement learning theory [32].”

“A normalized error signal would factor out the predicted risk of outcomes and may contribute to stable learning irrespective of risk (Preuschoff & Bossaerts 2007).”

“In addition to the rapid glutamatergic (and GABAergic) signaling between layers in a cortical hierarchy, there are slower neuormodulatory effects, for example, dopamine and acetylcholine appear to code a confidence estimate or uncertainty in predictions and prediction errors (Preuschoff and Bossaerts, 2007; Yu and Day…”

“But another important finding was that bilateral anterior insula also showed an ongoing hemodynamic response to the variance in payoff (i.e., risk) and to an error signal related to this variance (also see Preuschoff and Bossaerts, 2007; Knutson and Bossaerts, 2007).”

“…sound statistical reasoning; these considerations typically lead to models containing both a RW-like error-driven update, scaled by a PH-like associability mechanism (Sutton, 1992; Dayan & Long, 1998; Kakade & Dayan, 2002; Courville et al., 2006; Behrens et al., 2007; Bossaerts & Preuschoff, 2007).”

“The prediction error in those models reflects the magnitude of the difference between actual and expected rewards, unscaled by the anticipated range or standard deviation of rewards (but see Preuschoff & Bossaerts, 2007).”

“…the pattern of striatal activity is consistent with a reward prediction error (PE) signal, the pattern of activity in the insula may instead reflect a risk prediction error signal (r-PE) (Preuschoff and Bossaerts, 2007) that is referenced to the position of the subject either as seller or buyer.”

“It is only more recently that researchers have realized that the learning rate should also depend on confidence (10, 40).”

“A high degree of variability can be encoded and leads in time to an adaptation of learning such that a given magnitude of a specific PE produces less updating (Diederen and Schultz, 2015; Preuschoff and Bossaerts, 2007; Preuschoff et al., 2008).”

“How exactly the learning rate is to be changed is complicated ( Preuschoff and Bossaerts 2007 ), which again makes one wonder how the brain may go about optimizing the learning rate.”

“In fact, in order to determine the correct value of an uncertain outcome, it is necessary to evaluate the risk, which is then used to estimate the probability of a possible reward (Preuschoff and Bossaerts, 2007).”

“In a stable environment, estimation uncertainty—uncertainty about the parameters of the environment—generally decreases over time, as more and more observations are made to be consistent with a particular state of the environment.”

“…prediction parallels previous work demonstrating that humans (Behrens et al., 2007; Speekenbrink & Shanks, 2010) and animals (Dayan et al., 2000; Courville et al., 2006; Preuschoff & Bossaerts, 2007) modulate their learning rates in response to the volatility of changes in associational structure.”

“The precision weight is also updated with every trial and can be regarded as equivalent to a dynamic learning rate in reward learning models (Preuschoff and Bossaerts, 2007).”

“, 2010), are used to scale the signed errors that develop in the dACC (Behrens et al., 2007; Courville et al., 2006; Kakade and Dayan, 2002; Preuschoff and Bossaerts, 2007).”

“To avoid unstable learning, it is thus essential to compare PEs to the expected fluctuation in reward value (Preuschoff and Bossaerts, 2007), and update values more when PEs with higher precision are encountered.”

“The second, which we termed prior reliability, is a form of reducible uncertainty that reflects ambiguity, typically resulting from undersampling, about the current generative process 32,33 .”

“reward feedback: (1) to determine the optimal learning rates based on optimization or on the level of uncertainty in a given environment (RL models) (Doya, 2002; Schweighofer and Doya, 2003; Preuschoff and Bossaerts, 2007); (2) to identify the correct model of the world, which includes the amount of uncertainty to pro…”

“…rt and the estimated predicted reward ̂vt : (3) Empirical data and theoretical considerations suggest that reward prediction error is scaled by predicted risk in the brain (Preushoff and Bossaerts, 2007; Tobler et al., 2005), allowing the learning rate to be independent of the payoff variance.”

“, the expected outcome variance, also plays a significant role in reward learning because it represents a predictability measure (Preuschoff and Bossaerts, 2007).”

“Thus far, it is unclear whether scaling of prediction errors relative to the variability of reward distributions results in improved performance, as predicted by learning models (Preuschoff and Bossaerts 2007).”

“…of research into the inner workings of the human brain and have subsequently made appeals to brain images common in contemporary discussions on just about everything from the causes of gambling addiction (Preuschoff and Bossaerts, 2007) to debates about why music stirs the emotions (Koelsch, 2005).”

“However, an observer who is using a statistical decision strategy will revise this estimate more when category variance is low than high (Preuschoff and Bossaerts, 2007).”

“…errors (Pearce & Hall, 1980; Hayden, Heilbronner, Pearson, & Platt, 2011; Roesch, Esber, Li, Daw, & Schoenbaumn, 2012), risk prediction errors (Preuschoff & Bossaerts, 2007; Preuschoff, Quartz, & Bossaerts, 2008), uncertainty-based jump detection (Nassar, Wilson, Heasly, & Gold, 2010;…”

“Over time, a haphazard error signal – even a strong one – will become discounted – it will not contribute to updating (Preuschoff & Bossaerts, 2007).”

“Given the role of the insula in reward learning [Preuschoff and Bossaerts, 2007] and interoception, our results suggest that with increased age, heavy alcohol exposure might lead to deficits in self awareness and increase sensitivity to physical sensations of craving [Goldstein et al., 2009].”

“Theoretical models have suggested that it would be useful to track prediction risk (i.e., the expected size of a prediction error) (Yu and Dayan, 2005; Preuschoff and Bossaerts, 2007), as it could be used to optimize the reaction to the prediction error.”

“4Faculty of Medicine and University Hospital Cologne, Department I of Internal Medicine, Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf (CIO ABCD) and Excellence Center for Medical Mycology (ECMM), University of Cologne, Cologne, Germany.”

“Adaptive coding of PEs enables an enhanced discriminability to remain sensitive for changes of all sizes, which optimizes efficient coding in neural circuits with given limitations, namely, its firing ranges (28).”

“One way to understand this effect (due to Preuschoff and Bossaerts, 2007) is to assume that the neurons report not the raw prediction error δk but instead the prediction error scaled by the learning rate, κkUδk, which is, anyway, the quantity that should ultimately control the amount of plasticity…”

“By contrast, strong sensitivity to recent trials is suboptimal in tasks where there is substantial uncertainty about reward delivery (16).”

“…past functional neuroimaging studies have identified multiple brain regions associated with making decisions involving risk [43,44]: The (posterior) parietal cortex is thought to code probability [45] and its gray matter density has been related to individual differences in risk preference [46,47].”

“) Note that this will be the qualitative direction of update even in a variance-sensitive version of the mean-tracking approach in which the variance estimate controls the step size for the update (Dayan and Long, 1998; Dayan et al., 2000; Courville et al., 2006; Preuschoff and Bossaerts, 2007).”

“Likewise, risk prediction is also incorporated in the reinforcement learning process in which the trade-off between expected reward and risk determines behavior (Preuschoff and Bossaerts, 2007).”

“…interact with other brain areas that are modulated by reward uncertainty (Hsu et al., 2005; McCoy and Platt, 2005; Preuschoff et al., 2006, 2008; Preuschoff and Bossaerts, 2007; O’Neill and Schultz, 2010; Burke and Tobler, 2011a; Schultz et al., 2011; Monosov and Hikosaka, 2012, 2013; Monosov…”

“The intuition has a rigorous underpinning [see (Preuschoff and Bossaerts 2007)].”

“Several Bayesian approaches (Kakade and Dayan, 2002; Courville et al., 2006; Preuschoff and Bossaerts, 2007) suggested that after a surprising event, animals should pay more attention to stimuli and that these events provoke the animals into learning faster.”

“On the other hand, the VTA is involved in risk-taking and reward evaluation (Schultz, 1998; Tobler et al., 2005; Preuschoff and Bossaerts, 2007) as well as the processing of nociception (Sotres-Bayón et al.”

“This model included prediction errors for both reward and risk, following the proposition of Preushoff and Bossaerts (Preuschoff and Bossaerts, 2007).”

“…influenced by external contextual factors (Behrens et al., 2007; Fiorillo, Tobler & Schultz, 2007; Hariri et al., 2006; Porcelli & Delgado, 2009; Preuschoff & Bossaerts, 2007; Rangel, 2008; Seaman et al., 2018), and/or by subjective internal factors (Hetherington, Pirie & Nabb, 2002; Juechems…”

“In this context, conventional model-free RPE learning algorithms (Rescorla and Wagner, 1972) can only learn the average or “expected” reward that is constant over cues, while more recent theories permit subjects to discriminate based on reward variance, risk or precision (Preuschoff and Bossaerts, 2007; Schultz et al.…”

“While recent RL developments in neuroscience incorporate some notion of uncertainty [21], learning and generalization are typically non-probabilistic.”

“This parameter should change with the uncertainty in the environment (Behrens, Woolrich, Walton, & Rushworth, 2007; Preuschoff & Bossaerts, 2007; Yu & Dayan, 2003).”

“…corresponding to precision, modulates the influence of prediction errors on updating priors, thereby adjusting the learning rate from current sensory information (Preuschoff et al., 2006; Preuschoff and Bossaerts, 2007; Nassar et al., 2012; Mcguire et al., 2014; Diederen and Schultz, 2015).”

“There, the learning rate increases, and hence, learning accelerates, when an outcome is dissimilar to outcomes experienced in the past; it decreases if surprise is low (Pearce and Hall, 1980; Behrens et al., 2007; Preuschoff and Bossaerts, 2007; Payzan-LeNestour et al., 2013; McGuire et al., 2014).”

“RPEs were included in this model, following the suggestion by Preushoff and Bossaert (Preuschoff and Bossaerts, 2007).”

“At the second level of the HGF, the precision weight (c 2) is updated with every trial, so that it also can be considered as the equivalence of a dynamic learning rate in reward-learning models (Preuschoff and Bossaerts, 2007; Mathys et al., 2011).”

“Or should we learn to ignore our errors in prediction, as would be needful in a setting that was simply noisy and unpredictable (Preuschoff & Bossaerts, 2007)?”