A Neural Computational Model of Incentive Salience

“The estimate is based on summed values from reward prediction errors – the discrepancy between the reward expected from a stimulus (technically, a state) and the reward actually received [20,23–29].”

“In addicts, amplified motivation may maladaptively pull the addict like a magnet towards compulsively ‘wanted’ drugs, and so make it harder to escape from the addiction [8,13,98].”

“(4) is an expression of what is known as the ‘‘quasi-hyperbolic’’ discounting model encountered in economics literature [56,57].”

“(4) is an expression of what is known as the ‘‘quasi-hyperbolic’’ discounting model encountered in economics literature [56,57].”

“The proposal here exploits the two parameters in the quasi-hyperbolic discount model format and is the basis of our current postulate of a gain-control mechanism to implement incentive salience computations, in a manner that can be sensitive to inter-temporal comparisons of values when such comparisons play important…”

“Further, Eqn (3a) would also apply to changes in aversiveness from bad to worse (or vice versa) for fear, disgust or other negative evaluations that involve a negative version of motivation salience, such as changes in active fear or in psychotic paranoia produced by dopamine-blocking drugs or by psychological mood ma…”

“The estimate is based on summed values from reward prediction errors – the discrepancy between the reward expected from a stimulus (technically, a state) and the reward actually received [20,23–29].”

“This decision utility can be changed by shifts of physiological state without necessarily changing either the remembered utility of the reward previously associated with that cue, the predicted utility of expected future rewards, or the hedonic experienced utility of those rewards when eventually consumed [2,9,16].”

“Yet differences remain between our Pavlovian-guided incentive salience model (mediated by mesolimbic brain circuits) and a cognitive map or full look-ahead tree model (plausibly mediated by cortical brain circuits) [39,95].”

“One influential neurobiological view identifies the predictive error signal, which lies at the core of temporal difference learning, with the firing of dopaminergic neurons projecting to the nucleus accumbens and neostriatum [30,31] (though this notion is not without controversy regarding the causal role of dopamine s…”

“, how states unfold successively and how one’s actions affect such statetransition) [27,31,34–36].”

“, how states unfold successively and how one’s actions affect such statetransition) [27,31,34–36].”

“[59].”

“In addicts, amplified motivation may maladaptively pull the addict like a magnet towards compulsively ‘wanted’ drugs, and so make it harder to escape from the addiction [8,13, 98 ].”

“One influential neurobiological view identifies the predictive error signal, which lies at the core of temporal difference learning, with the firing of dopaminergic neurons projecting to the nucleus accumbens and neostriatum [30,31] (though this notion is not without controversy regarding the causal role of dopamine s…”

“feature is precisely why hungers make their relevant cues much more powerful motivators, as well as why a food CS can trigger an increase in appetite for its UCS in a not particularly hungry person or a drug CS trigger relapse in an addict [2–5,7,8,19,89,96,97].”

“In studies to tease apart ‘wanting’ from ‘liking’ and from learned predictions of reward, we used a simple, serial paradigm containing two CSs that predicted a sucrose UCS [13, 75 ].”

“One solution to account for immediate state-based changes in behavior has been to posit that dopamine also modulates the ‘‘vigor’’ of all responses in a general fashion [37,38].”

“Psychological counterparts include cognitive maps of goal outcomes, with values obtained by episodic memories of experience with those goals in states similar to current conditions, and understanding of act-outcome relationships needed to obtain those goals [17,18,89,92–94].”

“The estimate is based on summed values from reward prediction errors – the discrepancy between the reward expected from a stimulus (technically, a state) and the reward actually received [20,23–29].”

“One influential neurobiological view identifies the predictive error signal, which lies at the core of temporal difference learning, with the firing of dopaminergic neurons projecting to the nucleus accumbens and neostriatum [30,31] (though this notion is not without controversy regarding the causal role of dopamine s…”

“In PIT, the phasic peaks of cue-triggered ‘wanting’ are manifest as a burst of pressing by the rat on a lever that previously earned sucrose reward: these peaks were dynamically enhanced by microinjection of amphetamine directly into the nucleus accumbens, or by sensitizing drug binges given weeks earlier [72,73] (Fig…”

“One influential neurobiological view identifies the predictive error signal, which lies at the core of temporal difference learning, with the firing of dopaminergic neurons projecting to the nucleus accumbens and neostriatum [30,31] (though this notion is not without controversy regarding the causal role of dopamine s…”

“Cached values and model-based predictions of reward both often assume that a reward will be about as good in the future as it was in the past [14,22,26,40].”

“In addition to receiving mesolimbic outputs, dopamine projections from VTA also ascend directly to ventral pallidum [67,68].”

“To the extent that computational models aim to capture real psychological cognition, retasting may remain an important feature of many model-based systems [17,18,21,22,28,39,40,44].”

“alliesthesia modulation of the hedonic value of their UCS rewards, but perhaps even more strongly [15]).”

“In PIT, the phasic peaks of cue-triggered ‘wanting’ are manifest as a burst of pressing by the rat on a lever that previously earned sucrose reward: these peaks were dynamically enhanced by microinjection of amphetamine directly into the nucleus accumbens, or by sensitizing drug binges given weeks earlier [72,73] (Fig…”

“One difference is that attribution of incentive salience to a CS makes the cue itself become motivationally ‘wanted’, beyond being a signal or trigger to ‘want’ the UCS goal [84,85].”

“One solution to account for immediate state-based changes in behavior has been to posit that dopamine also modulates the ‘‘vigor’’ of all responses in a general fashion [37,38].”

“To the extent that computational models aim to capture real psychological cognition, retasting may remain an important feature of many model-based systems [17,18,21,22,28,39,40,44].”

“From ventral pallidum these signals then pass to a thalamic relay for return to mesocorticolimbic loops, or directly descend to other subcortical outputs [62,65,66].”

“Consistent with contemporary views, we believe that multiple reward-related learning processes exist within a single brain, mediated by separable brain systems [7,14,17,18,21,22,38–40,42– 45].”

“feature is precisely why hungers make their relevant cues much more powerful motivators, as well as why a food CS can trigger an increase in appetite for its UCS in a not particularly hungry person or a drug CS trigger relapse in an addict [2–5,7,8,19,89,96,97].”

“The TD error d was identified with the ‘‘reboosting’’ process of a CS posited by the incentive salience hypothesis to occur at the moment of UCS [1,46].”

“For example, sensitization or an increased physiological appetitive state (k becomes greater than 1) might lead to a decrease in the temporal horizon c [60], producing sharper temporal discounting effects, such that motivational value increases with degree of temporal proximity to reward UCS [61].”

“sensitization changed this profile by enhancing only incentive salience signals to the CS2, at the expense of the signal for CS1 (and without altering UCS signal) [13] (Figure 4).”

“More flexible are tree-search models, sometimes called modelbased systems because they model or map the world in a way that explicitly represents goal values and relationships in a nested tree or similar recursive structure [14,17,21,22,39,40,92].”

“Thus it seems safe to conclude that dynamic increases in incentive salience are expressed in behavior as well as in neural activation [15,86–90].”

“Polarity reversal would similarly encompass cases in which motivational salience changes valence between desire and dread [53,54].”

“Resolving this translation of univalent neural firing into bivalent psychological values is beyond our current scope, though we have discussed it at greater length in other papers [50,51,71].”

“We suggest additionally that a model of incentive salience should also incorporate dynamic physiological modulation by current states to capture more sudden changes in CS motivational value, such as in specific appetites or drug enhancement of ‘wanting’, which do not proceed by gradual reboosting via UCS prediction er…”

“One difference is that attribution of incentive salience to a CS makes the cue itself become motivationally ‘wanted’, beyond being a signal or trigger to ‘want’ the UCS goal [84,85].”

“These circuits include input from mesolimbic dopamine projections from ventral tegmentum and substantia nigra to the nucleus accumbens, ventral pallidum, and amygdala; and output projections from nucleus accumbens that converge through ventral pallidum [62–64].”

“Polarity reversal would similarly encompass cases in which motivational salience changes valence between desire and dread [53,54].”

“Psychological counterparts include cognitive maps of goal outcomes, with values obtained by episodic memories of experience with those goals in states similar to current conditions, and understanding of act-outcome relationships needed to obtain those goals [17,18,89,92–94].”

“Reboosting was a concept added a decade earlier by the original incentive salience proposals to account for the gradual decrement effects on rewarded behavior produced by administering neuroleptic drugs that partially blocked dopamine receptors (anhedonia-like effects or extinction mimicry) [47].”

“We suggest additionally that a model of incentive salience should also incorporate dynamic physiological modulation by current states to capture more sudden changes in CS motivational value, such as in specific appetites or drug enhancement of ‘wanting’, which do not proceed by gradual reboosting via UCS prediction er…”

“This decision utility can be changed by shifts of physiological state without necessarily changing either the remembered utility of the reward previously associated with that cue, the predicted utility of expected future rewards, or the hedonic experienced utility of those rewards when eventually consumed [2,9,16].”

“The most convincing example occurs when a never-before experienced physiological state, such as a specific appetite, modulates cue-triggered motivation appropriately for a previously learned Pavlovian cue [2,3,9–14].”

“Thus it seems safe to conclude that dynamic increases in incentive salience are expressed in behavior as well as in neural activation [15,86–90].”

“Such dynamic enhancement of CS incentive salience is also consistent with other behavioral demonstrations of incentive motivation enhancement by pharmacological dopamine activation or by psychostimulant-induced neural sensitization [72,73,79,80], even in the absence of additional learning [81–83].”

“In addition to receiving mesolimbic outputs, dopamine projections from VTA also ascend directly to ventral pallidum [67,68].”

“From ventral pallidum these signals then pass to a thalamic relay for return to mesocorticolimbic loops, or directly descend to other subcortical outputs [62,65,66].”

“The rationale is similar to a method proposed by one of us earlier (called ‘‘Locus Analysis’’) to characterize neurons in the primary motor cortex [78].”

“Such dynamic enhancement of CS incentive salience is also consistent with other behavioral demonstrations of incentive motivation enhancement by pharmacological dopamine activation or by psychostimulant-induced neural sensitization [72,73,79,80], even in the absence of additional learning [81–83].”

“From ventral pallidum these signals then pass to a thalamic relay for return to mesocorticolimbic loops, or directly descend to other subcortical outputs [62,65,66].”

“feature is precisely why hungers make their relevant cues much more powerful motivators, as well as why a food CS can trigger an increase in appetite for its UCS in a not particularly hungry person or a drug CS trigger relapse in an addict [2–5,7,8,19,89,96,97].”

“Specific candidates for polarity reversal include reversals in reward values from nasty to nice, such as described below where an intensely salty taste reward is re-encountered during a salt appetite, or from nice to nasty such as after taste-aversion devaluation [10,50–52] (Figure 1).”

“[77].”

“Such dynamic enhancement of CS incentive salience is also consistent with other behavioral demonstrations of incentive motivation enhancement by pharmacological dopamine activation or by psychostimulant-induced neural sensitization [72,73,79,80], even in the absence of additional learning [81–83].”

“Ordinarily the rats would not prefer to consume either CS solution [ 10 ,50].”

“But together in the right combinations they are motivationally compelling in a synergy that is greater than the sum of the parts (Zhang et al., 2009).”

“The first form, which is usually applicable, is multiplicative (A; equation 3a in Zhang et al., 2009): A : ~rðrt; jÞ ¼ j rt: This form applies to the amplification of ‘wanting’ for a reward by addictive drug administration or by permanent drug-induced neural sensitization.”

“Separate animals (n = 10) were used to calculate spread of microinjection using Fos plume measurements used to guide functional mapping (SI Materials and Methods and SI Results).”

“These findings support the observations that motivational state at the time of choice can impact an animal's behavior to approach an action which had been preferentially associated with negative outcomes during learning (Zhang et al., 2009).”