Habits without values.

In the paper

“as well as the effects of reinforcement schedule on the rate of habit formation.”

What the sources say

• Not checked yetcites Adams, 1982
• Not checked yetcites Dickinson, 1985
• Not checked yetcites Gremel & Costa, 2013

In the paper

“We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation”

In the paper

“The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system.”

In the paper

“This value-free habit mechanism accounts for key findings in the animal learning literature that dissociate habitual and goal-directed actions, namely the tendency for an animal to continue performing a previously learned action when that action is no longer predictive of the reinforcing outcome (contingency degradation; Hammond, 1980) or when the predicted outcome ceases to be desired by the subj”

What the sources say

• Not checked yetcites Hammond, 1980
• Not checked yetcites Adams & Dickinson, 1981

In the paper

“As a result, when the reversal occurs, the agent continues to select Action A for an extended period, past the point where the goal-directed controller has learned that Action B is more likely to be reinforced (compare Figures 3B and 3E).”

In the paper

“In addition, this model provides what is, to our knowledge, the first computational account of the difference in rate of habit formation under variable-interval and variable-ratio schedules of reinforcement (Adams, 1982; Dickinson, 1985; Gremel & Costa, 2013).”

What the sources say

• Not checked yetcites Adams, 1982
• Not checked yetcites Dickinson, 1985
• Not checked yetcites Gremel & Costa, 2013

In the paper

“Initially, behavior is driven by the goal-directed controller, which gradually learns the relative reinforcement rates (Figure 3B) and thus increasingly selects Action A.”

In the paper

“In the following sections we demonstrate that this model can capture all of the key diagnostic features of habitual behavior previously identified in animal behavior, including sensitivity to repetition frequency, reinforcement schedule, and selective modulation by lesions to one of two dissociable neural circuits.”

In the paper

“In addition to demonstrating the different kinds of learning that drive each of these controllers, this example demonstrates that our model captures the observation that behavioral control in a novel environment tends to evolve from goal-directed to habitual (i.e., habits form from actions that were originally selected in a value-based manner).”

In the paper

“A “control” agent, with intact habitual and goal-directed controllers, adopted a high press rate that adapted to both manipulations when given limited training, but did not adapt to either following extensive training.”

In the paper

“whereas lesioning the habitual controller led to a high press rate that adapted to both manipulations (Figure 5).”

In the paper

“We simulated performance in such an environment and found that the model was able to simultaneously reproduce both the reinforcement-seeking (value-based) and perseverative (value-free) components of these behaviors (Figure 7, right, blue points).”

In the paper

“As these habits strengthen, the habitual controller increasingly drives the choice of which action to select (Figure 3D).”

In the paper

“With longer training sessions, however, the model failed to reverse its actions within the same time period (Figure 3, left).”

In the paper

“As it does so, the habitual controller strengthens its association between the current stimuli and Action A (Figure 3C).”

In the paper

“with experience in this environment the agent learns to press the lever on a large fraction of trials (Figure 4, blue curves).”

In the paper

“When the agent was given a small number of training trials, it successfully learned to decrease probability of lever pressing following this reversal.”

In the paper

“We found that the frequency of choosing to lever-press in the testing stage strongly depended on the duration of the training stage (Figure 3, right)”

In the paper

“Replacing the habitual component of our model with a model-free reinforcement learning system rendered it unable to reproduce the perseverative pattern (Figure 7, right, red points).”

In the paper

“Furthermore, a value-free habit mechanism explains a variety of other behavioral phenomena in which responses are facilitated by simple repetition (i.e. perseveration”

In the paper

“we found that lesioning the goal-directed system produced a low press rate that was unaffected by either omission or devaluation”

In the paper

“We demonstrate that such a mechanism leads to perseveration of a previously higher-value action following contingency degradation or outcome devaluation”

In the paper

“We further show that such habitual behaviors are diminished by simulating lesions to a habitual system, consistent with classic findings in the animal behavior literature.”

In the paper

“and increased perseveration of all actions in a probabilistic choice task with varying action-outcome contingencies.”

In the paper

“Crucially, the system that generates these habitual behaviors does so without engaging in any manner of reinforcement learning (model-free or otherwise), consistent with theories that place habits outside the domain of RL.”

In the paper

“we also show that our model can still capture key features of behavior for a task that is thought to rely on both model-free and model-based control”

In the paper

“in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes.”

In the paper

“or when behavior is repeated with a high rate of consistency (Lally, van Jaarsveld, Potts, & Wardle, 2010).”

What the sources say

• Not checked yetcites Lally, van Jaarsveld, Potts, & Wardle, 2010

In the paper

“Such a value-free formulation of habits aligns well with Thorndike’s second law, the “Law of Exercise.””

In the paper

“First, habits only arise in familiar contexts, typically developing out of behaviors initially undertaken under goal-directed control (i.e., based on expected reward; Wood & Neal, 2007).”

What the sources say

• Not checked yetcites Wood & Neal, 2007

In the paper

“Second, habits tend to form most strongly in circumstances where actions are repeated very consistently (Dickinson, 1985; Wood & Rünger, 2016).”

What the sources say

• Not checked yetcites Dickinson, 1985
• Not checked yetcites Wood & Rünger, 2016

In the paper

“a large body of research has demonstrated that behaviors become habitual (i.e.., are faster, more accurate, and less susceptible to interference) the more often those behaviors are performed in the presence of a particular set of cues (reviewed in Wood & Neal, 2007; Wood & Runger, 2016).”

What the sources say

• Not checked yetcites Wood & Neal, 2007
• Not checked yetcites Wood & Runger, 2016

In the paper

“The latter, value-free definition for habits drove the development of critical assays that have been used to discriminate between actions that are habitual versus goal-directed (i.e., outcome-sensitive), testing whether an animal continues to pursue a previous course of action when that action is no longer the most beneficial (Adams & Dickinson, 1981; Hammond, 1980).”

What the sources say

• Not checked yetcites Adams & Dickinson, 1981
• Not checked yetcites Hammond, 1980

In the paper

“Here, we present a computational implementation of the Law of Exercise and show that it offers an alternative to model-free RL as a mechanism for habits, one that retains ideas about the nature of habits that have developed within other areas of psychology and neuroscience (Graybiel, 2008; James, 1890; Wood & Rünger, 2016).”

What the sources say

• Not checked yetcites Graybiel, 2008
• Not checked yetcites James, 1890
• Not checked yetcites Wood & Rünger, 2016

In the paper

“and quickly when the environment is relatively unpredictable (Derusso et al., 2010)”

What the sources say

• Not checked yetcites Derusso et al., 2010

In the paper

“Habits form slowly in environments where different behaviors lead to very different outcomes (Adams, 1982)”

What the sources say

• Not checked yetcites Adams, 1982

In the paper

“Once a behavior has become a habit, that behavior is rendered inflexible with respect to changes in the environment, including those which make the behavior undesirable (Adams & Dickinson, 1981; Hammond, 1980).”

What the sources say

• Not checked yetcites Adams & Dickinson, 1981
• Not checked yetcites Hammond, 1980

In the paper

“Habits, on the other hand, are understood to be driven by direct links between cues in the environment and the actions that have often followed those cues (i.e. “stimulus-response” associations).”

What the sources say

• Not checked yetcites Balleine & O’Doherty, 2010
• Not checked yetcites Dickinson, 1985
• Not checked yetcites Dolan & Dayan, 2013

In the paper

“In contrast to this view, modern computational accounts typically model habits as mediated by reinforcement-learning mechanisms, which are outcome-sensitive (Figure 1, right).”

What the sources say

• Not checked yetcites Daw, Niv, & Dayan, 2005
• Not checked yetcites Dolan & Dayan, 2013
• Not checked yetcites Sutton & Barto, 1998

In the paper

“We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data.”

In the paper

“Equating these terms, however, carries a critical assumption: that habits are driven by a reward-maximization process (i.e., a process that depends directly on potential outcomes).”

In the paper

“When combined with another classic idea from the literature on habits – that large action-outcome contingencies delay habit formation – our model is able to explain another classic finding in the literature on habitual control: the effect of reinforcement schedule on the rate of habit formation.”

In the paper

“Actions followed by a constant probability of reinforcement (variable-ratio [VR] schedules) take a long time to habitize (Adams, 1982).”

What the sources say

• Not checked yetcites Adams, 1982

In the paper

“By replacing model-free RL with a value-free mechanism, our model forces a critical realignment of this prevailing framework, thereby prompting a deeper consideration of how the computations and circuitry for model-free and model-based reinforcement learning might share more commonalities than differences.”

In the paper

“By contrast, when an action is “baited” at a constant probability per unit time, and only the first press following baiting results in reinforcement (variable-interval [VI] schedules), the action habitizes much more quickly, even when performance and overall rate of reinforcement are matched across these two reinforcement schedules (Dickinson, Nicholas, & Adams, 1983).”

What the sources say

• Not checked yetcites Dickinson, Nicholas, & Adams, 1983

In the paper

“We have constructed a computational model in which habits consist of value-free associations between stimuli and actions, and in which these associations are strengthened each time that action is performed in response to that stimulus. This model reproduces key features of the behavioral literature on habits.”

In the paper

“Overtrained animals will often continue to perform an action even when it no longer causes the desired outcome (indeed, even when it prevents the delivery of the outcome).”

What the sources say

• Not checked yetcites Dickinson, 1998

In the paper

“Lesions to our model’s goal-directed and habitual controllers thus reproduce behavioral patterns typical of DMS and DLS lesioned rats, respectively, in classic experiments on instrumental conditioning.”

In the paper

“In situations where goal-directed control consistently produces the same behavior in response to the same stimulus, that behavior is likely to become a habit.”

In the paper

“In tasks where humans and animals make repeated decisions between similar alternatives, a near-universal observation is a tendency to select actions that have frequently been selected in the past, regardless of their outcome or of the task stimuli.”

In the paper

“indicating that overtraining caused the agent to perseverate on the habitized behavior (lever-pressing).”

In the paper

“Another central finding from research on habitual control is that the reinforcement schedule has a profound effect on habit formation.”

In the paper

“This difference was largely driven by the difference in action-outcome contingencies inherent to each schedule: a small change in press rate resulted in a much larger change in reinforcement rate for a VR schedule relative to a VI schedule”

In the paper

“This perseverative behavior is diagnostic of habitual control (Dickinson, 1998).”

What the sources say

• Not checked yetcites Dickinson, 1998

In the paper

“we turn to the ubiquitous and poorly understood phenomenon of perseveration, which we argue can be understood as a manifestation of habitual control.”

In the paper

“Once a habit has formed, behavior can become inflexible in the face of changes to the environment that render it no longer desirable, such as contingency omission (in which the reinforcer that drove initial acquisition of the behavior is delivered only when the behavior is not performed) and outcome devaluation (in which the reinforcer is rendered no longer valuable to the subject).”

In the paper

“Lesions to regions of a putative “habit system,” such as the dorsolateral striatum (DLS), promote behavioral flexibility (i.e., alleviate perseveration) following overtraining (Graybiel, 2008; Yin & Knowlton, 2006).”

What the sources say

• Not checked yetcites Graybiel, 2008
• Not checked yetcites Yin & Knowlton, 2006

In the paper

“in this setup, overtrained animals will often continue performing an action that leads to an outcome they no longer desire (Adams, 1982).”

What the sources say

• Not checked yetcites Adams, 1982

In the paper

“For instance, in instructed task settings with human subjects, the speed and accuracy of an action are enhanced when that action has been recently performed (Bertelson, 1965; Cho et al., 2002).”

What the sources say

• Not checked yetcites Bertelson, 1965
• Not checked yetcites Cho et al., 2002

In the paper

“This is consistent with data from animal learning, in which overtraining abolishes behavioral sensitivity to omission contingencies (Dickinson, 1998)”

What the sources say

• Not checked yetcites Dickinson, 1998

In the paper

“goal-directed behavior can be disturbed by perturbations to any of a network of interconnected brain regions, including prelimbic cortex (PL; Balleine & Dickinson, 1998; Corbit & Balleine, 2003; Killcross & Coutureau, 2003), dorsomedial striatum (DMS; Yin et al., 2005), mediodorsal thalamus (Corbit, Muir, & Balleine, 2003), basolateral amygdala (Balleine, Killcross, & Dickinson, 2003), and orbitof”

What the sources say

• Not checked yetcites Balleine & Dickinson, 1998
• Not checked yetcites Yin et al., 2005
• Not checked yetcites Jones et al., 2012

In the paper

“Conversely, lesions to regions of a putative “goal-directed system,” such as the dorsomedial striatum (DMS) impair flexibility (Yin, Ostlund, Knowlton, & Balleine, 2005).”

What the sources say

• Not checked yetcites Yin, Ostlund, Knowlton, & Balleine, 2005

In the paper

“In contrast to the literature on the habitual/goal-directed dichotomy, such clean dissociations have largely evaded investigations into the neural substrates of model-based and model-free computations, which can theoretically be differentiated in several ways (Doll et al., 2012).”

What the sources say

• Not checked yetcites Doll et al., 2012

In the paper

“Our proposal avoids this tension entirely by positing that habits are instantiated not by model-free RL, but by mechanisms that are entirely value-free.”

In the paper

“Rather, regions that respond to hypothetical outcomes (a model-based construct) – such as the OFC, vmPFC, and dorsal ACC – tend also to respond to actual outcomes (Abe, Seo, & Lee, 2011; Camille et al., 2004; Coricelli et al., 2005; Hayden, Pearson, & Platt, 2011; Lohrenz, McCabe, Camerer, & Montague, 2007; Rushworth, Noonan, Boorman, Walton, & Behrens, 2011; Strait, Blanchard, & Hayden, 2014).”

What the sources say

• Not checked yetcites Abe, Seo, & Lee, 2011
• Not checked yetcites Camille et al., 2004
• Not checked yetcites Coricelli et al., 2005

In the paper

“Regions that respond to model-free prediction errors and/or model-free representations of expected value – such as ventral striatum, vmPFC, and even dopaminergic midbrain – also respond to their model-based analogs (Bromberg-Martin, Matsumoto, Hong, & Hikosaka, 2010; Daw, Gershman, Seymour, Dayan, & Dolan, 2011; Kishida et al., 2016; Wimmer, Daw, & Shohamy, 2012).”

What the sources say

• Not checked yetcites Bromberg-Martin, Matsumoto, Hong, & Hikosaka, 2010
• Not checked yetcites Daw, Gershman, Seymour, Dayan, & Dolan, 2011
• Not checked yetcites Kishida et al., 2016

In the paper

“Our second proposal is to re-unify model-free and model-based computations as being two different drivers of goal-directed (i.e., value-based) behavior, distinct from the class of value-free behaviors just described.”

In the paper

“More recent data, however, suggest that dopamine neurons likely carry model-based information as well.”

In the paper

“Habitual behavior, on the other hand, can be disturbed by perturbations to infralimbic cortex (Coutureau & Killcross, 2003) as well as the dorsolateral striatum (DLS; Yin et al., 2004, 2006).”

What the sources say

• Not checked yetcites Coutureau & Killcross, 2003
• Not checked yetcites Yin et al., 2004, 2006

In the paper

“our model suggests that disorders of habitual behavior (e.g., Obsessive-Compulsive Disorder, Tic Disorders) need not result from dysfunction in valuation (cf. Gillan & Robbins, 2014).”

What the sources say

• Not checked yetcites Gillan & Robbins, 2014

In the paper

“Activity in the putamen, a homologue of rodent DLS, has been found to track the behavioral development of habits (Tricomi, Balleine, & O’Doherty, 2009).”

What the sources say

• Not checked yetcites Tricomi, Balleine, & O’Doherty, 2009

In the paper

“In both of these cases, researchers have been unable to identify circuits that carry uniquely model-based value signals (Bornstein & Daw, 2011; Doll et al., 2012; D. Lee, Seo, & Jung, 2012; Shohamy, 2011).”

What the sources say

• Not checked yetcites Bornstein & Daw, 2011
• Not checked yetcites Doll et al., 2012
• Not checked yetcites D. Lee, Seo, & Jung, 2012

In the paper

“Collectively, these findings are at odds with the idea that the brain contains separable model-based and model-free systems.”

In the paper

“This latter proposal carries an additional benefit in that it obviates the need to cache value in a “common currency” (i.e., without reference to a specific outcome like juice or food type).”

In the paper

“This view is in tension, however, with the inflexibility of habits in the face of contingency omission: learning that an action which previously led to reinforcement now instead prevents reinforcement should be well within the capabilities of a model-free system.”

In the paper

“We have provided evidence that a value-free learning process -- according to which S-R associations are strengthened through action repetition in a Hebbian manner -- may be sufficient to generate behaviors that have been traditionally classified as habits, and held up in contrast to goal-directed behaviors.”

In the paper

“This lack of clear dissociation between model-based and model-free computations stands in stark contrast to the dissociations (described above) between circuits for goal-directed and for habitual control. This casts doubt on the idea that a one-to-one mapping exists between these two dichotomies, and motivates the search for alternative accounts (K. J. Miller, Shenhav, Pezzulo, & Ludvig, 2018).”

What the sources say

• Not checked yetcites K. J. Miller, Shenhav, Pezzulo, & Ludvig, 2018

In the paper

“Instead, they suggest that to the extent that model-free computations exist in the brain, they are intimately integrated with model-based control, consistent with some existing computational models (Gershman, Markman, & Otto, 2014; Ludvig, Mirian, Kehoe, & Sutton, 2017; Pezzulo, van der Meer, Lansink, & Pennartz, 2014; Silver, Sutton, & Müller, 2008; Sutton, 1990).”

What the sources say

• Not checked yetcites Gershman, Markman, & Otto, 2014
• Not checked yetcites Ludvig, Mirian, Kehoe, & Sutton, 2017
• Not checked yetcites Pezzulo, van der Meer, Lansink, & Pennartz, 2014

In the paper

“In contrast, the model of habits we are proposing completely cuts the tie between reinforcement learning and the habitual controller. Actions become habitized merely from use in a particular state, independent of any costs or benefits.”

In the paper

“This view provides an alternative explanation for the observation of habitized action sequences: when actions are typically performed in a particular order, the proprioceptive or other feedback associated with each action can become the “stimulus” that directly cues the subsequent action in the sequence (see James, 1890, chapter 4).”

What the sources say

• Not checked yetcites James, 1890

In the paper

“Collectively, we argue that these findings support a realignment of current computational models of decision-making, towards (re-)associating goal-directed/habitual with value-based/value-free rather than model-based/model-free.”