Habits without values.
Open accessPublished version of a preprint
Abstract
Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as arising from model-free reinforcement learning mechanisms, which typically select between available actions based on the future value expected to result from each. Traditionally, however, habits have been understood as behaviors that can be triggered directly by a stimulus, without requiring the animal to evaluate expected outcomes. Here, we develop a computational model instantiating this traditional view, in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes. We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the effects of reinforcement schedule on the rate of habit formation. The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors and help to better guide research into the neural mechanisms underlying control of instrumental behavior more generally. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
AI snapshot
Written from the paper’s full text.
TL;DR
The paper develops a computational model in which habits are value-free stimulus-response associations strengthened by repetition, rather than model-free reinforcement learning. It shows this model reproduces key habit phenomena, including insensitivity to outcome devaluation and contingency degradation, reinforcement-schedule effects on habit formation, lesion-like dissociations, and perseveration.
Key findings
- A value-free habitual controller, combined with a model-based goal-directed controller and an arbiter, reproduces overtraining-induced inflexibility: after longer training, simulated agents continued lever-pressing under omission and devaluation, whereas shorter training allowed reversal. Training duration was varied from 100 to 2000 trials in intervals of 100 for omission and from 10 to 2000 trials in intervals of 100 for devaluation.
- Simulated agents trained on variable-interval (VI6: pellets baited every 6 s, on average 10/min) lever-pressed at a moderate rate and habitized early, whereas agents trained on variable-ratio (VR10: 10% chance per press) lever-pressed at a high rate and habitized much later.
- Simulated lesions reproduced rodent striatal patterns: decreasing goal-directed parameters (DMS-like) produced a low press rate unaffected by omission or devaluation; decreasing habitual parameters (DLS-like) produced a high press rate that adapted to both manipulations. Control agents adapted after limited training (5,000 trials) but not after extensive training (15,000 trials).
- In a dynamic two-armed bandit task (10,000 trials; 50 model copies; reinforcer probabilities changed by independent Gaussian random walks, SD = 0.15), the model reproduced both reinforcement-seeking and value-free perseveration; replacing the habitual component with model-free RL failed to reproduce the perseverative pattern.
Methods
- Computational model with three modules: a habitual controller tracking recency-weighted action history \(H\), a goal-directed controller using model-based RL to learn separate reinforcer expectations \(R\) and utilities \(U\), and an arbiter weighting control by action-outcome contingency \(g\) and overall habitization \(h\).
- Simulated task environments included reversal learning, omission contingency, outcome devaluation, free-operant VR/VI schedules, simulated lesions, and a two-armed bandit. Most simulations used a single state with two actions; free-operant simulations used continuous press rates from 0 to 150 presses/min with function approximation and an effort cost quadratic in press rate.
- Analyses averaged simulated performance across agents and, for the two-armed bandit, used logistic regression quantifying influences of past actions, reinforcers, and their interaction on future choice.
Limitations
- The paper states that the model is described at an abstract computational level, limiting engagement with neurobiological detail, and that a neurobiologically detailed implementation is a critical next step.
- The paper states that the arbiter’s conditions require future research, that the model is agnostic about whether bypassing model-free RL entirely is appropriate, and that the form of value/motivational modulation of habit learning remains unresolved.
- Evident from the text: the findings come from simulations rather than new empirical experiments, and many simulations use simplified single-state, two-action environments with specific parameter and utility choices, so generality across richer environments is not established.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 99 cited sources checked · 0 supported · 0 flagged
- Findingown data
The computational model accounts for the effects of reinforcement schedule on the rate of habit formation.
“as well as the effects of reinforcement schedule on the rate of habit formation.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Adams, 1982
Not checked in the free run.
- cites Dickinson, 1985
Not checked in the free run.
- cites Gremel & Costa, 2013
Not checked in the free run.
- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
- Findingreasoning
The computational model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation.
“We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation”
Quote in the paperSubstantial claim- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
- Findingreasoning
The computational model explains perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system.
“The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system.”
Quote in the paperSubstantial claim- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
- Findingreasoning
The value-free habit mechanism accounts for key findings in the animal learning literature that dissociate habitual and goal-directed actions, namely contingency degradation and outcome devaluation.
“This value-free habit mechanism accounts for key findings in the animal learning literature that dissociate habitual and goal-directed actions, namely the tendency for an animal to continue performing a previously learned action when that action is no longer predictive of the reinforcing outcome (contingency degradation; Hammond, 1980) or when the predicted outcome ceases to be desired by the subj”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Hammond, 1980
Not checked in the free run.
- cites Adams & Dickinson, 1981
Not checked in the free run.
- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
- Findingreasoning
When the reversal occurs, the agent continues to select Action A for an extended period, past the point where the goal-directed controller has learned that Action B is more likely to be reinforced.
“As a result, when the reversal occurs, the agent continues to select Action A for an extended period, past the point where the goal-directed controller has learned that Action B is more likely to be reinforced (compare Figures 3B and 3E).”
Quote in the paperSubstantial claim
- Findingnone given
The model provides the first computational account of the difference in rate of habit formation under variable-interval and variable-ratio schedules of reinforcement.
“In addition, this model provides what is, to our knowledge, the first computational account of the difference in rate of habit formation under variable-interval and variable-ratio schedules of reinforcement (Adams, 1982; Dickinson, 1985; Gremel & Costa, 2013).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Adams, 1982
Not checked in the free run.
- cites Dickinson, 1985
Not checked in the free run.
- cites Gremel & Costa, 2013
Not checked in the free run.
- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
- Findingown data
In Simulation 1, initially behavior is driven by the goal-directed controller, which gradually learns the relative reinforcement rates and thus increasingly selects Action A.
“Initially, behavior is driven by the goal-directed controller, which gradually learns the relative reinforcement rates (Figure 3B) and thus increasingly selects Action A.”
Quote in the paperSubstantial claim - Findingreasoning
The model can capture all of the key diagnostic features of habitual behavior previously identified in animal behavior, including sensitivity to repetition frequency, reinforcement schedule, and selective modulation by lesions to one of two dissociable neural circuits.
“In the following sections we demonstrate that this model can capture all of the key diagnostic features of habitual behavior previously identified in animal behavior, including sensitivity to repetition frequency, reinforcement schedule, and selective modulation by lesions to one of two dissociable neural circuits.”
Quote in the paperSubstantial claim- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
- Findingown data
The model captures the observation that behavioral control in a novel environment tends to evolve from goal-directed to habitual (i.e., habits form from actions that were originally selected in a value-based manner).
“In addition to demonstrating the different kinds of learning that drive each of these controllers, this example demonstrates that our model captures the observation that behavioral control in a novel environment tends to evolve from goal-directed to habitual (i.e., habits form from actions that were originally selected in a value-based manner).”
Quote in the paperSubstantial claim - Findingown data
A control agent with intact habitual and goal-directed controllers adopted a high press rate that adapted to both manipulations when given limited training, but did not adapt to either following extensive training.
“A “control” agent, with intact habitual and goal-directed controllers, adopted a high press rate that adapted to both manipulations when given limited training, but did not adapt to either following extensive training.”
Quote in the paperSubstantial claim - Findingown data
In the model, lesioning the habitual controller led to a high press rate that adapted to both manipulations.
“whereas lesioning the habitual controller led to a high press rate that adapted to both manipulations (Figure 5).”
Quote in the paperSubstantial claim - Findingown data
The model was able to simultaneously reproduce both the reinforcement-seeking (value-based) and perseverative (value-free) components of the rats' behavior in the two-armed bandit task.
“We simulated performance in such an environment and found that the model was able to simultaneously reproduce both the reinforcement-seeking (value-based) and perseverative (value-free) components of these behaviors (Figure 7, right, blue points).”
Quote in the paperSubstantial claim - Findingreasoning
As habits strengthen, the habitual controller increasingly drives the choice of which action to select.
“As these habits strengthen, the habitual controller increasingly drives the choice of which action to select (Figure 3D).”
Quote in the paperSubstantial claim - Findingown data
With longer training sessions, the model failed to reverse its actions within the same time period.
“With longer training sessions, however, the model failed to reverse its actions within the same time period (Figure 3, left).”
Quote in the paperSubstantial claim - Findingreasoning
As the goal-directed controller selects Action A, the habitual controller strengthens its association between the current stimuli and Action A.
“As it does so, the habitual controller strengthens its association between the current stimuli and Action A (Figure 3C).”
Quote in the paperSubstantial claim - Findingown data
In the model, with experience in the initial environment, the agent learns to press the lever on a large fraction of trials.
“with experience in this environment the agent learns to press the lever on a large fraction of trials (Figure 4, blue curves).”
Quote in the paperSubstantial claim - Findingown data
When the agent was given a small number of training trials, it successfully learned to decrease probability of lever pressing following the reversal.
“When the agent was given a small number of training trials, it successfully learned to decrease probability of lever pressing following this reversal.”
Quote in the paperSubstantial claim - Findingown data
In the outcome devaluation simulation, the frequency of choosing to lever-press in the testing stage strongly depended on the duration of the training stage.
“We found that the frequency of choosing to lever-press in the testing stage strongly depended on the duration of the training stage (Figure 3, right)”
Quote in the paperSubstantial claim - Findingown data
Replacing the habitual component of the model with a model-free reinforcement learning system rendered it unable to reproduce the perseverative pattern.
“Replacing the habitual component of our model with a model-free reinforcement learning system rendered it unable to reproduce the perseverative pattern (Figure 7, right, red points).”
Quote in the paperSubstantial claim - Findingreasoning
A value-free habit mechanism explains a variety of other behavioral phenomena in which responses are facilitated by simple repetition (perseveration).
“Furthermore, a value-free habit mechanism explains a variety of other behavioral phenomena in which responses are facilitated by simple repetition (i.e. perseveration”
Quote in the paperSubstantial claim- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
- Findingown data
In the model, lesioning the goal-directed system produced a low press rate that was unaffected by either omission or devaluation.
“we found that lesioning the goal-directed system produced a low press rate that was unaffected by either omission or devaluation”
Quote in the paperSubstantial claim - Findingown data
Such a mechanism leads to perseveration of a previously higher-value action following contingency degradation or outcome devaluation.
“We demonstrate that such a mechanism leads to perseveration of a previously higher-value action following contingency degradation or outcome devaluation”
Quote in the paperSubstantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
- Findingown data
Such habitual behaviors are diminished by simulating lesions to a habitual system, consistent with classic findings in the animal behavior literature.
“We further show that such habitual behaviors are diminished by simulating lesions to a habitual system, consistent with classic findings in the animal behavior literature.”
Quote in the paperSubstantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
- Findingown data
Such a mechanism leads to increased perseveration of all actions in a probabilistic choice task with varying action-outcome contingencies.
“and increased perseveration of all actions in a probabilistic choice task with varying action-outcome contingencies.”
Quote in the paperSubstantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
- Findingreasoning
The system that generates these habitual behaviors does so without engaging in any manner of reinforcement learning (model-free or otherwise), consistent with theories that place habits outside the domain of RL.
“Crucially, the system that generates these habitual behaviors does so without engaging in any manner of reinforcement learning (model-free or otherwise), consistent with theories that place habits outside the domain of RL.”
Quote in the paperSubstantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
- Findingown data
The model can still capture key features of behavior for a task that is thought to rely on both model-free and model-based control.
“we also show that our model can still capture key features of behavior for a task that is thought to rely on both model-free and model-based control”
Quote in the paperSubstantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
- Interpretationnone given
Habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes.
“in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes.”
Quote in the paperSubstantial claim - Premisecitation
Habits form quickly when behavior is repeated with a high rate of consistency.
“or when behavior is repeated with a high rate of consistency (Lally, van Jaarsveld, Potts, & Wardle, 2010).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lally, van Jaarsveld, Potts, & Wardle, 2010
Not checked in the free run.
- Interpretationreasoning
The value-free formulation of habits aligns with Thorndike's Law of Exercise.
“Such a value-free formulation of habits aligns well with Thorndike’s second law, the “Law of Exercise.””
Quote in the paperNot a substantial claim - Premisecitation
Habits only arise in familiar contexts, typically developing out of behaviors initially undertaken under goal-directed control.
“First, habits only arise in familiar contexts, typically developing out of behaviors initially undertaken under goal-directed control (i.e., based on expected reward; Wood & Neal, 2007).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Wood & Neal, 2007
Not checked in the free run.
- Premisecitation
Habits tend to form most strongly in circumstances where actions are repeated very consistently.
“Second, habits tend to form most strongly in circumstances where actions are repeated very consistently (Dickinson, 1985; Wood & Rünger, 2016).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Dickinson, 1985
Not checked in the free run.
- cites Wood & Rünger, 2016
Not checked in the free run.
- Premisecitation
Behaviors become habitual (faster, more accurate, and less susceptible to interference) the more often those behaviors are performed in the presence of a particular set of cues.
“a large body of research has demonstrated that behaviors become habitual (i.e.., are faster, more accurate, and less susceptible to interference) the more often those behaviors are performed in the presence of a particular set of cues (reviewed in Wood & Neal, 2007; Wood & Runger, 2016).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Wood & Neal, 2007
Not checked in the free run.
- cites Wood & Runger, 2016
Not checked in the free run.
- Premisecitation
The value-free definition for habits drove the development of critical assays used to discriminate between habitual and goal-directed actions.
“The latter, value-free definition for habits drove the development of critical assays that have been used to discriminate between actions that are habitual versus goal-directed (i.e., outcome-sensitive), testing whether an animal continues to pursue a previous course of action when that action is no longer the most beneficial (Adams & Dickinson, 1981; Hammond, 1980).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Adams & Dickinson, 1981
Not checked in the free run.
- cites Hammond, 1980
Not checked in the free run.
- Interpretationreasoning
The value-free habit mechanism offers an alternative to model-free RL as a mechanism for habits.
“Here, we present a computational implementation of the Law of Exercise and show that it offers an alternative to model-free RL as a mechanism for habits, one that retains ideas about the nature of habits that have developed within other areas of psychology and neuroscience (Graybiel, 2008; James, 1890; Wood & Rünger, 2016).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Graybiel, 2008
Not checked in the free run.
- cites James, 1890
Not checked in the free run.
- cites Wood & Rünger, 2016
Not checked in the free run.
- Premisecitation
Habits form quickly when the environment is relatively unpredictable.
“and quickly when the environment is relatively unpredictable (Derusso et al., 2010)”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Derusso et al., 2010
Not checked in the free run.
- Premisecitation
Habits form slowly in environments where different behaviors lead to very different outcomes.
“Habits form slowly in environments where different behaviors lead to very different outcomes (Adams, 1982)”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Adams, 1982
Not checked in the free run.
- Premisecitation
Once a behavior has become a habit, that behavior is rendered inflexible with respect to changes in the environment, including those which make the behavior undesirable.
“Once a behavior has become a habit, that behavior is rendered inflexible with respect to changes in the environment, including those which make the behavior undesirable (Adams & Dickinson, 1981; Hammond, 1980).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Adams & Dickinson, 1981
Not checked in the free run.
- cites Hammond, 1980
Not checked in the free run.
- Premisecitation
Habits are driven by direct links between cues in the environment and the actions that have often followed those cues (stimulus-response associations).
“Habits, on the other hand, are understood to be driven by direct links between cues in the environment and the actions that have often followed those cues (i.e. “stimulus-response” associations).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Balleine & O’Doherty, 2010
Not checked in the free run.
- cites Dickinson, 1985
Not checked in the free run.
- cites Dolan & Dayan, 2013
Not checked in the free run.
- Premisecitation
Modern computational accounts typically model habits as mediated by reinforcement-learning mechanisms, which are outcome-sensitive.
“In contrast to this view, modern computational accounts typically model habits as mediated by reinforcement-learning mechanisms, which are outcome-sensitive (Figure 1, right).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Daw, Niv, & Dayan, 2005
Not checked in the free run.
- cites Dolan & Dayan, 2013
Not checked in the free run.
- cites Sutton & Barto, 1998
Not checked in the free run.
- Interpretationreasoning
Mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data.
“We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data.”
Quote in the paperSubstantial claim - Interpretationreasoning
Equating habit and model-free carries the assumption that habits are driven by a reward-maximization process.
“Equating these terms, however, carries a critical assumption: that habits are driven by a reward-maximization process (i.e., a process that depends directly on potential outcomes).”
Quote in the paperSubstantial claim - Interpretationreasoning
When combined with the idea that large action-outcome contingencies delay habit formation, the model explains the effect of reinforcement schedule on the rate of habit formation.
“When combined with another classic idea from the literature on habits – that large action-outcome contingencies delay habit formation – our model is able to explain another classic finding in the literature on habitual control: the effect of reinforcement schedule on the rate of habit formation.”
Quote in the paperSubstantial claim - Premisecitation
Actions followed by a constant probability of reinforcement (variable-ratio [VR] schedules) take a long time to habitize.
“Actions followed by a constant probability of reinforcement (variable-ratio [VR] schedules) take a long time to habitize (Adams, 1982).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Adams, 1982
Not checked in the free run.
- Interpretationreasoning
By replacing model-free RL with a value-free mechanism, the model forces a critical realignment of the prevailing framework equating habitual/goal-directed with model-free/model-based RL.
“By replacing model-free RL with a value-free mechanism, our model forces a critical realignment of this prevailing framework, thereby prompting a deeper consideration of how the computations and circuitry for model-free and model-based reinforcement learning might share more commonalities than differences.”
Quote in the paperNot a substantial claim - Premisecitation
When an action is baited at a constant probability per unit time, and only the first press following baiting results in reinforcement (variable-interval [VI] schedules), the action habitizes much more quickly, even when performance and overall rate of reinforcement are matched across VR and VI schedules.
“By contrast, when an action is “baited” at a constant probability per unit time, and only the first press following baiting results in reinforcement (variable-interval [VI] schedules), the action habitizes much more quickly, even when performance and overall rate of reinforcement are matched across these two reinforcement schedules (Dickinson, Nicholas, & Adams, 1983).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Dickinson, Nicholas, & Adams, 1983
Not checked in the free run.
- Interpretationown data
The constructed computational model, in which habits consist of value-free associations between stimuli and actions strengthened by repetition, reproduces key features of the behavioral literature on habits.
“We have constructed a computational model in which habits consist of value-free associations between stimuli and actions, and in which these associations are strengthened each time that action is performed in response to that stimulus. This model reproduces key features of the behavioral literature on habits.”
Quote in the paperSubstantial claim - Premisecitation
Overtrained animals will often continue to perform an action even when it no longer causes the desired outcome (contingency degradation).
“Overtrained animals will often continue to perform an action even when it no longer causes the desired outcome (indeed, even when it prevents the delivery of the outcome).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Dickinson, 1998
Not checked in the free run.
- Interpretationown data
Lesions to the model's goal-directed and habitual controllers reproduce behavioral patterns typical of DMS and DLS lesioned rats in classic instrumental conditioning experiments.
“Lesions to our model’s goal-directed and habitual controllers thus reproduce behavioral patterns typical of DMS and DLS lesioned rats, respectively, in classic experiments on instrumental conditioning.”
Quote in the paperSubstantial claim- Rests on an assumption: The model's goal-directed controller corresponds to the dorsomedial striatum (DMS) and its habitual controller corresponds to the dorsolateral striatum (DLS).
- Interpretationreasoning
In situations where goal-directed control consistently produces the same behavior in response to the same stimulus, that behavior is likely to become a habit.
“In situations where goal-directed control consistently produces the same behavior in response to the same stimulus, that behavior is likely to become a habit.”
Quote in the paperSubstantial claim - Premisecitation
In tasks where humans and animals make repeated decisions between similar alternatives, there is a near-universal tendency to select actions that have frequently been selected in the past, regardless of their outcome or of the task stimuli.
“In tasks where humans and animals make repeated decisions between similar alternatives, a near-universal observation is a tendency to select actions that have frequently been selected in the past, regardless of their outcome or of the task stimuli.”
Quote in the paperSubstantial claim - Interpretationown data
Overtraining caused the agent to perseverate on the habitized behavior (lever-pressing).
“indicating that overtraining caused the agent to perseverate on the habitized behavior (lever-pressing).”
Quote in the paperSubstantial claim - Premisecitation
The reinforcement schedule has a profound effect on habit formation.
“Another central finding from research on habitual control is that the reinforcement schedule has a profound effect on habit formation.”
Quote in the paperSubstantial claim - Interpretationown data
The difference in habitization between VI and VR schedules was largely driven by the difference in action-outcome contingencies inherent to each schedule: a small change in press rate resulted in a much larger change in reinforcement rate for VR relative to VI.
“This difference was largely driven by the difference in action-outcome contingencies inherent to each schedule: a small change in press rate resulted in a much larger change in reinforcement rate for a VR schedule relative to a VI schedule”
Quote in the paperSubstantial claim - Premisecitation
Perseverative behavior (continuing an action when it no longer causes the desired outcome) is diagnostic of habitual control.
“This perseverative behavior is diagnostic of habitual control (Dickinson, 1998).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Dickinson, 1998
Not checked in the free run.
- Interpretationnone given
Perseveration can be understood as a manifestation of habitual control.
“we turn to the ubiquitous and poorly understood phenomenon of perseveration, which we argue can be understood as a manifestation of habitual control.”
Quote in the paperSubstantial claim - Premisereasoning
Once a habit has formed, behavior can become inflexible in the face of changes to the environment that render it no longer desirable, such as contingency omission and outcome devaluation.
“Once a habit has formed, behavior can become inflexible in the face of changes to the environment that render it no longer desirable, such as contingency omission (in which the reinforcer that drove initial acquisition of the behavior is delivered only when the behavior is not performed) and outcome devaluation (in which the reinforcer is rendered no longer valuable to the subject).”
Quote in the paperSubstantial claim - Premisecitation
Lesions to the dorsolateral striatum (DLS) promote behavioral flexibility following overtraining.
“Lesions to regions of a putative “habit system,” such as the dorsolateral striatum (DLS), promote behavioral flexibility (i.e., alleviate perseveration) following overtraining (Graybiel, 2008; Yin & Knowlton, 2006).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Graybiel, 2008
Not checked in the free run.
- cites Yin & Knowlton, 2006
Not checked in the free run.
- Premisecitation
Overtrained animals will often continue performing an action that leads to an outcome they no longer desire (outcome devaluation).
“in this setup, overtrained animals will often continue performing an action that leads to an outcome they no longer desire (Adams, 1982).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Adams, 1982
Not checked in the free run.
- Premisecitation
In instructed task settings with human subjects, the speed and accuracy of an action are enhanced when that action has been recently performed.
“For instance, in instructed task settings with human subjects, the speed and accuracy of an action are enhanced when that action has been recently performed (Bertelson, 1965; Cho et al., 2002).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Bertelson, 1965
Not checked in the free run.
- cites Cho et al., 2002
Not checked in the free run.
- Interpretationcitation
The model's failure to reverse after longer training is consistent with animal learning data in which overtraining abolishes behavioral sensitivity to omission contingencies.
“This is consistent with data from animal learning, in which overtraining abolishes behavioral sensitivity to omission contingencies (Dickinson, 1998)”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Dickinson, 1998
Not checked in the free run.
- Premisecitation
Goal-directed behavior can be disturbed by perturbations to a network of interconnected brain regions including prelimbic cortex (PL), dorsomedial striatum (DMS), mediodorsal thalamus, basolateral amygdala, and orbitofrontal cortex (OFC).
“goal-directed behavior can be disturbed by perturbations to any of a network of interconnected brain regions, including prelimbic cortex (PL; Balleine & Dickinson, 1998; Corbit & Balleine, 2003; Killcross & Coutureau, 2003), dorsomedial striatum (DMS; Yin et al., 2005), mediodorsal thalamus (Corbit, Muir, & Balleine, 2003), basolateral amygdala (Balleine, Killcross, & Dickinson, 2003), and orbitof”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Balleine & Dickinson, 1998
Not checked in the free run.
- cites Yin et al., 2005
Not checked in the free run.
- cites Jones et al., 2012
Not checked in the free run.
- Premisecitation
Lesions to the dorsomedial striatum (DMS) impair flexibility.
“Conversely, lesions to regions of a putative “goal-directed system,” such as the dorsomedial striatum (DMS) impair flexibility (Yin, Ostlund, Knowlton, & Balleine, 2005).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Yin, Ostlund, Knowlton, & Balleine, 2005
Not checked in the free run.
- Premisecitation
Clean dissociations have largely evaded investigations into the neural substrates of model-based and model-free computations.
“In contrast to the literature on the habitual/goal-directed dichotomy, such clean dissociations have largely evaded investigations into the neural substrates of model-based and model-free computations, which can theoretically be differentiated in several ways (Doll et al., 2012).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Doll et al., 2012
Not checked in the free run.
- Interpretationreasoning
The authors' proposal avoids this tension by positing that habits are instantiated by entirely value-free mechanisms.
“Our proposal avoids this tension entirely by positing that habits are instantiated not by model-free RL, but by mechanisms that are entirely value-free.”
Quote in the paperSubstantial claim - Premisecitation
Regions that respond to hypothetical outcomes (a model-based construct) tend also to respond to actual outcomes.
“Rather, regions that respond to hypothetical outcomes (a model-based construct) – such as the OFC, vmPFC, and dorsal ACC – tend also to respond to actual outcomes (Abe, Seo, & Lee, 2011; Camille et al., 2004; Coricelli et al., 2005; Hayden, Pearson, & Platt, 2011; Lohrenz, McCabe, Camerer, & Montague, 2007; Rushworth, Noonan, Boorman, Walton, & Behrens, 2011; Strait, Blanchard, & Hayden, 2014).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Abe, Seo, & Lee, 2011
Not checked in the free run.
- cites Camille et al., 2004
Not checked in the free run.
- cites Coricelli et al., 2005
Not checked in the free run.
- Premisecitation
Regions that respond to model-free prediction errors and/or model-free representations of expected value also respond to their model-based analogs.
“Regions that respond to model-free prediction errors and/or model-free representations of expected value – such as ventral striatum, vmPFC, and even dopaminergic midbrain – also respond to their model-based analogs (Bromberg-Martin, Matsumoto, Hong, & Hikosaka, 2010; Daw, Gershman, Seymour, Dayan, & Dolan, 2011; Kishida et al., 2016; Wimmer, Daw, & Shohamy, 2012).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Bromberg-Martin, Matsumoto, Hong, & Hikosaka, 2010
Not checked in the free run.
- cites Daw, Gershman, Seymour, Dayan, & Dolan, 2011
Not checked in the free run.
- cites Kishida et al., 2016
Not checked in the free run.
- Interpretationreasoning
The proposed framework re-unifies model-free and model-based computations as two drivers of goal-directed behavior, distinct from value-free behaviors.
“Our second proposal is to re-unify model-free and model-based computations as being two different drivers of goal-directed (i.e., value-based) behavior, distinct from the class of value-free behaviors just described.”
Quote in the paperSubstantial claim - Premisecitation
Dopamine neurons likely carry model-based information as well.
“More recent data, however, suggest that dopamine neurons likely carry model-based information as well.”
Quote in the paperSubstantial claim - Premisecitation
Habitual behavior can be disturbed by perturbations to infralimbic cortex and the dorsolateral striatum (DLS).
“Habitual behavior, on the other hand, can be disturbed by perturbations to infralimbic cortex (Coutureau & Killcross, 2003) as well as the dorsolateral striatum (DLS; Yin et al., 2004, 2006).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Coutureau & Killcross, 2003
Not checked in the free run.
- cites Yin et al., 2004, 2006
Not checked in the free run.
- Interpretationreasoning
The model suggests that disorders of habitual behavior (e.g., Obsessive-Compulsive Disorder, Tic Disorders) need not result from dysfunction in valuation.
“our model suggests that disorders of habitual behavior (e.g., Obsessive-Compulsive Disorder, Tic Disorders) need not result from dysfunction in valuation (cf. Gillan & Robbins, 2014).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Gillan & Robbins, 2014
Not checked in the free run.
- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
- Premisecitation
Activity in the putamen, a homologue of rodent DLS, has been found to track the behavioral development of habits.
“Activity in the putamen, a homologue of rodent DLS, has been found to track the behavioral development of habits (Tricomi, Balleine, & O’Doherty, 2009).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Tricomi, Balleine, & O’Doherty, 2009
Not checked in the free run.
- Premisecitation
Researchers have been unable to identify circuits that carry uniquely model-based value signals.
“In both of these cases, researchers have been unable to identify circuits that carry uniquely model-based value signals (Bornstein & Daw, 2011; Doll et al., 2012; D. Lee, Seo, & Jung, 2012; Shohamy, 2011).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Bornstein & Daw, 2011
Not checked in the free run.
- cites Doll et al., 2012
Not checked in the free run.
- cites D. Lee, Seo, & Jung, 2012
Not checked in the free run.
- Interpretationown data
The findings are at odds with the idea that the brain contains separable model-based and model-free systems.
“Collectively, these findings are at odds with the idea that the brain contains separable model-based and model-free systems.”
Quote in the paperSubstantial claim - Interpretationreasoning
The realignment obviates the need to cache value in a common currency.
“This latter proposal carries an additional benefit in that it obviates the need to cache value in a “common currency” (i.e., without reference to a specific outcome like juice or food type).”
Quote in the paperSubstantial claim - Interpretationreasoning
The view that habits are model-free RL is in tension with the inflexibility of habits in the face of contingency omission.
“This view is in tension, however, with the inflexibility of habits in the face of contingency omission: learning that an action which previously led to reinforcement now instead prevents reinforcement should be well within the capabilities of a model-free system.”
Quote in the paperSubstantial claim - Interpretationown data
A value-free learning process, in which S-R associations are strengthened through action repetition in a Hebbian manner, may be sufficient to generate behaviors traditionally classified as habits and held up in contrast to goal-directed behaviors.
“We have provided evidence that a value-free learning process -- according to which S-R associations are strengthened through action repetition in a Hebbian manner -- may be sufficient to generate behaviors that have been traditionally classified as habits, and held up in contrast to goal-directed behaviors.”
Quote in the paperSubstantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
- Interpretationcitation
The lack of clear dissociation between model-based and model-free computations casts doubt on the idea that a one-to-one mapping exists between the goal-directed/habitual dichotomy and the model-based/model-free dichotomy.
“This lack of clear dissociation between model-based and model-free computations stands in stark contrast to the dissociations (described above) between circuits for goal-directed and for habitual control. This casts doubt on the idea that a one-to-one mapping exists between these two dichotomies, and motivates the search for alternative accounts (K. J. Miller, Shenhav, Pezzulo, & Ludvig, 2018).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites K. J. Miller, Shenhav, Pezzulo, & Ludvig, 2018
Not checked in the free run.
- Interpretationcitation
To the extent that model-free computations exist in the brain, they are intimately integrated with model-based control.
“Instead, they suggest that to the extent that model-free computations exist in the brain, they are intimately integrated with model-based control, consistent with some existing computational models (Gershman, Markman, & Otto, 2014; Ludvig, Mirian, Kehoe, & Sutton, 2017; Pezzulo, van der Meer, Lansink, & Pennartz, 2014; Silver, Sutton, & Müller, 2008; Sutton, 1990).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Gershman, Markman, & Otto, 2014
Not checked in the free run.
- cites Ludvig, Mirian, Kehoe, & Sutton, 2017
Not checked in the free run.
- cites Pezzulo, van der Meer, Lansink, & Pennartz, 2014
Not checked in the free run.
- Interpretationnone given
The authors' model of habits completely cuts the tie between reinforcement learning and the habitual controller; actions become habitized merely from use in a particular state, independent of costs or benefits.
“In contrast, the model of habits we are proposing completely cuts the tie between reinforcement learning and the habitual controller. Actions become habitized merely from use in a particular state, independent of any costs or benefits.”
Quote in the paperSubstantial claim - Interpretationcitation
The authors' view provides an alternative explanation for habitized action sequences: feedback associated with each action can become the stimulus that directly cues the subsequent action.
“This view provides an alternative explanation for the observation of habitized action sequences: when actions are typically performed in a particular order, the proprioceptive or other feedback associated with each action can become the “stimulus” that directly cues the subsequent action in the sequence (see James, 1890, chapter 4).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites James, 1890
Not checked in the free run.
- Interpretationreasoning
These findings support a realignment of current computational models of decision-making, towards (re-)associating goal-directed/habitual with value-based/value-free rather than model-based/model-free.
“Collectively, we argue that these findings support a realignment of current computational models of decision-making, towards (re-)associating goal-directed/habitual with value-based/value-free rather than model-based/model-free.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
- May combine two claims. 21
- Classified as probably not a substantial claim. 18
May combine two claims.
Findingown dataSimulated agents trained on VR schedules lever-pressed at a high rate and habitized much later in training.
“whereas agents trained on VR schedules lever-pressed at a high rate and habitized much later in training (Figure 4, right).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataSimulated agents trained on VI schedules lever-pressed at a moderate rate and habitized early in training.
“we found that simulated agents trained on VI schedules lever-pressed at a moderate rate and habitized early in training (Figure 4, left)”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenModel-free algorithms typically operate by learning the expected future reward associated with each possible action or state, relying crucially on these value representations.
“Model-free algorithms typically operate by learning the expected future reward associated with each possible action or state, relying crucially on these value representations.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationGoal-directed behaviors are driven by consideration of the outcomes they are likely to bring about (action-outcome representations).
“Goal-directed behaviors are understood to be driven by consideration of the outcomes that they are likely to bring about (i.e. “action-outcome” representations).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Balleine & O’Doherty, 2010
Not checked in the free run.
- cites Dickinson, 1985
Not checked in the free run.
- cites Dolan & Dayan, 2013
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThis category of value-free habits has been maintained in modern theorizing on habits, referred to as "direct cuing" of behavior, as distinct from value-based forms of habits ("motivated cueing").
“This category of value-free habits has been maintained in modern theorizing on habits, where it has been referred to as “direct cuing” of behavior, as distinct from value-based forms of habits (“motivated cueing”; Wood & Neal, 2007; Wood & Rünger, 2016).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Wood & Neal, 2007
Not checked in the free run.
- cites Wood & Rünger, 2016
Not checked in the free run.
Classified as probably not a substantial claim.
PremisecitationModel-free RL algorithms select actions based on cached values learned from previous experience.
“Modern reinforcement learning (RL) has elaborated this law into a set of computational algorithms, according to which actions are selected based on cached values learned from previous experience (Daw, Niv, & Dayan, 2005; Dolan & Dayan, 2013; Sutton & Barto, 1998).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Daw, Niv, & Dayan, 2005
Not checked in the free run.
- cites Dolan & Dayan, 2013
Not checked in the free run.
- cites Sutton & Barto, 1998
Not checked in the free run.
Classified as probably not a substantial claim.
PremisereasoningModel-free RL focuses only on potential rewards, ignoring all reward-unrelated elements of one's environment.
“This class of computations focuses only on potential rewards, ignoring all reward-unrelated elements of one’s environment (discussed below); it is therefore referred to as “model-free” RL.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationThe terms "habit" and "model-free" are now used interchangeably in much of the computational literature.
“This formulation for habits has become so prevalent that the terms “habit” and “model-free” are now used interchangeably in much of the computational literature (Dolan & Dayan, 2013; Doll, Simon, & Daw, 2012).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Dolan & Dayan, 2013
Not checked in the free run.
- cites Doll, Simon, & Daw, 2012
Not checked in the free run.
Classified as probably not a substantial claim.
PremisecitationThorndike's Law of Exercise holds that an action that has been taken often in the past is likely to be repeated in the future, independent of its past consequences.
“This law holds that an action that has been taken often in the past is likely to be repeated in the future, independent of its past consequences; in other words, it describes habits as a form of perseveration.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Thorndike, 1911
Not checked in the free run.
May combine two claims.
PremisecitationGoal-directed and habitual behaviors depend on different sets of computations and distinct underlying neural circuits.
“A large and growing body of work indicates that these behaviors depend on different sets of computations and distinct underlying neural circuits, suggesting that separable goal-directed and habitual systems implement fundamentally different strategies for the control of behavior”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Balleine & O’Doherty, 2010
Not checked in the free run.
- cites Dickinson, 1985
Not checked in the free run.
- cites Dolan & Dayan, 2013
Not checked in the free run.
May combine two claims.
Interpretationown dataValue-based mechanisms are not strictly necessary, and a value-free mechanism can account for the major behavioral phenomena that define habits.
“Here, we have shown computationally that such value-based mechanisms are not strictly necessary, and that a value-free mechanism can account for the major behavioral phenomena that define habits.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationSeparate goal-directed and habitual controllers exist in the brain, supported by distinct neural circuits, as strongly supported by lesion data from humans and other animals.
“The idea that separate goal-directed and habitual controllers exist in the brain, supported by distinct neural circuits, is strongly supported by lesion data from both humans and other animals.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationThe finding that VI schedules result in rapid habit formation has been widely replicated and represents a key element of the experimental toolkit for the study of habits.
“The finding that VI schedules result in rapid habit formation has been widely replicated and represents a key element of the experimental toolkit for the study of habits (Gremel & Costa, 2013; Tanaka, Balleine, & O’Doherty, 2008; Yin & Knowlton, 2006).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Gremel & Costa, 2013
Not checked in the free run.
- cites Tanaka, Balleine, & O’Doherty, 2008
Not checked in the free run.
- cites Yin & Knowlton, 2006
Not checked in the free run.
May combine two claims.
PremisereasoningHabits form slowly over time and often depend on behaviors that are initially taken under goal-directed control.
“The first of these features is that habits form slowly over time and often depend on behaviors that are initially taken under goal-directed control.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationExtensive training in a static environment (overtraining) can render behavior inflexible in the face of changes to that environment.
“We first test whether our model can capture a central finding from research on habits: that extensive training in a static environment (overtraining) can render behavior inflexible in the face of changes to that environment.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Dickinson, 1998
Not checked in the free run.
- cites Adams, 1982
Not checked in the free run.
May combine two claims.
Interpretationown dataThis example validates the predictive abilities of the model and highlights the importance of a value-free habitual controller in explaining habit-like behaviors that cannot be accounted for by a model-free RL-based algorithm alone.
“This example not only begins to validate the predictive abilities of our particular model, but also highlights the importance of a value-free habitual controller more generally in explaining habit-like behaviors that cannot otherwise be accounted for by a model-free RL-based algorithm alone.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationRelative to control rats, rats with lesions to DMS lever-press at a lower rate and are less able to decrease their press rate when reinforcers are omitted or devalued.
“relative to control rats, rats with lesions to DMS lever-press at a lower rate and are less able to decrease their press rate when reinforcers are omitted or devalued (Yin et al., 2005)”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Yin et al., 2005
Not checked in the free run.
May combine two claims.
PremisecitationRats with lesions to DLS lever-press at a similar rate to controls and are more successful than controls at adapting their press rate to omitted or devalued reinforcement.
“rats with lesions to DLS lever-press at a similar rate to controls and are more successful than controls at adapting their press rate to omitted or devalued reinforcement (Yin, Knowlton, & Balleine, 2004, 2006).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Yin, Knowlton, & Balleine, 2004, 2006
Not checked in the free run.
May combine two claims.
Interpretationnone givenThe computational account in which habits are value-free stimulus-response associations provides a closer match to classic psychological theories of habits, an account for classic behavioral data on habit formation, and a novel framework for understanding additional behavioral phenomena.
“This computational account in which habits are understood as value-free stimulus-response associations therefore provides a closer match to classic psychological theories of habits, an account for classic behavioral data on habit formation, and a novel framework for understanding additional behavioral phenomena.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationRats performing a dynamic two-armed bandit task exhibited behavioral patterns consistent with both reinforcement-seeking and choice perseveration.
“In one recent example, rats performing a dynamic two-armed bandit task exhibited behavioral patterns consistent with both reinforcement-seeking (i.e., being more likely to select a recently reinforced action), as well as with choice perseveration (i.e., being more likely to select a recently chosen action).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Miller et al., in prep
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationIn Daw et al. (2005), the extra computational complexity of the goal-directed system was a source of uncertainty, leading the arbiter to favor the habitual system in well-learned environments.
“The extra computational complexity associated with the goal-directed system was a source of uncertainty, leading the arbiter to favor the habitual system in well-learned environments.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Daw et al., 2005
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe arbiter in the model is similar to mechanisms in computational accounts that map habitual/goal-directed control onto model-free/model-based reinforcement learning.
“This arbiter is similar to mechanisms found in computational accounts mapping habitual/goal-directed control onto model-free/model-based reinforcement learning.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Daw et al., 2005
Not checked in the free run.
- cites S. W. Lee et al., 2014
Not checked in the free run.
- cites Keramati et al., 2011
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationA large body of computational work assumes that habitual control arises from model-free reinforcement learning algorithms.
“A large and influential body of computational work is built on the assumption that habitual control arises from model-free reinforcement learning algorithms (Dolan & Dayan, 2013; O’Doherty, Lee, & McNamee, 2015).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Dolan & Dayan, 2013
Not checked in the free run.
- cites O’Doherty, Lee, & McNamee, 2015
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationSubsequent accounts have developed arbiters that consider proxies for uncertainty or other advantages of habitual over goal-directed control, such as response time.
“Subsequent accounts have developed arbiters that consider proxies for this uncertainty (S. W. Lee et al., 2014), or other possible advantages of habitual over goal-directed control, such as response time (Keramati et al., 2011).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites S. W. Lee et al., 2014
Not checked in the free run.
- cites Keramati et al., 2011
Not checked in the free run.
May combine two claims.
PremisecitationIn rodents, goal-directed behavior results in greater activity in OFC and DMS, while habitual behavior results in greater activity in DLS.
“In rodents, goal-directed behavior results in greater activity in OFC and DMS, while habitual behavior results in greater activity in DLS (Gremel & Costa, 2013).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Gremel & Costa, 2013
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningThe value-free mechanisms proposed for habits do not preclude a role for value or motivational state (e.g., hunger or satiety) in habit learning.
“the value-free mechanisms we have proposed for habits by no means preclude a role for value or motivational state (e.g., hunger or satiety) in habit learning.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
InterpretationreasoningValue or motivational state may modulate the strengthening of habit associations either directly (e.g., through a feedback mechanism influencing the S-R association) or indirectly (e.g., by influencing action vigor, which increases associative strengths).
“These may modulate the strengthening of habit associations either directly (e.g., through a feedback mechanism that influences the S-R association) or indirectly (e.g., by influencing the vigor of an action, which in turn results in greater associative strengths).”
Quote in the paperNot a substantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
May combine two claims.
PremisecitationIn human subjects, goal-directed value signals during outcome devaluation have been identified in vmPFC, and similar signals during contingency degradation have been identified both in vmPFC and in the caudate nucleus.
“In human subjects, goal-directed value signals during outcome devaluation have been identified in vmPFC (Valentin, Dickinson, & O’Doherty, 2007), and similar signals during contingency degradation have been identified both in vmPFC and in the caudate nucleus, a homolog of DMS (Tanaka et al., 2008).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Valentin, Dickinson, & O’Doherty, 2007
Not checked in the free run.
- cites Tanaka et al., 2008
Not checked in the free run.
May combine two claims.
InterpretationreasoningThe authors' proposed framework re-maps habits to the perseveration kernel and provides an account of how that kernel might operate with a goal-directed controller.
“Our proposed framework re-maps habits to the perseveration kernel and provides an account of how that kernel might plausibly operate in tandem with a goal-directed controller so as to account for behaviors that have previously been described by RL models.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationcitationDopamine neurons are unlikely to play a role in a uniquely model-free control system, but instead have access to model-based information and play a role in model-based control.
“These data indicate that dopamine neurons are unlikely to play a role in a uniquely model-free control system, but instead have access to model-based information and play a role in model-based control.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningThe model-free component of some previous formalizations might not be necessary, and an elaboration of the perseveration kernel serves as a better model of habits.
“In effect, we are showing that the model-free component of some of these previous formalizations might not be necessary and that an elaboration of this perseveration kernel actually serves as a better model of habits.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationcitationConsiderable evidence supports the idea that anatomically separate goal-directed and habitual controllers exist in the brain, and that either controller can be responsible for a given action.
“In sum, considerable evidence supports the idea that anatomically separate goal-directed and habitual controllers exist in the brain, and that either controller can be responsible for a given action.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationHabits are promoted in situations where the contingencies between actions and outcomes are weak.
“habits are promoted in situations where the contingencies between actions and outcomes are weak (Dickinson, 1985).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Dickinson, 1985
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningThe proposed revised framework severs the tie between habits and model-free RL, defining value-free behaviors distinct from RL computations.
“We first propose severing the tie between habits and model-free RL, and instead defining a category of behaviors that are “value-free” and therefore distinct from either type of RL computation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationIn Daw et al. (2005), each system determined the uncertainty in its value estimates, and the arbiter selected the system with less overall uncertainty.
“In the initial formalization of this idea (Daw et al., 2005), each system determined the uncertainty in its value estimates, and the arbiter selected the system with less overall uncertainty.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Daw et al., 2005
Not checked in the free run.
May combine two claims.
InterpretationreasoningThe authors' model differs from corticostriatal models in the level of analysis, which limits neurobiological detail but facilitates engagement with a wide variety of behaviors.
“In addition to utilizing different mechanisms for value-based control and for arbitrating between the value-based and value-free controllers, our model also differs from these corticostriatal models in the level of analysis at which it is described. While this limits the level of detail with which our model can engage neurobiological data, it greatly facilitates engagement with a wide variety of”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Frank, 2015
Not checked in the free run.
- cites Frank & Badre, 2015
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningSuch a realignment may provide a more parsimonious account of previous behavioral results.
“Beyond providing a potentially more parsimonious account of previous behavioral results”
Quote in the paperNot a substantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
May combine two claims.
InterpretationreasoningSuch a realignment may offer a better account of extant neural findings, including the fact that structures associated with model-free and model-based computations (i.e., value-based computations) tend to overlap, whereas lesion/inactivation studies have revealed clear dissociations between structures associated with goal-directed behavior versus (potentially value-free) habits.
“such a realignment may offer a better account of extant neural findings, including the fact that structures associated with model-free and model-based computations (i.e., value-based computations) tend to overlap, whereas lesion/inactivation studies have revealed clear dissociations between structures associated with goal-directed behavior versus (potentially value-free) habits.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
The paper relies on this without support.
The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
“To illustrate the behavior of the model and the dynamics of its various internal variables, we simulated behavior in a probabilistic reversal learning task (Fig 3).”
The paper relies on this without support.
The model's goal-directed controller corresponds to the dorsomedial striatum (DMS) and its habitual controller corresponds to the dorsolateral striatum (DLS).
“We lesioned the goal-directed controller (DMS) or habitual controller (DLS) in our model”
“Lesions to our model’s goal-directed and habitual controllers thus reproduce behavioral patterns typical of DMS and DLS lesioned rats”
The authors state this as an assumption.
In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
“In our model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Two parallel forms of reinforcement learning, model-based and model-free (or value-free), correspond to goal-directed and habitual strategies and develop concurrently during behavior acquisition.
“Supporting this, computational studies have proposed two parallel forms of reinforcement learning, model-based and model-free (or value-free) learning corresponding to goal-directed and habitual strategies, respectively, that develop concurrently during behavior acquisition2,41,42.”
From Dissociable roles of prefrontal plasticity in decision-making strategy and execution of habitual behavior (2026) · cites this paper as [42]
The Results describe parallel goal-directed and habitual controllers whose learning develops together during a task; the Discussion describes an earlier framework pairing model-based learning with goal-directed behavior and model-free learning with habits. But this paper’s own model replaces the model-free habitual controller with value-free perseveration, which it says is not reinforcement learning. Thus “model-free (or value-free)” incorrectly treats the two as interchangeable. Page numbers were not supplied. (LLM, conf=0.89)
Experimental — accuracy not yet measuredThe habitual system simplifies decision-making by shifting to a stimulus-response mode of behavior, making decisions potentially value-less and inflexible but less cognitively demanding.
“With overtraining, the habitual system simplifies decision-making by shifting to a stimulus-response mode of behavior51, making decisions potentially value-less and inflexible but less cognitively demanding52.”
From Revealing abrupt transitions from goal-directed to habitual behavior (2026) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Training reinforcement probability differs between Methods and Results
MajorThe Methods specify a 25% pellet probability during omission training, and say devaluation training was identical. The Results describe a 50% pellet probability for both training phases.
“In the first phase, performance of one action (Press Lever) resulted in a reinforcer of one type (Pellet) 25% of the time”
“The agent was initially trained in an environment where performance of Action A (“press lever”) was followed by a reinforcer of one type (“food pellet”) 50% of the time”
Habit-weight update uses a different basis function
MinorThe habit-strength function uses basis functions γi, so its derivative with respect to weight ci is γi. The stated weight-update equation instead substitutes χi, which is not the defined habit basis function.
“=ct(i)+αH⋅∑a′((δa′,at−H(a′))⋅χi(a′))”
“11Ht(a)=∑i=130ct(i)⋅γi(a)γi(a)=e−(a−5i)22⋅52”
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
James, W. (1891) · Henry Holt and Company eBooks · doi:10.1037/11059-000 · Cited by 25,799 · see citing works
An Integrative Theory of Prefrontal Cortex Function
Miller, E. K., & Cohen, J. D. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.167 · Cited by 12,803 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995) · Psychological Review · doi:10.1037/0033-295x.102.3.419 · Cited by 5,256 · see citing works
The principles of psychology, Vol I.
James, W. (1890) · doi:10.1037/10538-000 · Cited by 4,514 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
The role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402 · see citing works
How are habits formed: modelling habit formation in the real world
Lally, P., van Jaarsveld, C. H. M., Potts, H. W. W., & Wardle, J. (2010) · European Journal of Social Psychology · doi:10.1002/ejsp.674 · Cited by 2,036 · see citing works
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779 · see citing works
Balleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775 · see citing works
Principles of Behavior. An Introduction to Behavior Theory
N., E., & Hull, C. L. (1943) · The Journal of Philosophy · doi:10.2307/2019960 · Cited by 1,731 · see citing works
Habits, Rituals, and the Evaluative Brain
Graybiel, A. M. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.112851 · Cited by 1,697 · see citing works
Goal-directed instrumental action: contingency and incentive learning and their cortical substrates
Balleine, B. W., & Dickinson, A. (1998) · Neuropharmacology · doi:10.1016/s0028-3908(98)00033-1 · Cited by 1,541 · see citing works
A new look at habits and the habit-goal interface.
Wood, W., & Neal, D. T. (2007) · Psychological Review · doi:10.1037/0033-295x.114.4.843 · Cited by 1,470 · see citing works
Wood, W., & Rünger, D. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033417 · Cited by 1,445 · cited 9× · see citing works
“This prediction also provides a mechanistic underpinning for the repetition strategies that have shown to be effective at improving workplace and health-related performance through habit formation (Gardner, Lally, & Wardle, 2012; Lally et al., 2010; Wood & Rünger, 2016).”
BackgroundMethodInfluentialSutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,348 · see citing works
Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03095.x · Cited by 1,226 · see citing works
Actions and habits: The development of behavioural autonomy
Dickinson, A. (1985) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1985.0010 · Cited by 1,217 · see citing works
A neuropsychological theory of multiple systems in category learning.
Ashby, F. G., Alfonso-Reese, L. A., Turken, A. U., & Waldron, E. M. (1998) · Psychological Review · doi:10.1037/0033-295x.105.3.442 · Cited by 1,216 · see citing works
Frontal cortex and reward-guided learning and decision-making
Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.014 · Cited by 1,111 · see citing works
Animal intelligence; experimental studies
Thorndike, E. L. (1911) · doi:10.5962/bhl.title.55072 · Cited by 1,100 · see citing works
The role of the dorsomedial striatum in instrumental conditioning
Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2005.04218.x · Cited by 1,099 · see citing works
O’Reilly, R. C., & Frank, M. J. (2005) · Neural Computation · doi:10.1162/089976606775093909 · Cited by 1,066 · see citing works
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977 · see citing works
Predicting Behavior From Actions in the Past: Repeated Decision Making or a Matter of Habit?
Aarts, H., Verplanken, B., & van Knippenberg, A. (1998) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.1998.tb01681.x · Cited by 926 · see citing works
Animal Intelligence. Experimental Studies
Thorndike, E. L., & Jelliffe (1912) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-191205000-00016 · Cited by 824 · see citing works
A specific role for posterior dorsolateral striatum in human habit learning
Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06796.x · Cited by 769 · see citing works
Coordination of Actions and Habits in the Medial Prefrontal Cortex of Rats
Killcross, S. (2003) · Cerebral Cortex · doi:10.1093/cercor/13.4.400 · Cited by 748 · see citing works
Conservation of hippocampal memory function in rats and humans
Bunsey, M. D., & Eichenbaum, H. (1996) · Nature · doi:10.1038/379255a0 · Cited by 733 · see citing works
The Involvement of the Orbitofrontal Cortex in the Experience of Regret
Camille, N., Coricelli, G., Sallet, J., Pradat-Diehl, P., Duhamel, J.-R., & Sirigu, A. (2004) · Science · doi:10.1126/science.1094550 · Cited by 716 · see citing works
Instrumental Responding following Reinforcer Devaluation
Adams, C. D., & Dickinson, A. (1981) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748108400816 · Cited by 687 · see citing works
Regret and its avoidance: a neuroimaging study of choice behavior
Coricelli, G., Critchley, H., Joffily, M., O’Doherty, J. P., Angela Sirigu, B., & Dolan, R. J. (2005) · Nature Neuroscience · doi:10.1038/nn1514 · Cited by 673 · see citing works
Active Inference, homeostatic regulation and adaptive behavioural control
Pezzulo, G., Rigoli, F., & Friston, K. (2015) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2015.09.001 · Cited by 650 · see citing works
Neural computations underlying arbitration between model-based and model-free learning
Lee, S. W., Shimojo, S., & O’Doherty, J. P. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.028 · Cited by 642 · see citing works
Variations in the Sensitivity of Instrumental Responding to Reinforcer Devaluation
Adams, C. D. (1982) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748208400878 · Cited by 635 · see citing works
Gremel, C. M., & Costa, R. M. (2013) · Nature Communications · doi:10.1038/ncomms3264 · Cited by 618 · see citing works
The hippocampus and memory for orderly stimulus relations
Dusek, J. A., & Eichenbaum, H. (1997) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.94.13.7109 · Cited by 610 · see citing works
Making health habitual: the psychology of ‘habit-formation’ and general practice
Gardner, B., Lally, P., & Wardle, J. (2012) · British Journal of General Practice · doi:10.3399/bjgp12x659466 · Cited by 591 · cited 1× · see citing works
“This prediction also provides a mechanistic underpinning for the repetition strategies that have shown to be effective at improving workplace and health-related performance through habit formation (Gardner, Lally, & Wardle, 2012; Lally et al., 2010; Wood & Rünger, 2016).”
BackgroundCharacterizing a psychiatric symptom dimension related to deficits in goal-directed control
Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A., & Daw, N. D. (2016) · eLife · doi:10.7554/elife.11305 · Cited by 586 · see citing works
Neural Basis of Reinforcement Learning and Decision Making
Lee, D., Seo, H., & Jung, M. W. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150512 · Cited by 542 · see citing works
YIN, H., KNOWLTON, B., & BALLEINE, B. (2006) · Behavioural Brain Research · doi:10.1016/j.bbr.2005.07.012 · Cited by 532 · see citing works
Determining the neural substrates of goal-directed learning in the human brain
Valentin, V. V., Dickinson, A., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0564-07.2007 · Cited by 529 · see citing works
Neuronal basis of sequential foraging decisions in a patchy environment
Hayden, B. Y., Pearson, J. M., & Platt, M. L. (2011) · Nature Neuroscience · doi:10.1038/nn.2856 · Cited by 513 · see citing works
Learning and selective attention
Dayan, P., Kakade, S., & Montague, P. R. (2000) · Nature Neuroscience · doi:10.1038/81504 · Cited by 495 · see citing works
DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS
Lau, B., & Glimcher, P. W. (2005) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.2005.110-04 · Cited by 485 · see citing works
Cortical and basal ganglia contributions to habit learning and automaticity
Ashby, F. G., Turner, B. O., & Horvitz, J. C. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.02.001 · Cited by 481 · see citing works
Cushman, F. A. (2013) · Personality and Social Psychology Review · doi:10.1177/1088868313495594 · Cited by 472 · see citing works
Redish, A. D. (2016) · Nature reviews. Neuroscience · doi:10.1038/nrn.2015.30 · Cited by 462 · see citing works
The role of prelimbic cortex in instrumental conditioning
Corbit, L. H., & Balleine, B. W. (2003) · Behavioural Brain Research · doi:10.1016/j.bbr.2003.09.023 · Cited by 423 · see citing works
The ubiquity of model-based reinforcement learning
Doll, B. B., Simon, D. A., & Daw, N. D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.08.003 · Cited by 380 · see citing works
The effect of lesions of the basolateral amygdala on instrumental conditioning
Balleine, B. W., Killcross, A. S., & Dickinson, A. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-02-00666.2003 · Cited by 355 · see citing works
Otto (2013) · doi:10.1177/0956797612463080 · Cited by 352 · see citing works
Goal-directed learning and obsessive-compulsive disorder
Gillan, C. M., & Robbins, T. W. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0475 · Cited by 348 · see citing works
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · see citing works
Redish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346 · see citing works
Coutureau, E., & Killcross, S. (2003) · Behavioural Brain Research · doi:10.1016/j.bbr.2003.09.025 · Cited by 342 · see citing works
The effect of the instrumental training contingency on susceptibility to reinforcer devaluation
Dickinson, A., Nicholas, D. J., & Adams, C. D. (1983) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748308400912 · Cited by 338 · see citing works
Orbitofrontal Cortex Supports Behavior and Learning Using Inferred But Not Cached Values
Jones, J. L., Esber, G. R., McDannald, M. A., Gruber, A. J., Hernández, A., Mirenzi, A., & Schoenbaum, G. M. (2012) · Science · doi:10.1126/science.1227489 · Cited by 332 · see citing works
Contributions of the striatum to learning, motivation, and performance: an associative account
Liljeholm, M., & O’Doherty, J. P. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2012.07.007 · Cited by 317 · see citing works
Mapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310 · see citing works
Habits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297 · see citing works
Covert expectation-of-reward in rat ventral striatum at decision points
van der Meer, M. A. A. (2009) · Frontiers in Integrative Neuroscience · doi:10.3389/neuro.07.001.2009 · Cited by 296 · see citing works
Reward value comparison via mutual inhibition in ventromedial prefrontal cortex
Strait, C. E., Blanchard, T. C., & Hayden, B. Y. (2014) · Neuron · doi:10.1016/j.neuron.2014.04.032 · Cited by 294 · see citing works
THE EFFECT OF CONTINGENCY UPON THE APPETITIVE CONDITIONING OF FREE‐OPERANT BEHAVIOR
Hammond, L. J. (1980) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1980.34-297 · Cited by 291 · see citing works
Validation of Decision-Making Models and Analysis of Decision Variables in the Rat Basal Ganglia
Ito, M., & Doya, K. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.6157-08.2009 · Cited by 287 · see citing works
A neurobiological theory of automaticity in perceptual categorization
Ashby, F. G., Ennis, J. M., & Spiering, B. J. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.632 · Cited by 286 · see citing works
Dopamine transients are sufficient and necessary for acquisition of model-based associations
Sharpe, M. J., Chang, C. Y., Liu, M. A., Batchelor, H. M., Mueller, L. E., Jones, J. L., Niv, Y., & Schoenbaum, G. M. (2017) · Nature Neuroscience · doi:10.1038/nn.4538 · Cited by 284 · see citing works
Model-based learning protects against forming habits
Gillan, C. M., Otto, A. R., Phelps, E. A., & Daw, N. D. (2015) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-015-0347-6 · Cited by 283 · see citing works
Crockett, M. J. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.06.005 · Cited by 274 · see citing works
Calculating Consequences: Brain Systems That Encode the Causal Effects of Actions
Tanaka, S. C., Balleine, B. W., & O'Doherty, J. P. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1808-08.2008 · Cited by 268 · see citing works
Neural signature of fictive learning signals in a sequential investment task
Lohrenz, T. M., McCabe, K., Camerer, C. F., & Montague, P. R. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0608842104 · Cited by 263 · see citing works
Rutledge, R. B., Lazzaro, S. C., Lau, B., Myers, C. E., Gluck, M. A., & Glimcher, P. W. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.3524-09.2009 · Cited by 255 · see citing works
Regulation of dietary choice by the decision-making circuitry
Rangel, A. (2013) · Nature Neuroscience · doi:10.1038/nn.3561 · Cited by 246 · see citing works
Testing the Reward Prediction Error Hypothesis with an Axiomatic Model
Rutledge, R. B., Dean, M., Caplin, A., & Glimcher, P. W. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.1747-10.2010 · Cited by 244 · see citing works
Smittenaar, P., FitzGerald, T. H. B., Romei, V., Wright, N. D., & Dolan, R. J. (2013) · Neuron · doi:10.1016/j.neuron.2013.08.009 · Cited by 244 · see citing works
Dorsal hippocampus contributes to model-based planning
Miller, K. J., Botvinick, M., & Brody, C. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4613 · Cited by 243 · see citing works
Self-citationDoes the orbitofrontal cortex signal value?
Schoenbaum, G., Takahashi, Y., Liu, T., & McDannald, M. A. (2011) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.06210.x · Cited by 232 · see citing works
Animal Intelligence: Experimental Studies
Brown, W. M. (1912) · Nature · doi:10.1038/088306a0 · Cited by 230 · see citing works
Role of Striatum in Updating Values of Chosen Actions
Kim, H., Sul, J. H., Huh, N., Lee, D., & Jung, M. W. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.2728-09.2009 · Cited by 225 · see citing works
Internally generated sequences in learning and executing goal-directed behavior
Pezzulo, G., van der Meer, M. A. A., Lansink, C. S., & Pennartz, C. M. A. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.06.011 · Cited by 217 · see citing works
Retrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217 · see citing works
Kishida, K. T., Sáez, I., Lohrenz, T. M., Witcher, M. R., Laxton, A. W., Tatter, S. B., White, J. P., Ellis, T. L., Phillips, P. E. M., & Montague, P. R. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1513619112 · Cited by 216 · see citing works
McDannald, M. A., Lucantonio, F., Burke, K. A., Niv, Y., & Schoenbaum, G. M. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.5499-10.2011 · Cited by 215 · see citing works
Serial Choice Reaction-time as a Function of Response versus Signal-and-Response Repetition
Bertelson, P. (1965) · Nature · doi:10.1038/206217a0 · Cited by 207 · see citing works
Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum
Keramati, M., Smittenaar, P., Dolan, R. J., & Dayan, P. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1609094113 · Cited by 203 · see citing works
Dopamine neurons respond to errors in the prediction of sensory features of expected rewards
Takahashi, Y. K., Batchelor, H. M., Liu, B., Khanna, A., Morales, M., & Schoenbaum, G. (2017) · Neuron · doi:10.1016/j.neuron.2017.08.025 · Cited by 200 · see citing works
Autonomous Mechanism of Internal Choice Estimate Underlies Decision Inertia
Akaishi, R., Umeda, K., Nagase, A. M., & Sakai, K. (2013) · Neuron · doi:10.1016/j.neuron.2013.10.018 · Cited by 188 · see citing works
Corbit, L. H., Muir, J. L., & Balleine, B. W. (2003) · European Journal of Neuroscience · doi:10.1046/j.1460-9568.2003.02833.x · Cited by 186 · see citing works
Cognitive Control Predicts Use of Model-based Reinforcement Learning
Otto, A. R., Skatova, A., Madlon‐Kay, S., & Daw, N. D. (2014) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00709 · Cited by 186 · see citing works
A Pallidus-Habenula-Dopamine Pathway Signals Inferred Stimulus Values
Bromberg-Martin, E. S., Matsumoto, M., Hong, S., & Hikosaka, O. (2010) · Journal of Neurophysiology · doi:10.1152/jn.00158.2010 · Cited by 182 · see citing works
Solway, A., & Botvinick, M. (2012) · Psychological Review · doi:10.1037/a0026435 · Cited by 180 · see citing works
Learning robust cortico-cortical associations with the basal ganglia: An integrative review
Hélie, S., Ell, S. W., & Ashby, F. G. (2014) · Cortex · doi:10.1016/j.cortex.2014.10.011 · Cited by 172 · see citing works
Model-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161 · see citing works
Neuronal Origins of Choice Variability in Economic Decisions
Padoa‐Schioppa, C. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.013 · Cited by 157 · see citing works
Social norms, self-control, and the value of antisocial behavior
Buckholtz, J. W. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.03.004 · Cited by 145 · see citing works
Social Norms Shift Behavioral and Neural Responses to Foods
Nook, E. C., & Zaki, J. (2015) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00795 · Cited by 145 · see citing works
Cho, R. Y., Nystrom, L. E., Brown, E., Jones, A. D., Braver, T. S., Holmes, P. J., & COHEN, J. D. (2002) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.2.4.283 · Cited by 140 · see citing works
Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. M. (2016) · eLife · doi:10.7554/elife.13665 · Cited by 137 · see citing works
Dopamine selectively remediates ‘model-based’ reward learning: a computational approach
Sharp, M. E., Foerde, K., Daw, N. D., & Daphna Shohamy, A. (2015) · Brain · doi:10.1093/brain/awv347 · Cited by 129 · see citing works
O’Doherty, J. P. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.03.027 · Cited by 125 · see citing works
Reber, J., Feinstein, J. S., O’Doherty, J. P., Liljeholm, M., Adolphs, R., & Tranel, D. (2017) · Brain · doi:10.1093/brain/awx105 · Cited by 125 · see citing works
Learning and decision making in monkeys during a rock–paper–scissors game
Lee, D., McGreevy, B. P., & Barraclough, D. J. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2005.07.003 · Cited by 123 · see citing works
Gold, J. I., Law, C.-T., Connolly, P. M., & Bennur, S. (2008) · Journal of Neurophysiology · doi:10.1152/jn.90629.2008 · Cited by 123 · see citing works
Generalization of value in reinforcement learning by humans
Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08017.x · Cited by 111 · see citing works
Instrumental uncertainty as a determinant of behavior under interval schedules of reinforcement
DeRusso, A. L. (2010) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2010.00017 · Cited by 111 · see citing works
Sample-based learning and search with permanent and transient memories
Silver, D., Sutton, R. S., & Müller, M. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390278 · Cited by 110 · see citing works
Learning and motivation in the human striatum
Daphna Shohamy, A. (2011) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2011.05.009 · Cited by 100 · see citing works
The structure of reinforcement-learning mechanisms in the human brain
O’Doherty, J. P., Lee, S. W., & McNamee, D. (2014) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2014.10.004 · Cited by 100 · see citing works
A theoretical account of cognitive effects in delay discounting
Kurth-Nelson, Z. L., Bickel, W. K., & Redish, A. D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08058.x · Cited by 99 · see citing works
Multiplicity of control in the basal ganglia: computational roles of striatal subregions
Bornstein, A. M., & Daw, N. D. (2011) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2011.02.009 · Cited by 97 · see citing works
Bornstein, A. M., & Daw, N. D. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003387 · Cited by 89 · see citing works
Omission Learning after Instrumental Pretraining
Dickinson, A. (2010) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/713932679 · Cited by 88 · see citing works
Hierarchical Action Control: Adaptive Collaboration Between Actions and Habits
Balleine, B. W., & Dezfouli, A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02735 · Cited by 88 · see citing works
Scott, B. B., Constantinople, C. M., Erlich, J. C., Tank, D. W., & Brody, C. D. (2015) · eLife · doi:10.7554/elife.11308 · Cited by 82 · see citing works
Lucantonio, F., Caprioli, D., & Schoenbaum, G. M. (2013) · Neuropharmacology · doi:10.1016/j.neuropharm.2013.05.033 · Cited by 76 · see citing works
Goal-Directed Decision Making with Spiking Neurons
Friedrich, J. H., & Lengyel, M. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.2854-15.2016 · Cited by 57 · see citing works
Coherency-maximizing exploration in the supermarket
Riefer, P. S., Prior, R., Blair, N., Pavey, G., & Love, B. C. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0017 · Cited by 53 · see citing works
The Law of Recency: An Episodic Stimulus-Response Retrieval Account of Habit Acquisition
Giesen, C. G., Schmidt, J. R., & Rothermund, K. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02927 · Cited by 53 · see citing works
Caching mechanisms for habit formation in Active Inference
Maisto, D., Friston, K. J., & Pezzulo, G. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.05.083 · Cited by 46 · see citing works
How cognitive theory guides neuroscience
Frank, M. J., & Badre, D. (2014) · Cognition · doi:10.1016/j.cognition.2014.11.009 · Cited by 42 · see citing works
Realigning Models of Habitual and Goal-Directed Decision-Making
Miller, K. J., Ludvig, E. A., Pezzulo, G., & Shenhav, A. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-812098-9.00018-8 · Cited by 41 · see citing works
Self-citationNeurons in the Nucleus Accumbens Promote Selection Bias for Nearer Objects
Morrison, S. E., & Nicola, S. M. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.2197-14.2014 · Cited by 32 · see citing works
Jung, D., & Dorner, V. (2017) · Lecture notes in information systems and organisation · doi:10.1007/978-3-319-67431-5_18 · Cited by 29 · see citing works
Balcarras, M., Ardid, S., Kaping, D., Everling, S., & Womelsdorf, T. (2015) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00894 · Cited by 25 · see citing works
Model-Free RL or Action Sequences?
Morris, A., & Cushman, F. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02892 · Cited by 25 · see citing works
Akam, T. E., Rodrigues-Vaz, I., Marcelo, I., Zhang, X., Pereira, M., Oliveira, R. F., Dayan, P., & Costa, R. M. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/126292 · Cited by 23 · see citing works
Linking Across Levels of Computation in Model-Based Cognitive Neuroscience
Frank, M. J. (2015) · Springer eBooks · doi:10.1007/978-1-4939-2236-9_8 · Cited by 22 · see citing works
Associative Learning from Replayed ExperiencePreprint
Ludvig, E. A., Mirian, M. S., Kehoe, E. J., & Sutton, R. S. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/100800 · Cited by 20 · see citing works
Self-citationA Neurocomputational Model of Automatic Sequence Production
Hélie, S., Roeder, J. L., Vucovich, L. E., Rünger, D., & Ashby, F. G. (2015) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00794 · Cited by 12 · see citing works
The prefrontal cortex and hybrid learning during iterative competitive games
Abe, H., Seo, H., & Lee, D. (2011) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.06223.x · Cited by 11 · see citing works
O’Reilly, R. C., Nair, A., Russin, J., & Herd, S. A. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.00380 · Cited by 11 · see citing works
Time course of a repetition effect on saccadic reaction time in non-human primates.
Gore, J. L., Dorris, M. C., & Munoz, D. P. (2002) · PubMed · Cited by 10 · see citing works
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. M. (2016) · doi:10.7554/elife.13665.010 · Cited by 4 · see citing works
Dual Competition between the Basal Ganglia and the Cortex: from Action-Outcome to Stimulus-ResponsePreprint
Topalidou, M., Kase, D., Boraud, T., & Rougier, N. P. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/187294 · Cited by 3 · see citing works
Flagged first, then by citations
1 of the 136 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Binding and Retrieval in Action Control (BRAC)
Frings, C., Hommel, B., Koch, I., Rothermund, K., Dignath, D., Giesen, C. G., Kiesel, A., Kunde, W., Mayr, S., Moeller, B., et al. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.02.004 · Cited by 376 · see citing works
Motivation and cognitive control in depression
Grahek, I., Shenhav, A., Musslick, S., Krebs, R. M., & Koster, E. H. W. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.04.011 · Cited by 361 · see citing works
Self-citationResolving uncertainty in a social world
FeldmanHall, O., & Shenhav, A. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0590-x · Cited by 276 · see citing works
Self-citationThe neural and computational systems of social learning
Olsson, A., Knapska, E., & Lindström, B. (2020) · Nature reviews. Neuroscience · doi:10.1038/s41583-020-0276-4 · Cited by 267 · see citing works
Habitual Behavior Is Goal-Driven
Kruglanski, A. W., & Szumowska, E. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620917676 · Cited by 217 · see citing works
Pro-environmental habits: An underexplored research agenda in sustainability science
Linder, N., Giusti, M., Samuelsson, K., & Barthel, S. (2021) · AMBIO · doi:10.1007/s13280-021-01619-6 · Cited by 184 · see citing works
Humans primarily use model-based inference in the two-stage task
Feher da Silva, C., & Hare, T. A. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0905-y · Cited by 153 · see citing works
Habits and Goals in Human Behavior: Separate but Interacting Systems
Wood, W., Mazar, A., & Neal, D. T. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691621994226 · Cited by 154 · see citing works
Beyond dichotomies in reinforcement learning
Collins, A. G. E., & Cockburn, J. (2020) · Nature reviews. Neuroscience · doi:10.1038/s41583-020-0355-6 · Cited by 128 · see citing works
The computational roots of positivity and confirmation biases in reinforcement learning
Palminteri, S., & Lebreton, M. P. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.04.005 · Cited by 127 · see citing works
Ho, M. K., Saxe, R. R., & Cushman, F. A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.08.003 · Cited by 123 · see citing works
The case against economic values in the orbitofrontal cortex (or anywhere else in the brain).
Hayden, B. Y., & Niv, Y. (2021) · Behavioral Neuroscience · doi:10.1037/bne0000448 · Cited by 127 · see citing works
Sharpe, M. J., Stalnaker, T. A., Schuck, N. W., Killcross, S., Schoenbaum, G. M., & Niv, Y. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102824 · Cited by 107 · see citing works
Advances in modeling learning and decision-making in neuroscience
Collins, A. G. E., & Shenhav, A. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-021-01126-y · Cited by 104 · see citing works
Self-citationTsutsui‐Kimura, I., Matsumoto, H., Akiti, K., Yamada, M. M., Uchida, N., & Watabe‐Uchida, M. (2020) · eLife · doi:10.7554/elife.62390 · Cited by 91 · see citing works
Origin of perseveration in the trade-off between reward and complexity
Gershman, S. J. (2020) · Cognition · doi:10.1016/j.cognition.2020.104394 · Cited by 97 · see citing works
The relationship between habits and motor skills in humans
Du, Y., Krakauer, J. W., & Haith, A. M. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.02.002 · Cited by 94 · see citing works
Perez, O. D., & Dickinson, A. (2020) · Psychological Review · doi:10.1037/rev0000201 · Cited by 92 · see citing works
Hierarchical Action Control: Adaptive Collaboration Between Actions and Habits
Balleine, B. W., & Dezfouli, A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02735 · Cited by 88 · see citing works
Dopaminergic action prediction errors serve as a value-free teaching signal
Greenstreet, F., Vergara, H. M., Johansson, Y., Pati, S., Schwarz, L., Lenzi, S. C., Geerts, J. P., Wisdom, M., Gubanova, A., Rollik, L. B., et al. (2025) · Nature · doi:10.1038/s41586-025-09008-9 · Cited by 87 · see citing works
Time pressure changes how people explore and respond to uncertainty
Wu, C. M., Schulz, E., Pleskac, T. J., & Speekenbrink, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-07901-1 · Cited by 89 · see citing works
Positive affect and behavior change
Shiota, M. N., Papies, E. K., Preston, S. D., & Sauter, D. A. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.04.022 · Cited by 83 · see citing works
Dopamine role in learning and action inference
Bogacz, R. (2020) · eLife · doi:10.7554/elife.53262 · Cited by 79 · see citing works
Linear reinforcement learning in planning, grid fields, and cognitive control
Piray, P., & Daw, N. D. (2021) · Nature Communications · doi:10.1038/s41467-021-25123-3 · Cited by 83 · see citing works
Context-dependent outcome encoding in human reinforcement learning
Palminteri, S., & Lebreton, M. P. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.06.006 · Cited by 78 · see citing works
The role of the lateral orbitofrontal cortex in creating cognitive maps
Costa, K. M., Scholz, R., Lloyd, K., Moreno‐Castilla, P., Gardner, M., Dayan, P., & Schoenbaum, G. M. (2022) · Nature Neuroscience · doi:10.1038/s41593-022-01216-0 · Cited by 74 · see citing works
Gardner, B., Rebar, A. L., de Wit, S., & Lally, P. J. (2024) · Social and Personality Psychology Compass · doi:10.1111/spc3.12975 · Cited by 87 · see citing works
van Elzelingen, W., Warnaar, P., de Matos, J. M. S., Bastet, W., Jonkman, R., Smulders, D., Goedhoop, J. N., Denys, D. A. J. P., Arbab, T., & Willuhn, I. (2022) · Current Biology · doi:10.1016/j.cub.2021.12.027 · Cited by 63 · see citing works
Value representations in the rodent orbitofrontal cortex drive learning, not choice
Miller, K. J., Botvinick, M., & Brody, C. D. (2022) · eLife · doi:10.7554/elife.64575 · Cited by 67 · see citing works
Self-citationFintz, M., Osadchy, M., & Hertz, U. (2022) · Scientific Reports · doi:10.1038/s41598-022-08863-0 · Cited by 63 · see citing works
Reward prediction error does not explain movement selectivity in DMS-projecting dopamine neurons
Lee, R. S., Mattar, M. G., Parker, N. F., Witten, I. B., & Daw, N. D. (2019) · eLife · doi:10.7554/elife.42992 · Cited by 60 · see citing works
A Normative Account of Confirmation Bias During Reinforcement Learning
Lefebvre, G., Summerfield, C., & Bogacz, R. (2021) · Neural Computation · doi:10.1162/neco_a_01455 · Cited by 58 · see citing works
Making habits measurable beyond what they are not: A focus on associative dual-process models
Watson, P., O’Callaghan, C., Perkes, I. E., Bradfield, L. A., & Turner, K. M. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104869 · Cited by 61 · see citing works
Reinforcement learning and its connections with neuroscience and psychology
Subramanian, A., Chitlangia, S., & Baths, V. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.10.003 · Cited by 57 · see citing works
The Law of Recency: An Episodic Stimulus-Response Retrieval Account of Habit Acquisition
Giesen, C. G., Schmidt, J. R., & Rothermund, K. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02927 · Cited by 53 · see citing works
Bavard, S., Rustichini, A., & Palminteri, S. (2021) · Science Advances · doi:10.1126/sciadv.abe0340 · Cited by 53 · see citing works
Frömer, R., & Shenhav, A. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.12.006 · Cited by 50 · see citing works
Self-citationPolicy compression: An information bottleneck in action selection
Lai, L., & Gershman, S. (2021) · The psychology of learning and motivation · doi:10.1016/bs.plm.2021.02.004 · Cited by 77 · see citing works
Gardner, B., & Lally, P. J. (2022) · Health Psychology Review · doi:10.1080/17437199.2022.2105249 · Cited by 55 · see citing works
Zhang, C., Lakens, D., & IJsselsteijn, W. A. (2021) · Journal of Medical Internet Research · doi:10.2196/17127 · Cited by 56 · see citing works
Interfacing behavioral and neural circuit models for habit formation
Lerner, T. N. (2020) · Journal of Neuroscience Research · doi:10.1002/jnr.24581 · Cited by 46 · see citing works
Cognitive Control as a Multivariate Optimization Problem
Ritz, H., Leng, X., & Shenhav, A. (2022) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01822 · Cited by 49 · see citing works
Self-citationGenerating Options and Choosing Between Them Depend on Distinct Forms of Value Representation
Morris, A., Phillips, J., Huang, K., & Cushman, F. A. (2021) · Psychological Science · doi:10.1177/09567976211005702 · Cited by 49 · see citing works
Caching mechanisms for habit formation in Active Inference
Maisto, D., Friston, K. J., & Pezzulo, G. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.05.083 · Cited by 46 · see citing works
Value-free reinforcement learning: policy optimization as a minimal model of operant behavior
Bennett, D., Niv, Y., & Langdon, A. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.04.020 · Cited by 46 · see citing works
Smith, A. C. W., Jonkman, S., DiFeliceantonio, A. G., O’Connor, R. M., Ghoshal, S., Romano, M. F., Everitt, B. J., & Kenny, P. J. (2021) · Nature Communications · doi:10.1038/s41467-021-25460-3 · Cited by 44 · see citing works
Linder, N., Rosenthal, S., Sörqvist, P., & Barthel, S. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.699410 · Cited by 39 · see citing works
Choice-confirmation bias and gradual perseveration in human reinforcement learning.
Palminteri, S. (2022) · Behavioral Neuroscience · doi:10.1037/bne0000541 · Cited by 40 · see citing works
Studying Human Habits in Societal Context: Examining Support for a Basic Stimulus–Response Mechanism
Marien, H., Ruud Custers, A., & Aarts, H. (2019) · Current Directions in Psychological Science · doi:10.1177/0963721419868211 · Cited by 39 · see citing works
The Willpower Paradox: Possible and Impossible Conceptions of Self-Control
Goschke, T., & Job, V. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916221146158 · Cited by 38 · see citing works
Realigning Models of Habitual and Goal-Directed Decision-Making
Miller, K. J., Ludvig, E. A., Pezzulo, G., & Shenhav, A. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-812098-9.00018-8 · Cited by 41 · see citing works
Self-citationThe affective gradient hypothesis: An affect-centered account of motivated behavior
Shenhav, A. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.08.003 · Cited by 43 · see citing works
Self-citationLeveraging cognitive neuroscience for making and breaking real-world habits
Buabang, E. K., Donegan, K. R., Rafei, P., & Gillan, C. M. (2025) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.10.006 · Cited by 34 · see citing works
The learning of prospective and retrospective cognitive maps within neural circuits
Namboodiri, V. M. K., & Stuber, G. D. (2021) · Neuron · doi:10.1016/j.neuron.2021.09.034 · Cited by 31 · see citing works
Goal-Directed and Habitual Control in Human Substance Use: State of the Art and Future Directions
Doñamayor, N., Ebrahimi, C., Arndt, V. A., Weiß, F., Schlagenhauf, F., & Endraß, T. (2022) · Neuropsychobiology · doi:10.1159/000527663 · Cited by 33 · see citing works
A mismatch between striatal cholinergic pauses and dopaminergic reward prediction errors
Duhne, M., Mohebi, A., Kim, K., Pelattini, L., & Berke, J. D. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2410828121 · Cited by 33 · see citing works
Marzuki, A. A., Tomić, I., Ip, S. H. Y., Gottwald, J., Kanen, J. W., Kaser, M., Sule, A., Conway-Morris, A., Sahakian, B. J., & Robbins, T. W. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.36195 · Cited by 32 · see citing works
An fMRI meta-analysis of the role of the striatum in everyday-life vs laboratory-developed habits
Guida, P., Michiels, M., Redgrave, P., Luque, D., & Obeso, I. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104826 · Cited by 32 · see citing works
Wu, W., Hu, L., Chen, Y., Cao, F., Ding, S., Wu, T., & Xu, J. (2022) · BMC Oral Health · doi:10.1186/s12903-022-02219-w · Cited by 31 · see citing works
Characterizing Human Habits in the Lab
Nebe, S., Kretzschmar, A., Brandt, M. C., & Tobler, P. N. (2024) · Collabra Psychology · doi:10.1525/collabra.92949 · Cited by 33 · see citing works
Hackel, L. M., Berg, J. J., Lindström, B., & Amodio, D. M. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02592 · Cited by 29 · see citing works
Pro-environmental messages have more effect when they come from less familiar brands
Herédia-Colaço, V. (2022) · Journal of Product & Brand Management · doi:10.1108/jpbm-12-2021-3782 · Cited by 30 · see citing works
Parvizi-Wayne, D., Sandved-Smith, L., Pitliya, R. J., Limanowski, J., Tufft, M. R. A., & Friston, K. J. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1354719 · Cited by 34 · see citing works
Binary Theorizing Does Not Account for Action Control
Hommel, B. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02542 · Cited by 32 · see citing works
Balancing control: A Bayesian interpretation of habitual and goal-directed behavior
Schwöbel, S., Marković, D., Smolka, M. N., & Kiebel, S. J. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102472 · Cited by 28 · see citing works
Moneta, N., Grossman, S., & Schuck, N. W. (2024) · Trends in Neurosciences · doi:10.1016/j.tins.2024.10.005 · Cited by 33 · see citing works
Luo, Q., Kanen, J. W., Bari, A., Skandali, N., Langley, C., Knudsen, G. M., Alsiö, J., Phillips, B. U., Sahakian, B. J., Cardinal, R. N., et al. (2023) · Neuropsychopharmacology · doi:10.1038/s41386-023-01762-6 · Cited by 26 · see citing works
Herz, D. M., Frank, M. J., Tan, H., & Groppa, S. (2024) · Brain · doi:10.1093/brain/awae184 · Cited by 27 · see citing works
Mixing memory and desire: How memory reactivation supports deliberative decision‐making
Wang, S., Feng, S. F., & Bornstein, A. M. (2021) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1581 · Cited by 26 · see citing works
Reinforcer predictability and stimulus salience promote discriminated habit learning.
Thrailkill, E. A., Michaud, N. L., & Bouton, M. E. (2021) · Journal of Experimental Psychology Animal Learning and Cognition · doi:10.1037/xan0000285 · Cited by 28 · see citing works
Group value learned through interactions with members: A reinforcement learning account
Hackel, L. M., Kogon, D., Amodio, D. M., & Wood, W. (2021) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2021.104267 · Cited by 27 · see citing works
Value Representations in the Rodent Orbitofrontal Cortex Drive Learning, not ChoicePreprint
Miller, K. J., Botvinick, M., & Brody, C. D. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/245720 · Cited by 27 · see citing works
Self-citationInvasive Computational Psychiatry
Sáez, I., & Gu, X. (2022) · Biological Psychiatry · doi:10.1016/j.biopsych.2022.09.032 · Cited by 25 · see citing works
Model-Free RL or Action Sequences?
Morris, A., & Cushman, F. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02892 · Cited by 25 · see citing works
Miller, K. J., Botvinick, M., & Brody, C. D. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/461129 · Cited by 25 · see citing works
Self-citationTheory-based habit modeling for enhancing behavior prediction in behavior change support systems
Zhang, C., Vanschoren, J., van Wissen, A., Lakens, D., de Ruyter, B., & IJsselsteijn, W. A. (2022) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-022-09326-x · Cited by 24 · see citing works
Akam, T. E., Rodrigues-Vaz, I., Marcelo, I., Zhang, X., Pereira, M., Oliveira, R. F., Dayan, P., & Costa, R. M. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/126292 · Cited by 23 · see citing works
Eckstein, M. K., Summerfield, C., Daw, N. D., & Miller, K. J. (2023) · doi:10.1101/2023.05.17.541226 · Cited by 18 · see citing works
Self-citationThe consequences of AI training on human decision-making
Treiman, L. S., Ho, C.-J., & Kool, W. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2408731121 · Cited by 22 · see citing works
Schmidt, J. (2021) · L’Année psychologique · doi:10.3917/anpsy1.212.0077 · Cited by 17 · see citing works
Tanner, C., Schmocker, D., Katsarov, J. W., & Christen, M. (2021) · Computers & Education · doi:10.1016/j.compedu.2021.104381 · Cited by 27 · see citing works
Frith, C., & Frith, U. (2023) · The MIT Press eBooks · doi:10.7551/mitpress/10400.001.0001 · Cited by 22 · see citing works
Associative Learning from Replayed ExperiencePreprint
Ludvig, E. A., Mirian, M. S., Kehoe, E. J., & Sutton, R. S. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/100800 · Cited by 20 · see citing works
Self-citationPiray, P., & Daw, N. D. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/856849 · Cited by 22 · see citing works
Ferry, L., Funnell, W., & Oldroyd, D. (2023) · Accounting, Organizations and Society · doi:10.1016/j.aos.2023.101466 · Cited by 23 · see citing works
Neurobehavioral Precursors of Compulsive Cocaine Seeking in Dual Frontostriatal Circuits
Jones, J. A., Belin‐Rauscent, A., Jupp, B., Fouyssac, M., Sawiak, S. J., Zühlsdorff, K., Zhukovsky, P., Hebdon, L., Velázquez-Sánchez, C., Robbins, T. W., et al. (2023) · Biological Psychiatry Global Open Science · doi:10.1016/j.bpsgos.2023.06.001 · Cited by 20 · see citing works
Experiential values are underweighted in decisions involving symbolic options
Garcia, B., Lebreton, M. P., Bourgeois‐Gironde, S., & Palminteri, S. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01496-3 · Cited by 19 · see citing works
Toyama, A., Katahira, K., & Kunisato, Y. (2023) · Computational Brain & Behavior · doi:10.1007/s42113-023-00175-4 · Cited by 18 · see citing works
Testing models of context-dependent outcome encoding in reinforcement learning
Hayes, W. M., & Wedell, D. H. (2022) · Cognition · doi:10.1016/j.cognition.2022.105280 · Cited by 18 · see citing works
Representation of Anticipated Rewards and Punishments in the Human Brain
Ben‐Zion, Z., & Levy, I. (2024) · Annual Review of Psychology · doi:10.1146/annurev-psych-022324-042614 · Cited by 20 · see citing works
Old Strategies, New Environments: Reinforcement Learning on Social Media
Turner, G., Ferguson, A. M., Katiyar, T., Palminteri, S., & Orben, A. C. (2024) · Biological Psychiatry · doi:10.1016/j.biopsych.2024.12.012 · Cited by 26 · see citing works
Gamification of Behavior Change: Mathematical Principle and Proof-of-Concept Study
Lieder, F., Chen, P.-Z., Prentice, M. S., Amo, V., & Tošić, M. (2023) · JMIR Serious Games · doi:10.2196/43078 · Cited by 17 · see citing works
Environmental habitus of chefs and food waste management in hospitality operations
Bhaskara, G. I., Filimonau, V., Kusumawardhana, I., Putra, I. N. D., Yanthy, P. S., & Hakim, I. N. (2024) · Journal of Sustainable Tourism · doi:10.1080/09669582.2024.2408008 · Cited by 17 · see citing works
Womelsdorf, T., Watson, M. R., & Tiesinga, P. H. E. (2021) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01780 · Cited by 17 · see citing works
Theory-driven computational models of drug addiction in humans: Fruitful or futile?
Lim, T. V., & Ersche, K. D. (2023) · Addiction Neuroscience · doi:10.1016/j.addicn.2023.100066 · Cited by 15 · see citing works
Computational approaches to habits in a model-free world
Pauli, W. M., Cockburn, J., Pool, E. R., Perez, O. D., & O’Doherty, J. P. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.12.001 · Cited by 16 · see citing works
de Oliveira, A. R., Reimer, A. E., Simandl, G. J., Nagrale, S. S., & Widge, A. S. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01448-x · Cited by 14 · see citing works
Zhang, C., Adriaanse, M. A., Potgieter, R., Tummers, L. G., de Wit, J. B. F., Broersen, J., de Bruin, M., & Aarts, H. (2022) · BMC Public Health · doi:10.1186/s12889-022-13977-1 · Cited by 15 · see citing works
Zühlsdorff, K., López‐Cruz, L., Dutcher, E. G., Jones, J. A., Pama, C., Sawiak, S. J., Khan, S. A., Milton, A. L., Robbins, T. W., Bullmore, E. T., et al. (2022) · Neurobiology of Stress · doi:10.1016/j.ynstr.2022.100507 · Cited by 15 · see citing works
Representational Exchange in Human Social Learning
Wu, C. M., Vélez, N., & Cushman, F. A. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009026949.009 · Cited by 15 · see citing works
- Authors
- Miller, K. J., Shenhav, A., & Ludvig, E. A.
- Affiliations
- Princeton University (US)
- Venue
- Psychological Review
- Published
- 2019-03
- Volume
- 126
- Issue
- 2
- Pages
- 292-311
- ISSN
- 0033-295X (ISSN-L)
- 1939-1471
- PMC
- PMC6548181
- OpenAlex
- W2949941589
- Cited by
- 280
- References
- 136
- Citations by year
- 2026: 32 · 2025: 38 · 2024: 52 · 2023: 29 · 2022: 41 · 2021: 41 · 2020: 24 · 2019: 14 · 2018: 6 · 2017: 3
- FWCI
- 17.32
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Psychological Review (publishedVersion, other-oa)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Funders
- National Institutes of Health (T-32 MH065214)
- Princeton University
- Sponsor name not included
- National Institute of General Medical Sciences (P20GM103645)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Social Sciences › Psychology › Applied Psychology › Behavioral Health and Interventions
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- habit, stimulus-response learning, model-free reinforcement learning, outcome devaluation, contingency degradation, reinforcement schedules, perseveration, instrumental behavior, habit formation
- MeSH terms
- Animals, Behavior, Animal, Decision Making, Goals, Habits, Humans, Models, Psychological, Reinforcement, Psychology, Prefrontal Cortex
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
