The DOI matches its Crossref record; 135 of the 136 references it cites were found in a registry or an index; none of the 99 sources its claims cite is checked for support yet.
Habits without values.
Open accessPublished version of a preprint
Abstract
Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as arising from model-free reinforcement learning mechanisms, which typically select between available actions based on the future value expected to result from each. Traditionally, however, habits have been understood as behaviors that can be triggered directly by a stimulus, without requiring the animal to evaluate expected outcomes. Here, we develop a computational model instantiating this traditional view, in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes. We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the effects of reinforcement schedule on the rate of habit formation. The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors and help to better guide research into the neural mechanisms underlying control of instrumental behavior more generally. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper develops a computational model in which habits are value-free stimulus-response associations strengthened by repetition, rather than model-free reinforcement learning. It shows this model reproduces key habit phenomena, including insensitivity to outcome devaluation and contingency degradation, reinforcement-schedule effects on habit formation, lesion-like dissociations, and perseveration.
Key findings
- A value-free habitual controller, combined with a model-based goal-directed controller and an arbiter, reproduces overtraining-induced inflexibility: after longer training, simulated agents continued lever-pressing under omission and devaluation, whereas shorter training allowed reversal. Training duration was varied from 100 to 2000 trials in intervals of 100 for omission and from 10 to 2000 trials in intervals of 100 for devaluation.
- Simulated agents trained on variable-interval (VI6: pellets baited every 6 s, on average 10/min) lever-pressed at a moderate rate and habitized early, whereas agents trained on variable-ratio (VR10: 10% chance per press) lever-pressed at a high rate and habitized much later.
- Simulated lesions reproduced rodent striatal patterns: decreasing goal-directed parameters (DMS-like) produced a low press rate unaffected by omission or devaluation; decreasing habitual parameters (DLS-like) produced a high press rate that adapted to both manipulations. Control agents adapted after limited training (5,000 trials) but not after extensive training (15,000 trials).
- In a dynamic two-armed bandit task (10,000 trials; 50 model copies; reinforcer probabilities changed by independent Gaussian random walks, SD = 0.15), the model reproduced both reinforcement-seeking and value-free perseveration; replacing the habitual component with model-free RL failed to reproduce the perseverative pattern.
Methods
- Computational model with three modules: a habitual controller tracking recency-weighted action history \(H\), a goal-directed controller using model-based RL to learn separate reinforcer expectations \(R\) and utilities \(U\), and an arbiter weighting control by action-outcome contingency \(g\) and overall habitization \(h\).
- Simulated task environments included reversal learning, omission contingency, outcome devaluation, free-operant VR/VI schedules, simulated lesions, and a two-armed bandit. Most simulations used a single state with two actions; free-operant simulations used continuous press rates from 0 to 150 presses/min with function approximation and an effort cost quadratic in press rate.
- Analyses averaged simulated performance across agents and, for the two-armed bandit, used logistic regression quantifying influences of past actions, reinforcers, and their interaction on future choice.
Limitations
- The paper states that the model is described at an abstract computational level, limiting engagement with neurobiological detail, and that a neurobiologically detailed implementation is a critical next step.
- The paper states that the arbiter’s conditions require future research, that the model is agnostic about whether bypassing model-free RL entirely is appropriate, and that the form of value/motivational modulation of habit learning remains unresolved.
- Evident from the text: the findings come from simulations rather than new empirical experiments, and many simulations use simplified single-state, two-action environments with specific parameter and utility choices, so generality across richer environments is not established.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 99 cited sources checked · 0 supported · 0 flagged
None of the sources cited for this claim have been checked yet.
In the paper
“as well as the effects of reinforcement schedule on the rate of habit formation.”
§ Habits without Values > Abstract
What the sources say
- Not checked yetcites Adams, 1982
- Not checked yetcites Dickinson, 1985
- Not checked yetcites Gremel & Costa, 2013
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation”
§ Habits without Values > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system.”
§ Habits without Values > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
None of the sources cited for this claim have been checked yet.
In the paper
“This value-free habit mechanism accounts for key findings in the animal learning literature that dissociate habitual and goal-directed actions, namely the tendency for an animal to continue performing a previously learned action when that action is no longer predictive of the reinforcing outcome (contingency degradation; Hammond, 1980) or when the predicted outcome ceases to be desired by the subj”
§ Habits without Values > Introduction
What the sources say
- Not checked yetcites Hammond, 1980
- Not checked yetcites Adams & Dickinson, 1981
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As a result, when the reversal occurs, the agent continues to select Action A for an extended period, past the point where the goal-directed controller has learned that Action B is more likely to be reinforced (compare Figures 3B and 3E).”
§ Habits without Values > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“In addition, this model provides what is, to our knowledge, the first computational account of the difference in rate of habit formation under variable-interval and variable-ratio schedules of reinforcement (Adams, 1982; Dickinson, 1985; Gremel & Costa, 2013).”
§ Habits without Values > Introduction
What the sources say
- Not checked yetcites Adams, 1982
- Not checked yetcites Dickinson, 1985
- Not checked yetcites Gremel & Costa, 2013
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Initially, behavior is driven by the goal-directed controller, which gradually learns the relative reinforcement rates (Figure 3B) and thus increasingly selects Action A.”
§ Habits without Values > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the following sections we demonstrate that this model can capture all of the key diagnostic features of habitual behavior previously identified in animal behavior, including sensitivity to repetition frequency, reinforcement schedule, and selective modulation by lesions to one of two dissociable neural circuits.”
§ Habits without Values > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition to demonstrating the different kinds of learning that drive each of these controllers, this example demonstrates that our model captures the observation that behavioral control in a novel environment tends to evolve from goal-directed to habitual (i.e., habits form from actions that were originally selected in a value-based manner).”
§ Habits without Values > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A “control” agent, with intact habitual and goal-directed controllers, adopted a high press rate that adapted to both manipulations when given limited training, but did not adapt to either following extensive training.”
§ Habits without Values > Results > Effects of Striatal Lesions on Habit Formation and Behavioral Flexibility
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- It may state two separate claims as one. 21
- Probably not a substantial claim. 18
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“whereas agents trained on VR schedules lever-pressed at a high rate and habitized much later in training (Figure 4, right).”
§ Habits without Values > Results > Effects of Reinforcement Schedule on Habit Formation
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we found that simulated agents trained on VI schedules lever-pressed at a moderate rate and habitized early in training (Figure 4, left)”
§ Habits without Values > Results > Effects of Reinforcement Schedule on Habit Formation
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Model-free algorithms typically operate by learning the expected future reward associated with each possible action or state, relying crucially on these value representations.”
§ Habits without Values > Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Goal-directed behaviors are understood to be driven by consideration of the outcomes that they are likely to bring about (i.e. “action-outcome” representations).”
§ Habits without Values > Introduction
What the sources say
- Not checked yetcites Balleine & O’Doherty, 2010
- Not checked yetcites Dickinson, 1985
- Not checked yetcites Dolan & Dayan, 2013
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“This category of value-free habits has been maintained in modern theorizing on habits, where it has been referred to as “direct cuing” of behavior, as distinct from value-based forms of habits (“motivated cueing”; Wood & Neal, 2007; Wood & Rünger, 2016).”
§ Habits without Values > Introduction
What the sources say
- Not checked yetcites Wood & Neal, 2007
- Not checked yetcites Wood & Rünger, 2016
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Modern reinforcement learning (RL) has elaborated this law into a set of computational algorithms, according to which actions are selected based on cached values learned from previous experience (Daw, Niv, & Dayan, 2005; Dolan & Dayan, 2013; Sutton & Barto, 1998).”
§ Habits without Values > Introduction
What the sources say
- Not checked yetcites Daw, Niv, & Dayan, 2005
- Not checked yetcites Dolan & Dayan, 2013
- Not checked yetcites Sutton & Barto, 1998
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This class of computations focuses only on potential rewards, ignoring all reward-unrelated elements of one’s environment (discussed below); it is therefore referred to as “model-free” RL.”
§ Habits without Values > Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“This formulation for habits has become so prevalent that the terms “habit” and “model-free” are now used interchangeably in much of the computational literature (Dolan & Dayan, 2013; Doll, Simon, & Daw, 2012).”
§ Habits without Values > Introduction
What the sources say
- Not checked yetcites Dolan & Dayan, 2013
- Not checked yetcites Doll, Simon, & Daw, 2012
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“This law holds that an action that has been taken often in the past is likely to be repeated in the future, independent of its past consequences; in other words, it describes habits as a form of perseveration.”
§ Habits without Values > Introduction
What the sources say
- Not checked yetcites Thorndike, 1911
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“A large and growing body of work indicates that these behaviors depend on different sets of computations and distinct underlying neural circuits, suggesting that separable goal-directed and habitual systems implement fundamentally different strategies for the control of behavior”
§ Habits without Values > Introduction
What the sources say
- Not checked yetcites Balleine & O’Doherty, 2010
- Not checked yetcites Dickinson, 1985
- Not checked yetcites Dolan & Dayan, 2013
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here, we have shown computationally that such value-based mechanisms are not strictly necessary, and that a value-free mechanism can account for the major behavioral phenomena that define habits.”
§ Habits without Values > Discussion
Load-bearingStrength:boosted· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The idea that separate goal-directed and habitual controllers exist in the brain, supported by distinct neural circuits, is strongly supported by lesion data from both humans and other animals.”
§ Habits without Values > Discussion > Tensions in Neuroscientific Data
Strength:boosted· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The finding that VI schedules result in rapid habit formation has been widely replicated and represents a key element of the experimental toolkit for the study of habits (Gremel & Costa, 2013; Tanaka, Balleine, & O’Doherty, 2008; Yin & Knowlton, 2006).”
§ Habits without Values > Results > Effects of Reinforcement Schedule on Habit Formation
What the sources say
- Not checked yetcites Gremel & Costa, 2013
- Not checked yetcites Tanaka, Balleine, & O’Doherty, 2008
- Not checked yetcites Yin & Knowlton, 2006
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The first of these features is that habits form slowly over time and often depend on behaviors that are initially taken under goal-directed control.”
§ Habits without Values > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“We first test whether our model can capture a central finding from research on habits: that extensive training in a static environment (overtraining) can render behavior inflexible in the face of changes to that environment.”
§ Habits without Values > Results > Effects of Training Duration on Behavioral Flexibility
What the sources say
- Not checked yetcites Dickinson, 1998
- Not checked yetcites Adams, 1982
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This example not only begins to validate the predictive abilities of our particular model, but also highlights the importance of a value-free habitual controller more generally in explaining habit-like behaviors that cannot otherwise be accounted for by a model-free RL-based algorithm alone.”
§ Habits without Values > Results > Perseverative Behavior in Sequential Choice Tasks
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“relative to control rats, rats with lesions to DMS lever-press at a lower rate and are less able to decrease their press rate when reinforcers are omitted or devalued (Yin et al., 2005)”
§ Habits without Values > Results > Effects of Striatal Lesions on Habit Formation and Behavioral Flexibility
What the sources say
- Not checked yetcites Yin et al., 2005
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“rats with lesions to DLS lever-press at a similar rate to controls and are more successful than controls at adapting their press rate to omitted or devalued reinforcement (Yin, Knowlton, & Balleine, 2004, 2006).”
§ Habits without Values > Results > Effects of Striatal Lesions on Habit Formation and Behavioral Flexibility
What the sources say
- Not checked yetcites Yin, Knowlton, & Balleine, 2004, 2006
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This computational account in which habits are understood as value-free stimulus-response associations therefore provides a closer match to classic psychological theories of habits, an account for classic behavioral data on habit formation, and a novel framework for understanding additional behavioral phenomena.”
§ Habits without Values > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
In the paper
“In one recent example, rats performing a dynamic two-armed bandit task exhibited behavioral patterns consistent with both reinforcement-seeking (i.e., being more likely to select a recently reinforced action), as well as with choice perseveration (i.e., being more likely to select a recently chosen action).”
§ Habits without Values > Results > Perseverative Behavior in Sequential Choice Tasks
What the sources say
- Not checked yetcites Miller et al., in prep
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“The extra computational complexity associated with the goal-directed system was a source of uncertainty, leading the arbiter to favor the habitual system in well-learned environments.”
§ Habits without Values > Discussion > Relationship to Previous Computational Models
What the sources say
- Not checked yetcites Daw et al., 2005
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“This arbiter is similar to mechanisms found in computational accounts mapping habitual/goal-directed control onto model-free/model-based reinforcement learning.”
§ Habits without Values > Discussion > Relationship to Previous Computational Models
What the sources say
- Not checked yetcites Daw et al., 2005
- Not checked yetcites S. W. Lee et al., 2014
- Not checked yetcites Keramati et al., 2011
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“A large and influential body of computational work is built on the assumption that habitual control arises from model-free reinforcement learning algorithms (Dolan & Dayan, 2013; O’Doherty, Lee, & McNamee, 2015).”
§ Habits without Values > Discussion > Relationship to Previous Computational Models
What the sources say
- Not checked yetcites Dolan & Dayan, 2013
- Not checked yetcites O’Doherty, Lee, & McNamee, 2015
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Subsequent accounts have developed arbiters that consider proxies for this uncertainty (S. W. Lee et al., 2014), or other possible advantages of habitual over goal-directed control, such as response time (Keramati et al., 2011).”
§ Habits without Values > Discussion > Relationship to Previous Computational Models
What the sources say
- Not checked yetcites S. W. Lee et al., 2014
- Not checked yetcites Keramati et al., 2011
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In rodents, goal-directed behavior results in greater activity in OFC and DMS, while habitual behavior results in greater activity in DLS (Gremel & Costa, 2013).”
§ Habits without Values > Discussion > Tensions in Neuroscientific Data
What the sources say
- Not checked yetcites Gremel & Costa, 2013
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the value-free mechanisms we have proposed for habits by no means preclude a role for value or motivational state (e.g., hunger or satiety) in habit learning.”
§ Habits without Values > Discussion > Implications
Strength:plain· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These may modulate the strengthening of habit associations either directly (e.g., through a feedback mechanism that influences the S-R association) or indirectly (e.g., by influencing the vigor of an action, which in turn results in greater associative strengths).”
§ Habits without Values > Discussion > Implications
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
None of the sources cited for this claim have been checked yet.
In the paper
“In human subjects, goal-directed value signals during outcome devaluation have been identified in vmPFC (Valentin, Dickinson, & O’Doherty, 2007), and similar signals during contingency degradation have been identified both in vmPFC and in the caudate nucleus, a homolog of DMS (Tanaka et al., 2008).”
§ Habits without Values > Discussion > Tensions in Neuroscientific Data
What the sources say
- Not checked yetcites Valentin, Dickinson, & O’Doherty, 2007
- Not checked yetcites Tanaka et al., 2008
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our proposed framework re-maps habits to the perseveration kernel and provides an account of how that kernel might plausibly operate in tandem with a goal-directed controller so as to account for behaviors that have previously been described by RL models.”
§ Habits without Values > Discussion > Relationship to Previous Computational Models
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These data indicate that dopamine neurons are unlikely to play a role in a uniquely model-free control system, but instead have access to model-based information and play a role in model-based control.”
§ Habits without Values > Discussion > Tensions in Neuroscientific Data
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In effect, we are showing that the model-free component of some of these previous formalizations might not be necessary and that an elaboration of this perseveration kernel actually serves as a better model of habits.”
§ Habits without Values > Discussion > Relationship to Previous Computational Models
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In sum, considerable evidence supports the idea that anatomically separate goal-directed and habitual controllers exist in the brain, and that either controller can be responsible for a given action.”
§ Habits without Values > Discussion > Tensions in Neuroscientific Data
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“habits are promoted in situations where the contingencies between actions and outcomes are weak (Dickinson, 1985).”
§ Habits without Values > Discussion > Relationship to Previous Computational Models
What the sources say
- Not checked yetcites Dickinson, 1985
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We first propose severing the tie between habits and model-free RL, and instead defining a category of behaviors that are “value-free” and therefore distinct from either type of RL computation.”
§ Habits without Values > Discussion > A Proposed Realignment
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“In the initial formalization of this idea (Daw et al., 2005), each system determined the uncertainty in its value estimates, and the arbiter selected the system with less overall uncertainty.”
§ Habits without Values > Discussion > Relationship to Previous Computational Models
What the sources say
- Not checked yetcites Daw et al., 2005
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“In addition to utilizing different mechanisms for value-based control and for arbitrating between the value-based and value-free controllers, our model also differs from these corticostriatal models in the level of analysis at which it is described. While this limits the level of detail with which our model can engage neurobiological data, it greatly facilitates engagement with a wide variety of”
§ Habits without Values > Discussion > Relationship to Previous Computational Models
What the sources say
- Not checked yetcites Frank, 2015
- Not checked yetcites Frank & Badre, 2015
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Beyond providing a potentially more parsimonious account of previous behavioral results”
§ Habits without Values > Conclusions
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“such a realignment may offer a better account of extant neural findings, including the fact that structures associated with model-free and model-based computations (i.e., value-based computations) tend to overlap, whereas lesion/inactivation studies have revealed clear dissociations between structures associated with goal-directed behavior versus (potentially value-free) habits.”
§ Habits without Values > Conclusions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
- The authors state this as an assumption: In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
The paper relies on this without support.
The simulated task environments capture the essential features of the corresponding animal learning experiments, so that the model's behavior in them can be compared to animal behavior.
“To illustrate the behavior of the model and the dynamics of its various internal variables, we simulated behavior in a probabilistic reversal learning task (Fig 3).”
The paper relies on this without support.
The model's goal-directed controller corresponds to the dorsomedial striatum (DMS) and its habitual controller corresponds to the dorsolateral striatum (DLS).
“We lesioned the goal-directed controller (DMS) or habitual controller (DLS) in our model”
“Lesions to our model’s goal-directed and habitual controllers thus reproduce behavioral patterns typical of DMS and DLS lesioned rats”
The authors state this as an assumption.
In the model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.
“In our model, control of behavior is allocated on each trial to either the habitual or the goal-directed system by an arbiter.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Supporting this, computational studies have proposed two parallel forms of reinforcement learning, model-based and model-free (or value-free) learning corresponding to goal-directed and habitual strategies, respectively, that develop concurrently during behavior acquisition2,41,42.”
From Dissociable roles of prefrontal plasticity in decision-making strategy and execution of habitual behavior (2026) · cites this paper as [42]
Experimental — accuracy not yet measuredIn the citing paper
“With overtraining, the habitual system simplifies decision-making by shifting to a stimulus-response mode of behavior51, making decisions potentially value-less and inflexible but less cognitively demanding52.”
From Revealing abrupt transitions from goal-directed to habitual behavior (2026) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Training reinforcement probability differs between Methods and Results
MajorThe Methods specify a 25% pellet probability during omission training, and say devaluation training was identical. The Results describe a 50% pellet probability for both training phases.
“In the first phase, performance of one action (Press Lever) resulted in a reinforcer of one type (Pellet) 25% of the time”
“The agent was initially trained in an environment where performance of Action A (“press lever”) was followed by a reinforcer of one type (“food pellet”) 50% of the time”
Habit-weight update uses a different basis function
MinorThe habit-strength function uses basis functions γi, so its derivative with respect to weight ci is γi. The stated weight-update equation instead substitutes χi, which is not the defined habit basis function.
“=ct(i)+αH⋅∑a′((δa′,at−H(a′))⋅χi(a′))”
“11Ht(a)=∑i=130ct(i)⋅γi(a)γi(a)=e−(a−5i)22⋅52”
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
James, W. (1891) · Personality Culture Society · doi:10.1037/11059-000 · Cited by 25,876
An Integrative Theory of Prefrontal Cortex Function
Miller, E. K., & Cohen, J. D. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.167 · Cited by 12,803
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995) · Psychological Review · doi:10.1037/0033-295x.102.3.419 · Cited by 5,256
The principles of psychology, Vol I.
James, W. (1890) · doi:10.1037/10538-000 · Cited by 4,514
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
The role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402
How are habits formed: modelling habit formation in the real world
Lally, P., van Jaarsveld, C. H. M., Potts, H. W. W., & Wardle, J. (2010) · European Journal of Social Psychology · doi:10.1002/ejsp.674 · Cited by 2,036
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779
Balleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775
Principles of Behavior. An Introduction to Behavior Theory
N., E., & Hull, C. L. (1943) · The Journal of Philosophy · doi:10.2307/2019960 · Cited by 1,736
Habits, Rituals, and the Evaluative Brain
Graybiel, A. M. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.112851 · Cited by 1,697 · cited 4×
“Here, we present a computational implementation of the Law of Exercise and show that it offers an alternative to model-free RL as a mechanism for habits, one that retains ideas about the nature of habits that have developed within other areas of psychology and neuroscience (Graybiel, 2008; James, 1890; Wood & Rünger,…”
BackgroundMethodInfluentialGoal-directed instrumental action: contingency and incentive learning and their cortical substrates
Balleine, B. W., & Dickinson, A. (1998) · Neuropharmacology · doi:10.1016/s0028-3908(98)00033-1 · Cited by 1,541
A new look at habits and the habit-goal interface.
Wood, W., & Neal, D. T. (2007) · Psychological Review · doi:10.1037/0033-295x.114.4.843 · Cited by 1,470
Wood, W., & Rünger, D. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033417 · Cited by 1,445 · cited 9×
“This prediction also provides a mechanistic underpinning for the repetition strategies that have shown to be effective at improving workplace and health-related performance through habit formation (Gardner, Lally, & Wardle, 2012; Lally et al., 2010; Wood & Rünger, 2016).”
BackgroundMethodInfluentialSutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03095.x · Cited by 1,226
Actions and habits: The development of behavioural autonomy
Dickinson, A. (1985) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1985.0010 · Cited by 1,217
A neuropsychological theory of multiple systems in category learning.
Ashby, F. G., Alfonso-Reese, L. A., Turken, A. U., & Waldron, E. M. (1998) · Psychological Review · doi:10.1037/0033-295x.105.3.442 · Cited by 1,216
O’Reilly, R. C., & Frank, M. J. (2005) · Neural Computation · doi:10.1162/089976606775093909 · Cited by 1,124
Frontal cortex and reward-guided learning and decision-making
Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.014 · Cited by 1,111
Animal intelligence; experimental studies
Thorndike, E. L. (1911) · doi:10.5962/bhl.title.55072 · Cited by 1,100
The role of the dorsomedial striatum in instrumental conditioning
Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2005.04218.x · Cited by 1,099
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
Predicting Behavior From Actions in the Past: Repeated Decision Making or a Matter of Habit?
Aarts, H., Verplanken, B., & van Knippenberg, A. (1998) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.1998.tb01681.x · Cited by 926
Animal Intelligence. Experimental Studies
Thorndike, E. L., & Jelliffe (1912) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-191205000-00016 · Cited by 824
A specific role for posterior dorsolateral striatum in human habit learning
Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06796.x · Cited by 769
Conservation of hippocampal memory function in rats and humans
Bunsey, M. D., & Eichenbaum, H. (1996) · Nature · doi:10.1038/379255a0 · Cited by 752
Coordination of Actions and Habits in the Medial Prefrontal Cortex of Rats
Killcross, S. (2003) · Cerebral Cortex · doi:10.1093/cercor/13.4.400 · Cited by 748
The Involvement of the Orbitofrontal Cortex in the Experience of Regret
Camille, N., Coricelli, G., Sallet, J., Pradat-Diehl, P., Duhamel, J.-R., & Sirigu, A. (2004) · Science · doi:10.1126/science.1094550 · Cited by 716
Instrumental Responding following Reinforcer Devaluation
Adams, C. D., & Dickinson, A. (1981) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748108400816 · Cited by 687
Regret and its avoidance: a neuroimaging study of choice behavior
Coricelli, G., Critchley, H., Joffily, M., O’Doherty, J. P., Angela Sirigu, B., & Dolan, R. J. (2005) · Nature Neuroscience · doi:10.1038/nn1514 · Cited by 673
Active Inference, homeostatic regulation and adaptive behavioural control
Pezzulo, G., Rigoli, F., & Friston, K. (2015) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2015.09.001 · Cited by 650
Neural computations underlying arbitration between model-based and model-free learning
Lee, S. W., Shimojo, S., & O’Doherty, J. P. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.028 · Cited by 642
Variations in the Sensitivity of Instrumental Responding to Reinforcer Devaluation
Adams, C. D. (1982) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748208400878 · Cited by 635
Gremel, C. M., & Costa, R. M. (2013) · Nature Communications · doi:10.1038/ncomms3264 · Cited by 618
The hippocampus and memory for orderly stimulus relations
Dusek, J. A., & Eichenbaum, H. (1997) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.94.13.7109 · Cited by 615
Making health habitual: the psychology of ‘habit-formation’ and general practice
Gardner, B., Lally, P., & Wardle, J. (2012) · British Journal of General Practice · doi:10.3399/bjgp12x659466 · Cited by 591 · cited 1×
“This prediction also provides a mechanistic underpinning for the repetition strategies that have shown to be effective at improving workplace and health-related performance through habit formation (Gardner, Lally, & Wardle, 2012; Lally et al., 2010; Wood & Rünger, 2016).”
BackgroundCharacterizing a psychiatric symptom dimension related to deficits in goal-directed control
Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A., & Daw, N. D. (2016) · eLife · doi:10.7554/elife.11305 · Cited by 586
Neural Basis of Reinforcement Learning and Decision Making
Lee, D., Seo, H., & Jung, M. W. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150512 · Cited by 581
Neuronal basis of sequential foraging decisions in a patchy environment
Hayden, B. Y., Pearson, J. M., & Platt, M. L. (2011) · Nature Neuroscience · doi:10.1038/nn.2856 · Cited by 547
YIN, H., KNOWLTON, B., & BALLEINE, B. (2006) · Behavioural Brain Research · doi:10.1016/j.bbr.2005.07.012 · Cited by 532
Determining the neural substrates of goal-directed learning in the human brain
Valentin, V. V., Dickinson, A., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0564-07.2007 · Cited by 529
Cushman, F. A. (2013) · Personality and Social Psychology Review · doi:10.1177/1088868313495594 · Cited by 507
Learning and selective attention
Dayan, P., Kakade, S., & Montague, P. R. (2000) · Nature Neuroscience · doi:10.1038/81504 · Cited by 495
DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS
Lau, B., & Glimcher, P. W. (2005) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.2005.110-04 · Cited by 485
Redish, A. D. (2016) · Nature reviews. Neuroscience · doi:10.1038/nrn.2015.30 · Cited by 485
Cortical and basal ganglia contributions to habit learning and automaticity
Ashby, F. G., Turner, B. O., & Horvitz, J. C. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.02.001 · Cited by 481
The role of prelimbic cortex in instrumental conditioning
Corbit, L. H., & Balleine, B. W. (2003) · Behavioural Brain Research · doi:10.1016/j.bbr.2003.09.023 · Cited by 423
The ubiquity of model-based reinforcement learning
Doll, B. B., Simon, D. A., & Daw, N. D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.08.003 · Cited by 380
The effect of lesions of the basolateral amygdala on instrumental conditioning
Balleine, B. W., Killcross, A. S., & Dickinson, A. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-02-00666.2003 · Cited by 355
Otto (2013) · doi:10.1177/0956797612463080 · Cited by 352
Goal-directed learning and obsessive-compulsive disorder
Gillan, C. M., & Robbins, T. W. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0475 · Cited by 348
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · cited 3×
“Subsequent accounts have developed arbiters that consider proxies for this uncertainty (S. W. Lee et al., 2014), or other possible advantages of habitual over goal-directed control, such as response time (Keramati et al., 2011).”
BackgroundRedish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346
Orbitofrontal Cortex Supports Behavior and Learning Using Inferred But Not Cached Values
Jones, J. L., Esber, G. R., McDannald, M. A., Gruber, A. J., Hernández, A., Mirenzi, A., & Schoenbaum, G. M. (2012) · Science · doi:10.1126/science.1227489 · Cited by 343
Coutureau, E., & Killcross, S. (2003) · Behavioural Brain Research · doi:10.1016/j.bbr.2003.09.025 · Cited by 342
The effect of the instrumental training contingency on susceptibility to reinforcer devaluation
Dickinson, A., Nicholas, D. J., & Adams, C. D. (1983) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748308400912 · Cited by 338
Contributions of the striatum to learning, motivation, and performance: an associative account
Liljeholm, M., & O’Doherty, J. P. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2012.07.007 · Cited by 317
Mapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310
Covert expectation-of-reward in rat ventral striatum at decision points
van der Meer, M. A. A. (2009) · Frontiers in Integrative Neuroscience · doi:10.3389/neuro.07.001.2009 · Cited by 299
Dopamine transients are sufficient and necessary for acquisition of model-based associations
Sharpe, M. J., Chang, C. Y., Liu, M. A., Batchelor, H. M., Mueller, L. E., Jones, J. L., Niv, Y., & Schoenbaum, G. M. (2017) · Nature Neuroscience · doi:10.1038/nn.4538 · Cited by 298
Habits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297
Reward value comparison via mutual inhibition in ventromedial prefrontal cortex
Strait, C. E., Blanchard, T. C., & Hayden, B. Y. (2014) · Neuron · doi:10.1016/j.neuron.2014.04.032 · Cited by 294
THE EFFECT OF CONTINGENCY UPON THE APPETITIVE CONDITIONING OF FREE‐OPERANT BEHAVIOR
Hammond, L. J. (1980) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1980.34-297 · Cited by 291
A neurobiological theory of automaticity in perceptual categorization
Ashby, F. G., Ennis, J. M., & Spiering, B. J. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.632 · Cited by 286
Validation of Decision-Making Models and Analysis of Decision Variables in the Rat Basal Ganglia
Ito, M., & Doya, K. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.6157-08.2009 · Cited by 284
Model-based learning protects against forming habits
Gillan, C. M., Otto, A. R., Phelps, E. A., & Daw, N. D. (2015) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-015-0347-6 · Cited by 283
Crockett, M. J. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.06.005 · Cited by 279
Neural signature of fictive learning signals in a sequential investment task
Lohrenz, T. M., McCabe, K., Camerer, C. F., & Montague, P. R. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0608842104 · Cited by 276
Calculating Consequences: Brain Systems That Encode the Causal Effects of Actions
Tanaka, S. C., Balleine, B. W., & O'Doherty, J. P. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1808-08.2008 · Cited by 268
Regulation of dietary choice by the decision-making circuitry
Rangel, A. (2013) · Nature Neuroscience · doi:10.1038/nn.3561 · Cited by 263
Rutledge, R. B., Lazzaro, S. C., Lau, B., Myers, C. E., Gluck, M. A., & Glimcher, P. W. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.3524-09.2009 · Cited by 259
Dorsal hippocampus contributes to model-based planning
Miller, K. J., Botvinick, M., & Brody, C. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4613 · Cited by 252
Self-citationTesting the Reward Prediction Error Hypothesis with an Axiomatic Model
Rutledge, R. B., Dean, M., Caplin, A., & Glimcher, P. W. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.1747-10.2010 · Cited by 244
Smittenaar, P., FitzGerald, T. H. B., Romei, V., Wright, N. D., & Dolan, R. J. (2013) · Neuron · doi:10.1016/j.neuron.2013.08.009 · Cited by 244
Does the orbitofrontal cortex signal value?
Schoenbaum, G., Takahashi, Y., Liu, T., & McDannald, M. A. (2011) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.06210.x · Cited by 232
Role of Striatum in Updating Values of Chosen Actions
Kim, H., Sul, J. H., Huh, N., Lee, D., & Jung, M. W. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.2728-09.2009 · Cited by 229
Animal Intelligence: Experimental Studies
Brown, W. M. (1912) · Nature · doi:10.1038/088306a0 · Cited by 227
Kishida, K. T., Sáez, I., Lohrenz, T. M., Witcher, M. R., Laxton, A. W., Tatter, S. B., White, J. P., Ellis, T. L., Phillips, P. E. M., & Montague, P. R. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1513619112 · Cited by 224
Internally generated sequences in learning and executing goal-directed behavior
Pezzulo, G., van der Meer, M. A. A., Lansink, C. S., & Pennartz, C. M. A. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.06.011 · Cited by 217
Retrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217
McDannald, M. A., Lucantonio, F., Burke, K. A., Niv, Y., & Schoenbaum, G. M. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.5499-10.2011 · Cited by 216
Serial Choice Reaction-time as a Function of Response versus Signal-and-Response Repetition
Bertelson, P. (1965) · Nature · doi:10.1038/206217a0 · Cited by 207
Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum
Keramati, M., Smittenaar, P., Dolan, R. J., & Dayan, P. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1609094113 · Cited by 205
Autonomous Mechanism of Internal Choice Estimate Underlies Decision Inertia
Akaishi, R., Umeda, K., Nagase, A. M., & Sakai, K. (2013) · Neuron · doi:10.1016/j.neuron.2013.10.018 · Cited by 202
Cognitive Control Predicts Use of Model-based Reinforcement Learning
Otto, A. R., Skatova, A., Madlon‐Kay, S., & Daw, N. D. (2014) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00709 · Cited by 202
Dopamine neurons respond to errors in the prediction of sensory features of expected rewards
Takahashi, Y. K., Batchelor, H. M., Liu, B., Khanna, A., Morales, M., & Schoenbaum, G. (2017) · Neuron · doi:10.1016/j.neuron.2017.08.025 · Cited by 200
Solway, A., & Botvinick, M. (2012) · Psychological Review · doi:10.1037/a0026435 · Cited by 194
Corbit, L. H., Muir, J. L., & Balleine, B. W. (2003) · European Journal of Neuroscience · doi:10.1046/j.1460-9568.2003.02833.x · Cited by 186
A Pallidus-Habenula-Dopamine Pathway Signals Inferred Stimulus Values
Bromberg-Martin, E. S., Matsumoto, M., Hong, S., & Hikosaka, O. (2010) · Journal of Neurophysiology · doi:10.1152/jn.00158.2010 · Cited by 186
Learning robust cortico-cortical associations with the basal ganglia: An integrative review
Hélie, S., Ell, S. W., & Ashby, F. G. (2014) · Cortex · doi:10.1016/j.cortex.2014.10.011 · Cited by 181
Neuronal Origins of Choice Variability in Economic Decisions
Padoa‐Schioppa, C. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.013 · Cited by 169
Model-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161
Social norms, self-control, and the value of antisocial behavior
Buckholtz, J. W. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.03.004 · Cited by 154
Social Norms Shift Behavioral and Neural Responses to Foods
Nook, E. C., & Zaki, J. (2015) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00795 · Cited by 147
Cho, R. Y., Nystrom, L. E., Brown, E., Jones, A. D., Braver, T. S., Holmes, P. J., & COHEN, J. D. (2002) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.2.4.283 · Cited by 140
O’Doherty, J. P. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.03.027 · Cited by 136
Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. M. (2016) · eLife · doi:10.7554/elife.13665 · Cited by 136
Reber, J., Feinstein, J. S., O’Doherty, J. P., Liljeholm, M., Adolphs, R., & Tranel, D. (2017) · Brain · doi:10.1093/brain/awx105 · Cited by 134
Dopamine selectively remediates ‘model-based’ reward learning: a computational approach
Sharp, M. E., Foerde, K., Daw, N. D., & Shohamy, D. (2015) · Brain · doi:10.1093/brain/awv347 · Cited by 133
Gold, J. I., Law, C.-T., Connolly, P. M., & Bennur, S. (2008) · Journal of Neurophysiology · doi:10.1152/jn.90629.2008 · Cited by 131
Learning and decision making in monkeys during a rock–paper–scissors game
Lee, D., McGreevy, B. P., & Barraclough, D. J. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2005.07.003 · Cited by 121
Generalization of value in reinforcement learning by humans
Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08017.x · Cited by 111
Instrumental uncertainty as a determinant of behavior under interval schedules of reinforcement
DeRusso, A. L. (2010) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2010.00017 · Cited by 111
Sample-based learning and search with permanent and transient memories
Silver, D., Sutton, R. S., & Müller, M. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390278 · Cited by 109
The structure of reinforcement-learning mechanisms in the human brain
O’Doherty, J. P., Lee, S. W., & McNamee, D. (2014) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2014.10.004 · Cited by 106
Multiplicity of control in the basal ganglia: computational roles of striatal subregions
Bornstein, A. M., & Daw, N. D. (2011) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2011.02.009 · Cited by 106
A theoretical account of cognitive effects in delay discounting
Kurth-Nelson, Z. L., Bickel, W. K., & Redish, A. D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08058.x · Cited by 101
Learning and motivation in the human striatum
Shohamy, D. (2011) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2011.05.009 · Cited by 100
Hierarchical Action Control: Adaptive Collaboration Between Actions and Habits
Balleine, B. W., & Dezfouli, A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02735 · Cited by 98
Bornstein, A. M., & Daw, N. D. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003387 · Cited by 96
Omission Learning after Instrumental Pretraining
Dickinson, A. (2010) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/713932679 · Cited by 88
Scott, B. B., Constantinople, C. M., Erlich, J. C., Tank, D. W., & Brody, C. D. (2015) · eLife · doi:10.7554/elife.11308 · Cited by 82
Lucantonio, F., Caprioli, D., & Schoenbaum, G. M. (2013) · Neuropharmacology · doi:10.1016/j.neuropharm.2013.05.033 · Cited by 80
The Law of Recency: An Episodic Stimulus-Response Retrieval Account of Habit Acquisition
Giesen, C. G., Schmidt, J. R., & Rothermund, K. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02927 · Cited by 60
Goal-Directed Decision Making with Spiking Neurons
Friedrich, J. H., & Lengyel, M. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.2854-15.2016 · Cited by 58
Coherency-maximizing exploration in the supermarket
Riefer, P. S., Prior, R., Blair, N., Pavey, G., & Love, B. C. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0017 · Cited by 53
Caching mechanisms for habit formation in Active Inference
Maisto, D., Friston, K. J., & Pezzulo, G. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.05.083 · Cited by 49
Realigning Models of Habitual and Goal-Directed Decision-Making
Miller, K. J., Ludvig, E. A., Pezzulo, G., & Shenhav, A. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-812098-9.00018-8 · Cited by 46
Self-citationHow cognitive theory guides neuroscience
Frank, M. J., & Badre, D. (2014) · Cognition · doi:10.1016/j.cognition.2014.11.009 · Cited by 44
Neurons in the Nucleus Accumbens Promote Selection Bias for Nearer Objects
Morrison, S. E., & Nicola, S. M. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.2197-14.2014 · Cited by 30
Jung, D., & Dorner, V. (2017) · Lecture notes in information systems and organisation · doi:10.1007/978-3-319-67431-5_18 · Cited by 29
Model-Free RL or Action Sequences?
Morris, A., & Cushman, F. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02892 · Cited by 28
Balcarras, M., Ardid, S., Kaping, D., Everling, S., & Womelsdorf, T. (2015) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00894 · Cited by 25
Akam, T., Rodrigues-Vaz, I., Marcelo, I., Zhang, X., Pereira, M., Oliveira, R. F., Dayan, P., & Costa, R. M. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/126292 · Cited by 25
Linking Across Levels of Computation in Model-Based Cognitive Neuroscience
Frank, M. J. (2015) · Springer eBooks · doi:10.1007/978-1-4939-2236-9_8 · Cited by 24
Associative Learning from Replayed ExperiencePreprint
Ludvig, E. A., Mirian, M. S., Kehoe, E. J., & Sutton, R. S. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/100800 · Cited by 20
Self-citationO’Reilly, R. C., Nair, A., Russin, J., & Herd, S. A. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.00380 · Cited by 14
A Neurocomputational Model of Automatic Sequence Production
Hélie, S., Roeder, J. L., Vucovich, L. E., Rünger, D., & Ashby, F. G. (2015) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00794 · Cited by 12
The prefrontal cortex and hybrid learning during iterative competitive games
Abe, H., Seo, H., & Lee, D. (2011) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.06223.x · Cited by 11
Time course of a repetition effect on saccadic reaction time in non-human primates.
Gore, J. L., Dorris, M. C., & Munoz, D. P. (2002) · PubMed · Cited by 10
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. M. (2016) · doi:10.7554/elife.13665.010 · Cited by 4
Dual Competition between the Basal Ganglia and the Cortex: from Action-Outcome to Stimulus-ResponsePreprint
Topalidou, M., Kase, D., Boraud, T., & Rougier, N. P. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/187294 · Cited by 3
Flagged first, then by citations
1 of the 136 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Binding and Retrieval in Action Control (BRAC)
Frings, C., Hommel, B., Koch, I., Rothermund, K., Dignath, D., Giesen, C. G., Kiesel, A., Kunde, W., Mayr, S., Moeller, B., et al. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.02.004 · Cited by 379
Motivation and cognitive control in depression
Grahek, I., Shenhav, A., Musslick, S., Krebs, R. M., & Koster, E. H. W. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.04.011 · Cited by 389
Self-citationResolving uncertainty in a social world
FeldmanHall, O., & Shenhav, A. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0590-x · Cited by 298
Self-citationThe neural and computational systems of social learning
Olsson, A., Knapska, E., & Lindström, B. (2020) · Nature reviews. Neuroscience · doi:10.1038/s41583-020-0276-4 · Cited by 285
Habitual Behavior Is Goal-Driven
Kruglanski, A. W., & Szumowska, E. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620917676 · Cited by 231
Metadata
- Authors
- Miller, K. J., Shenhav, A., & Ludvig, E. A.
- Affiliations
- Princeton University (US)
- Venue
- Psychological Review
- Published
- 2019-03
- Volume
- 126
- Issue
- 2
- Pages
- 292-311
- ISSN
- 0033-295X (ISSN-L)
- 1939-1471
- PMC
- PMC6548181
- OpenAlex
- W2949941589
- Cited by
- 280
- References
- 136
- Citations by year
- 2026: 32 · 2025: 38 · 2024: 52 · 2023: 29 · 2022: 41 · 2021: 41 · 2020: 24 · 2019: 14
- FWCI
- 17.32
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Psychological Review (publishedVersion, other-oa)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Funders
- National Institutes of Health (T-32 MH065214)
- Princeton University
- Sponsor name not included
- National Institute of General Medical Sciences (P20GM103645)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Social Sciences › Psychology › Applied Psychology › Behavioral Health and Interventions
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- habit, stimulus-response learning, model-free reinforcement learning, outcome devaluation, contingency degradation, reinforcement schedules, perseveration, instrumental behavior, habit formation
- MeSH terms
- Animals, Behavior, Animal, Decision Making, Goals, Habits, Humans, Models, Psychological, Reinforcement, Psychology, Prefrontal Cortex
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
