CoChat
Sign Up
Cited by 977Neural and Behavioral Psychology StudiesDecision-Making and Behavioral EconomicsNeurotransmitter Receptor Influence on Behavior

Goals and Habits in the Brain

Open access
Dolan, Ray J., Dayan, Peter·2013·Neuron
DOI 10.1016/j.neuron.2013.09.007PMC PMC3807793
Open this paper in CoChat — free accountOpen PDF

Abstract

An enduring and richly elaborated dichotomy in cognitive neuroscience is that of reflective versus reflexive decision making and choice. Other literatures refer to the two ends of what is likely to be a spectrum with terms such as goal-directed versus habitual, model-based versus model-free or prospective versus retrospective. One of the most rigorous traditions of experimental work in the field started with studies in rodents and graduated via human versions and enrichments of those experiments to a current state in which new paradigms are probing and challenging the very heart of the distinction. We review four generations of work in this tradition and provide pointers to the forefront of the field's fifth generation.

AI snapshot

Written from the paper’s full text.

TL;DR

This review traces five “generations” of research on the dichotomy between reflective/goal-directed/model-based control and reflexive/habitual/model-free control in instrumental decision making. It argues that these systems are more richly intertwined than originally supposed, with unresolved questions about how model-based computation is realized and how the two systems compete or cooperate.

Key findings

  • Goal-directed instrumental behavior is defined by two criteria: response-outcome control and current motivational relevance of the outcome; habitual behavior persists despite outcome devaluation and is automatic, computationally efficient, and inflexible.
  • Rodent work implicates dorsomedial striatum and prelimbic prefrontal cortex in goal-directed action, and dorsolateral striatum and infralimbic cortex in habits; human fMRI studies report devaluation-related BOLD decreases in ventral orbitofrontal cortex and, after extensive training, increased cue-related activity in right posterior putamen/globus pallidus.
  • Model-based and model-free reinforcement learning map onto goal-directed and habitual control; two-step Markov tasks show hybrid control, and Daw et al. found ventral striatal prediction-error signal covaried with model-based predictions and with individual model-based behavior.
  • Pharmacological and clinical findings include L-DOPA increasing model-based choice, acute phenylalanine/tyrosine depletion increasing slips of action, and Parkinson’s disease severity negatively covarying with goal-directed devaluation performance; the review also links altered model-based/model-free balance to OCD, addiction, and speculatively psychosis.

Methods

  • Narrative review, not a primary empirical study; it organizes the literature into five “generations,” from cognitive maps and goal-directed/habit studies through human neuroimaging, model-based/model-free RL, elaborations, and future directions.
  • Focus is primarily on human instrumental behavior, with animal studies sketched only as essential background.
  • Main analysis is conceptual and computational synthesis of behavioral paradigms (outcome devaluation, contingency degradation, two-step Markov tasks, Pavlovian-instrumental transfer) and neural findings; no new data, systematic search, or formal meta-analysis is reported.

Limitations

  • The paper states it is an inevitably selective review and that space prevents discussion of many issues; animal studies are only sketched, and there is no consensus on the primate homolog of rat infralimbic cortex.
  • The paper states that how model-based calculations are realized remains unresolved, and that arbitration/competition between model-based and model-free control needs much more work.
  • Evident from the text: as a narrative review, it does not systematically search or quantitatively synthesize the literature, and some cited evidence is described as personal communication or unpublished data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.neuron.2013.09.007

Cites

173 of 173 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 144 cited sources checked · 0 supported · 0 flagged

  • Premisecitation

    Lesions to dorsomedial striatum result in an early emergence of habitual behavior.

    “a pattern that contrasts with the effect of lesions to dorsomedial striatum that result in an early emergence of habitual behavior (Yin and Knowlton, 2006).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Yin and Knowlton, 2006 source

      Not checked in the free run.

  • Interpretationreasoning

    The coexistence of the two classes of control has a normative rationale as offering efficient solutions to complex and changing environments.

    “The latter provides a normative rationale for their coexistence as offering efficient solutions to the demands of complex and changing environments and has also underpinned the design and interpretation of a collection of targeted empirical studies.”
    Quote in the paperSubstantial claim
  • Premisecitation

    If an outcome becomes equally available with and without a lever press, goal-directed control leads to a decrease in pressing.

    “If the outcome subsequently becomes equally available with and without a lever press, goal-directed control leads to a decrease in pressing (Dickinson and Balleine, 1994; Dickinson and Charnock, 1985).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Dickinson and Balleine, 1994 source

      Not checked in the free run.

    • cites Dickinson and Charnock, 1985

      Not checked in the free run.

  • Premisereasoning

    In a test session in extinction, there should be less instrumental responding for an outcome that has been devalued than for an outcome that has not, if behavior is goal directed.

    “In this case, there should be less instrumental responding for an outcome that has been devalued (for example, a food reinforcer that has just been rendered unpalatable) than for an outcome that has not.”
    Quote in the paperSubstantial claim
  • Premisecitation

    In the psychology of decision making and choice, there is more than one class of possible strategy for acting.

    “An important and pervasive idea in the psychology of decision making and choice is that there is more than one class of possible strategy for acting.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Dickinson, 1985

      Not checked in the free run.

    • cites Kahneman, 2011

      Not checked in the free run.

    • cites Stanovich and West, 2002

      Not checked in the free run.

  • Classified as probably not a substantial claim. 12
  • May combine two claims. 42

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The strand of work singled out by the paper provides behaviorally rigorous criteria for separating goal-directed and habitual control.

    “Here, we single out one particular strand that has arguably been the most fecund in cognitive and theoretical neuroscience, providing a set of behaviorally rigorous criteria for separating out the two classes of control.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Habitual instrumental control is characterized by automaticity, computational efficiency, and inflexibility, while goal-directed control is characterized by active deliberation, high computational cost, and adaptive flexibility to changing environmental contingencies.

    “Thus, key characteristics of habitual instrumental control include automaticity, computational efficiency, and inflexibility, while characteristics of goal-directed control include active deliberation, high computational cost, and an adaptive flexibility to changing environmental contingencies (Dayan, 2009).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Dayan, 2009 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Frontal prelimbic lesions abolish sensitivity to both outcome devaluation manipulations and degradation of instrumental contingency.

    “Frontal prelimbic lesions abolish sensitivity to both outcome devaluation manipulations as well as to degradation of instrumental contingency (Balleine and Dickinson, 1998; Corbit and Balleine, 2003).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Balleine and Dickinson, 1998 source

      Not checked in the free run.

    • cites Corbit and Balleine, 2003 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Building and evaluating a decision tree imposes processing and working memory demands that rapidly become unrealistic with increasing depth, so a model-based agent is confronted with overwhelming computational constraints reflecting capacity limitations within attention and working memory.

    “It is easy to appreciate that building and evaluating a tree imposes processing and working memory demands that rapidly become unrealistic with increasing depth. Consequently, a model-based agent is confronted with overwhelming computational constraints that in psychological terms reflect the known capacity limitations within attention and working memory.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: All the tasks discussed in the paper can be mapped onto the reinforcement learning framework in a straightforward manner.

  • May combine two claims.

    Premisecitation

    Selective lesions to prelimbic medial prefrontal cortex induce lack of sensitivity to goal value following either limited or extended training, whereas selective lesions to infralimbic regions result in retained sensitivity to goal value after both limited and extended training.

    “Furthermore, selective lesions to prelimbic medial prefrontal cortex induce lack of sensitivity to goal value following either limited or extended training, whereas selective lesions to infralimbic regions result in an opposite deficit, namely retained sensitivity to goal value after both limited and extended training (Killcross and Coutureau, 2003).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Killcross and Coutureau, 2003

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The key difference between model-free and model-based control makes it possible to discern the influence of each controller on behavior and to determine whether neural signals are correlated with predictions and prediction errors specific to each controller.

    “This key difference makes it possible to discern the influence of each controller on behavior and also to determine whether neural signals are correlated with predictions and prediction errors specific to each controller.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: All the tasks discussed in the paper can be mapped onto the reinforcement learning framework in a straightforward manner.

  • May combine two claims.

    Premisecitation

    In the rat, caudal, but not rostral, infralimbic cortex projects to ventral parts of medial caudate putamen, and there is no known projection to dorsolateral striatum.

    “The connection between infralimbic cotex and dorsolateral striatum is rather less clear and in the rat, caudal, but not rostral, infralimbic cortex projects to ventral parts of medial caudate putamen (Vertes, 2004), but there is no known projection to dorsolateral striatum.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Vertes, 2004 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Activity of ensembles of neurons in dorsolateral striatum and infralimbic cortex reflects the creation and stabilization of habits, with differences between the regions in the evolution of these patterns.

    “It is known that the activity of ensembles of neurons in dorsolateral striatum and the infralimbic cortex reflect the creation and stabilization of habits, with interesting differences between the regions in the evolution of these patterns (Smith and Graybiel, 2013).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Smith and Graybiel, 2013

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    A huge wealth of subsequent studies have confirmed and elaborated the picture that BOLD signal in dorsal and ventral striatum correlates with model-free temporal difference prediction error.

    “A huge wealth of subsequent studies have confirmed and elaborated this picture.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: All the tasks discussed in the paper can be mapped onto the reinforcement learning framework in a straightforward manner.

  • May combine two claims.

    Interpretationcitation

    In the intact animal, there is likely a dynamic interdependency between goal-directed and habitual systems, and control likely emerges simultaneously and competitively.

    “However, it is likely that in the intact animal, there is a dynamic interdependency between goal-directed and habitual systems and that control is likely to emerge simultaneously and competitively (Wassum et al., 2009).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wassum et al., 2009 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    vmPFC is likely to have a complex role in value representation, with evidence linking it to stimulus value, outcome value, and action value.

    “vmPFC is likely to have a complex role in value representation and there is strong evidence linking this region to both stimulus value and outcome value, and even recent evidence linking it to action value (FitzGerald et al., 2012).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites FitzGerald et al., 2012 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Cached values depend on past utilities and are divorced from the outcomes that they predict.

    “Further, the cached values depend on past utilities and so are divorced from the outcomes that they predict.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: All the tasks discussed in the paper can be mapped onto the reinforcement learning framework in a straightforward manner.

  • May combine two claims.

    Interpretationreasoning

    Model-free control is fundamentally retrospective, and new cached values can only be acquired through direct experience.

    “Thus, model-free control is fundamentally retrospective, and new cached values, as might arise with a change in the utility of an outcome in an environment, can only be acquired through direct experience.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: All the tasks discussed in the paper can be mapped onto the reinforcement learning framework in a straightforward manner.

  • May combine two claims.

    Interpretationreasoning

    The double dissociation makes a strong case that prelimbic regions are crucial for goal-directed performance, while infralimbic lesions prevent the emergence of habitual responding that overrides an initial dominance in goal-directed responding.

    “This double dissociation makes a strong case that prelimbic regions are crucial for goal-directed performance, while infralimbic lesions prevent the emergence of habitual responding that overrides an initial dominance in goal-directed responding.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Human lesion data has led to the suggestion that orbital prefrontal cortex implements encoding of stimulus value with dorsal cingulate cortex implementing encoding of action value.

    “We note also that human lesion data has led to the suggestion that orbital prefrontal cortex implements encoding of stimulus value with dorsal cingulate cortex implementing encoding of action value (Camille et al., 2011).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Camille et al., 2011 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Model-based control conceives of goal-directed control as sophisticated, computationally demanding, prospective planning, in which a decision tree of possible future states and actions is built using a learned internal model of the environment.

    “Model-based control conceives of this in terms of sophisticated, computationally demanding, prospective planning, in which a decision tree of possible future states and actions is built using a learned internal model of the environment.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: All the tasks discussed in the paper can be mapped onto the reinforcement learning framework in a straightforward manner.

  • May combine two claims.

    Interpretationcitation

    In complex circumstances, lesions to dorsolateral striatum enhanced learning, suggesting that a form of competition might be at work.

    “whereas in complex circumstances, lesions to dorsolateral striatal actually enhanced learning, suggesting that a form of competition might be at work.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Bradfield and Balleine, 2013

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Prelimbic prefrontal cortex and dorsomedial striatum both support goal-directed action, in line with the anatomical connectivity between these regions.

    “The fact that prelimbic prefrontal cortex and dorsomedial striatum both support goal-directed action is in line with the anatomical connectivity between these regions (Groenewegen et al., 1990; McGeorge and Faull, 1989).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Groenewegen et al., 1990 source

      Not checked in the free run.

    • cites McGeorge and Faull, 1989

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Model-free control is computationally efficient because it replaces computation with memory, but the forward-looking nature of the prediction error makes it statistically inefficient.

    “Model-free control is computationally efficient, since it replaces computation (i.e., the burdensome simulation of future states) with memory (i.e., stored discounted values of expected future reward); however, the forward-looking nature of the prediction error makes it statistically inefficient (Daw et al., 2005).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Daw et al., 2005 source

      Not checked in the free run.

    • Rests on an assumption: All the tasks discussed in the paper can be mapped onto the reinforcement learning framework in a straightforward manner.

  • May combine two claims.

    Premisecitation

    Two ends of a spectrum of reinforcement learning methods are model-based and model-free control, and these have been associated with goal-directed and habitual control, respectively.

    “Two ends of a spectrum of RL methods are model-based and model-free control (where the term model refers to a mental as opposed to a computational model); it is these that have been associated with goal-directed and habitual control, respectively (Daw et al., 2005; Doya et al., 2002).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Daw et al., 2005 source

      Not checked in the free run.

    • cites Doya et al., 2002 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Transfer over the course of experience from allocentric to egocentric reference frames of a sequence of manual button presses was proposed to depend on two separate cortical regions that both interact with the basal ganglia.

    “The other parallel is with the transfer over the course of experience from allocentric to egocentric reference frames of a sequence of manual button presses (Hikosaka et al., 1999), although this was proposed to depend on two separate cortical regions that both interact with the basal ganglia.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hikosaka et al., 1999 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Pretraining, but not posttraining, lesions disrupt acquisition, but not expression, of goal-directed behavior.

    “Pretraining, but not posttraining, lesions disrupt acquisition, but not expression, of goal-directed behavior (Ostlund and Balleine, 2005).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ostlund and Balleine, 2005

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Goal-directed or response-outcome learning has a strong declarative flavor but is conceptually distinct from hippocampal-dependent stimulus-stimulus learning.

    “It is also worth noting here that although goal-directed or response-outcome learning has a strong declarative flavor, it is conceptually distinct from a hippocampal-dependent stimulus-stimulus form of learning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The idea that prospective outcomes are explicitly represented allows states to be valued according to their current worth and choices to be immediately sensitive to devaluation.

    “Critically, the idea that prospective outcomes are explicitly represented allows these states to be valued (putatively via the orbitofrontal or ventromedial prefrontal cortex) (Valentin et al., 2007; Fellows, 2011; O’Doherty, 2011) according to their current worth and so choices can be immediately sensitive to devaluation.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Valentin et al., 2007 source

      Not checked in the free run.

    • cites Fellows, 2011

      Not checked in the free run.

    • cites O’Doherty, 2011 source

      Not checked in the free run.

    • Rests on an assumption: All the tasks discussed in the paper can be mapped onto the reinforcement learning framework in a straightforward manner.

  • May combine two claims.

    Premisecitation

    In Tricomi et al. (2009), after outcome devaluation, the minimally trained group retained outcome sensitivity, while the extensively trained group did not.

    “After outcome devaluation, performance showed that the minimally trained group retained outcome sensitivity, while the extensively trained group did not, just as in the animal studies.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Tricomi et al., 2009 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In the de Wit et al. instrumental conflict task, subjects with Parkinson's disease could solve the task, arguing that habit formation may not have been eliminated.

    “They showed that subjects with the disease could solve the task, arguing that habit formation may not have been eliminated.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites de Wit et al., 2011 source

      Not checked in the free run.

    • Rests on an assumption: Parkinson’s disease involves the progressive death of dopamine cells and causes a decrease in dopamine release.

  • May combine two claims.

    Premisecitation

    The strength of white matter connections between premotor cortex and posterior putamen predicts vulnerability to "slips of action," a vulnerability also predicted by gray matter density in the putamen.

    “For example, the strength of white matter connections between premotor cortex and posterior putamen is reported to predict vulnerability to “slips of action” (where non-goal-relevant, previously trained, actions are automatically elicited by environmental cues), a vulnerability also predicted by gray matter density in the putamen (de Wit et al., 2012b).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites de Wit et al., 2012b

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    In a double-blind, fully counterbalanced (repeated-measures) design, Wunderlich et al. administered either L-DOPA or placebo while subjects solved the two-step Markov decision task of Daw et al. (2011).

    “In a double-blind, fully counterbalanced (repeated-measures), design, Wunderlich et al. (2012b) administered either L-DOPA (to boost the influence of dopamine) or placebo while subjects solved the two-step Markov decision task of (Daw et al., 2011).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Wunderlich et al., 2012b

      Not checked in the free run.

    • cites Daw et al., 2011 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    After devaluation in the APTD study, dopamine depletion had no impact upon stimulus-response learning or response-outcome learning.

    “After devaluation, depletion had no impact upon stimulus-response learning or response-outcome learning.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites de Wit et al., 2012a

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The frequency of slips of action does not offer a very precise measurement of the relative influence of model-based and model-free systems.

    “The frequency of slips of action does not offer a very precise measurement of the relative influence of model-based and model-free systems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Dopamine projects to the entire striatum, including regions such as dorsomedial striatum (or caudate), which have been implicated in model-based control, and dorsolateral striatum (or putamen), implicated in model-free control.

    “Second, dopamine projects to the entire striatum, including regions such as dorsomedial striatum (or caudate), which have been implicated in model-based control, and dorsolateral striatum (or putamen), implicated in model-free control (Balleine, 2005).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Balleine, 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Depletion studies, whether experimental or disease based, are likely to exert a much less detrimental effect on dopamine function than the 6-OHDA lesions conventionally used in animal studies.

    “However, depletion studies, whether experimental or disease based, are likely to exert a much less detrimental effect on dopamine function than the 6-OHDA lesions conventionally used in animal studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The model-based system is not thought to use temporal difference prediction errors for its own calculations; rather, it uses the state prediction error.

    “One reason for a surprise at the presence of this signal is that the model-based system is not thought to use these prediction errors for its own calculations (rather, it uses the state prediction error, where a state prediction error is a measure of the surprise in a new state given a current estimate of state-action-state transition probabilities (Gläscher et al., 2010).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gläscher et al., 2010 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In the Gläscher et al. task, fMRI data highlighted a conventional model-free temporal difference reward prediction error in ventral striatum, whereas a different sort of state prediction error, associated with acquisition of the model, was seen in posterior inferior parietal and lateral prefrontal cortices.

    “fMRI data highlighted a conventional model-free temporal difference reward prediction error in ventral striatum, whereas a different sort of state prediction error, associated with the acquisition of the model, was seen in posterior inferior parietal and lateral prefrontal cortices.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gläscher et al., 2010 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In a rat experiment, a posttraining manipulation of value coupled to a dopamine infusion into ventromedial PFC produced a bidirectional effect whereby the dopamine infusion decreased responding to a devalued outcome and enhanced responding to nondevalued outcomes, suggesting an influence on model-based valuation.

    “For instance, in a rat experiment in which a posttraining manipulation of value was coupled to a dopamine infusion into ventromedial PFC (vmPFC) (Hitchcott et al., 2007), a bidirectional effect was evident whereby the dopamine infusion decreased responding to a devalued outcome and enhanced responding to nondevalued outcomes, suggesting an influence on model-based valuation.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hitchcott et al., 2007 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    vmPFC can access both model-based and model-free values, consistent with parallel, and independent, operation of model-based and model-free valuation systems.

    “Thus, vmPFC can access both model-based and model-free values, consistent with parallel, and independent, operation of model-based and model-free valuation systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Many early fMRI studies into prediction errors used model-free accounts in Pavlovian paradigms and located prediction errors in striatal BOLD.

    “Many early fMRI studies into prediction errors used model-free accounts in Pavlovian paradigms and located prediction errors in striatal BOLD (Berns et al., 2001; O’Doherty, 2004; O’Doherty et al., 2003; Haruno et al., 2004).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Berns et al., 2001 source

      Not checked in the free run.

    • cites O’Doherty, 2004 source

      Not checked in the free run.

    • cites O’Doherty et al., 2003 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    How control is parsed between model-based and model-free systems is likely to have psychopathological implications.

    “How control is parsed between model-based and model-free systems is likely to have psychopathological implications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    When subjects were required to choose between an overtrained pair and half the tree, the caudate represented the planned target value of the decision tree, while activity in the putamen pertained solely to the value of the overtrained pair.

    “By contrast, when subjects were required to choose between an overtrained pair and half the tree, a situation requiring access to both model-based and model-free values, the caudate represented the planned target value of the decision tree, while activity in the putamen pertained solely to the value of the overtrained pair.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wunderlich et al., 2012a

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Protracted exposure to addictive drugs recruits dopamine-dependent striato-nigro-striatal ascending spirals from the nucleus accumbens to more dorsal regions of the striatum.

    “One influential proposal is that a protracted exposure to addictive drugs recruits dopamine-dependent striato-nigro-striatal ascending spirals (Haber et al., 2000; Joel and Weiner, 2000) from the nucleus accumbens to more dorsal regions of the striatum (Everitt et al., 2008).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Haber et al., 2000 source

      Not checked in the free run.

    • cites Joel and Weiner, 2000 source

      Not checked in the free run.

    • cites Everitt et al., 2008 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The irrelevant incentive effect shows a form of model-based motivationally sensitive evaluation and appears to depend on something akin to PIT.

    “and the irrelevant incentive effect, which shows a form of model-based motivationally sensitive evaluation, appears to depend on something akin to PIT (Dickinson and Dawson, 1987a, 1987b) in a way that suggests this Pavlovian/instrumental difference.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Dickinson and Dawson, 1987a

      Not checked in the free run.

    • cites Dickinson and Dawson, 1987b

      Not checked in the free run.

    • Rests on an assumption: Animal findings on Pavlovian revaluation and PIT are relevant to human model-based and model-free control.

  • May combine two claims.

    Premisecitation

    There was little evidence that BOLD in the vmPFC was also related to value in the Simon and Daw spatial task, though parts of the medial temporal lobe also showed significant correlations with these and with reward predictions.

    “By contrast, there was little evidence that BOLD in the vmPFC was also related to value, as might have been expected, though parts of the medial temporal lobe also showed significant correlations with these and with reward predictions.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Simon and Daw, 2011

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Pavlovian conditioning differs from instrumental conditioning conceptually in the choice of action (automatic versus learned) rather than in the nature of the predictions, and so it is possible that it also has access to both model-free and model-based predictions.

    “Second, Pavlovian conditioning differs from instrumental conditioning conceptually in the choice of action (automatic versus learned) rather than in the nature of the predictions, and so it is possible that it also has access to both model-free and model-based predictions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    There is good evidence for Pavlovian predictions of actual outcomes, which what we argue underpins instrumental model-based control, and this seems to account for behavioral phenomena such as specific forms of Pavlovian instrumental transfer (PIT).

    “There is good evidence for Pavlovian predictions of actual outcomes, which what we argue underpins instrumental model-based control, and this seems to account for behavioral phenomena such as specific forms of Pavlovian instrumental transfer (PIT) (Ostlund and Maidment, 2012; Kruse et al., 1983).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Ostlund and Maidment, 2012

      Not checked in the free run.

    • cites Kruse et al., 1983

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Instant Pavlovian revaluation associated with saline deprivation occurs normally in decorticate animals and does not depend on regions strongly affiliated with model-based control such as the vmPFC.

    “For instance, instant Pavlovian revaluation associated with saline deprivation happens normally in decorticate animals, evidently not depending on regions strongly affiliated with model-based control such as the vmPFC (Wirsig and Grill, 1982).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wirsig and Grill, 1982

      Not checked in the free run.

    • Rests on an assumption: Animal findings on Pavlovian revaluation and PIT are relevant to human model-based and model-free control.

  • May combine two claims.

    Premisecitation

    Other regions including the anterior insula, the precentral cortex, and the medial cingulate covaried with facets of the transitions available from a room, suggesting that they might be involved in realizing the model of the world.

    “Other regions including the anterior insula, the precentral cortex, and the medial cingulate covaried with facets of the transitions available from a room, suggesting that they might be involved in realizing the model of the world.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Simon and Daw, 2011

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In the de Wit et al. study, goal-directed performance in a posttraining devaluation test covaried negatively with disease severity, arguing that model-based influences were impaired.

    “They also showed that (goal-directed) performance in a posttraining devaluation test covaried negatively with disease severity, arguing that model-based influences were impaired.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites de Wit et al., 2011 source

      Not checked in the free run.

    • Rests on an assumption: Parkinson’s disease involves the progressive death of dopamine cells and causes a decrease in dopamine release.

  • May combine two claims.

    Premisecitation

    Other related search tasks have found behavioral evidence for these calculations and have started to look at heuristics for pruning the tree, a necessity when it gets too wide or deep.

    “Other related search tasks have found behavioral evidence for these calculations and have started to look at heuristics for pruning the tree, a necessity when it gets too wide or deep (Huys et al., 2012).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Huys et al., 2012 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    There is no unique form of model-free or model-based control and evidence hints that there are intermediate points on the spectrum between them.

    “There is no unique form of model-free or model-based control and evidence hints that there are intermediate points on the spectrum between them.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    This recruitment results in a shift in control from action-outcome to stimulus-response mechanisms, a putative dominant mode of control in drug seeking and drug relapse.

    “This results in a shift in control from action-outcome to stimulus-response mechanisms, a putative dominant mode of control in drug seeking and drug relapse.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Everitt et al., 2008 source

      Not checked in the free run.

50 of 50

  • The paper relies on this without support.

    All the tasks discussed in the paper can be mapped onto the reinforcement learning framework in a straightforward manner.

    “All the tasks discussed above can be mapped onto this framework in a straightforward manner.”

  • The paper relies on this without support.

    The utilities in the reinforcement learning framework depend on the motivations of the subject (e.g., water is more valuable given thirst).

    “The utilities depend on the motivations of the subject (water is more valuable given thirst).”

  • The authors state this as an assumption.

    The forward sweeps relevant to immediate choices start at the subject’s current location.

    “The forward sweeps relevant to immediate choices are assumed to start at the subject’s current location.”

  • The paper relies on this without support.

    Building and searching a deep tree imposes a huge burden on cognitive control and working memory.

    “Building and searching a deep tree imposes a huge burden on cognitive control and working memory.”

  • The paper relies on this without support.

    Prefrontal cortical functions such as working memory are particularly critical for the operation of model-based control.

    “These functions are particularly critical for the operation of model-based control.”

  • The paper relies on this without support.

    Parkinson’s disease involves the progressive death of dopamine cells and causes a decrease in dopamine release.

    “Conversely, Parkinson’s disease involves the progressive death of dopamine cells and so causes a decrease in dopamine release.”

  • The paper relies on this without support.

    The effects of neuromodulatory therapies in OCD patients do not account for the observed lack of sensitivity to outcome devaluation and increased frequency of slips of action.

    “Patients with OCD (albeit potentially confounded by the effects of their neuromodulatory therapies) showed no deficit in using rewarding feedback to guide action but instead showed both lack of sensitivity to outcome devaluation and increased frequency in slips of action.”

  • The paper relies on this without support.

    Animal findings on Pavlovian revaluation and PIT are relevant to human model-based and model-free control.

    “For instance, instant Pavlovian revaluation associated with saline deprivation happens normally in decorticate animals, evidently not depending on regions strongly affiliated with model-based control such as the vmPFC (Wirsig and Grill, 1982).”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Two parallel forms of reinforcement learning, model-based and model-free (or value-free), correspond to goal-directed and habitual strategies and develop concurrently during behavior acquisition.

    “Supporting this, computational studies have proposed two parallel forms of reinforcement learning, model-based and model-free (or value-free) learning corresponding to goal-directed and habitual strategies, respectively, that develop concurrently during behavior acquisition2,41,42.”

    From Dissociable roles of prefrontal plasticity in decision-making strategy and execution of habitual behavior (2026) · cites this paper as [2]

    The 'Generation 3: Model-Based and Model-free Analyses' section associates model-based and model-free control with goal-directed and habitual control, respectively, and reports a hybrid model integrating both during learning (page unavailable). But the paper describes model-free control as using stored values of expected reward, so it does not support the parenthetical equivalence with 'value-free.' (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Rewarded experience updates the model for stimulus-response-outcome associations when using a model-based learning strategy.

    “As conceptualized by computational studies2,41,46, rewarded experience updates the model for stimulus-response-outcome associations when using a model-based learning strategy.”

    From Dissociable roles of prefrontal plasticity in decision-making strategy and execution of habitual behavior (2026) · cites this paper as [2]

  • More recent formulations emphasize cooperation, rather than only competition, between behavioural controllers.

    “More recent formulations emphasize cooperation, rather than only competition (Daw and Dayan, 2014, Dolan and Dayan, 2013, Keramati et al., 2011, O’Doherty et al., 2015, Pezzulo et al., 2013; see also Daw et al., 2011, Lee et al., 2014) for relevant empirical evidence that supports an integrative viewpoint.”

    From Active Inference, homeostatic regulation and adaptive behavioural control (2015) · cites this paper as Dolan and Dayan, 2013

  • The neural substrates of the goal-directed control system are dissociable from those of the stimulus-response habit system.

    “A host of basic neuroscience studies have been conducted in this area, converging on the notion that the neural substrates of the goal-directed control system are dissociable from that of the stimulus-response habit system (Dolan and Dayan, 2013).”

    From The role of habit in compulsivity (2016) · cites this paper as Dolan and Dayan, 2013

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Direction of training between systems is reversed

    Minor

    The account of the DYNA-Q task says the model-based system was trained by the model-free system. Earlier, the paper describes DYNA as the model-based system training the model-free system-the opposite direction.

1 of 1

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Dolan, R. J., & Dayan, P.
Affiliations
Gatsby Computational Neuroscience Unit (GB)
ORCID
Dayan, Peter
Venue
Neuron
Publisher
Elsevier BV
Published
2013-10
Volume
80
Issue
2
Pages
312-325
ISSN
0896-6273 (ISSN-L)
DOI
10.1016/j.neuron.2013.09.007
PMC
PMC3807793
OpenAlex
W1988699227
Semantic Scholar
d5e9f40771f7006f4901e437e6c54bfdf4f453e7
Cited by
977
References
173
Citations by year
2026: 54 · 2025: 87 · 2024: 78 · 2023: 69 · 2022: 83 · 2021: 94 · 2020: 98 · 2019: 96 · 2018: 81 · 2017: 79 · 2016: 66 · 2015: 52 · 2014: 39 · 2013: 1
FWCI
4.51
Type
journal article
Language
en
Open access
Licence
Locations
  • Neuron (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Max Planck Digital Library (submittedVersion, other-oa)
  • UCL Discovery (University College London) (submittedVersion)
  • MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
  • MPG.PuRe (Max Planck Society) (submittedVersion)
Funders
Wellcome Trust (091593, 098362, 098362/Z/12/Z, 091593/Z/10/Z)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
Keywords
goal-directed behavior, habitual behavior, decision-making, choice, model-based decision making
MeSH terms
Animals, Brain, Conditioning, Operant, Goals, Habits, Humans, Models, Psychological

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review