CoChat
Sign Up
Cited by 1,166Neural and Behavioral Psychology StudiesReinforcement Learning in RoboticsNeurotransmitter Receptor Influence on Behavior

States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning

Open access
Gläscher, Jan, Daw, Nathaniel, Dayan, Peter, O'Doherty, John P.·2010·Neuron
DOI 10.1016/j.neuron.2010.04.016PMC PMC2895323
Open this paper in CoChat — free accountOpen PDF

Abstract

Reinforcement learning (RL) uses sequential experience with situations (“states”) and outcomes to assess actions. Whereas model-free RL uses this experience directly, in the form of a reward prediction error (RPE), model-based RL uses it indirectly, building a model of the state transition and outcome structure of the environment, and evaluating actions by searching this model. A state prediction error (SPE) plays a central role, reporting discrepancies between the current model and the observed state transitions. Using functional magnetic resonance imaging in humans solving a probabilistic Markov decision task we found the neural signature of an SPE in the intraparietal sulcus and lateral prefrontal cortex, in addition to the previously well-characterized RPE in the ventral striatum. This finding supports the existence of two unique forms of learning signal in humans, which may form the basis of distinct computational strategies for guiding behavior.

AI snapshot

Written from the paper’s full text.

TL;DR

Using fMRI in humans solving a probabilistic Markov decision task, the authors found dissociable neural signals for model-based state prediction errors (SPE) and model-free reward prediction errors (RPE). SPEs correlated with BOLD in intraparietal sulcus and lateral prefrontal cortex, whereas RPEs correlated with ventral striatum, and behavior was best explained by a hybrid of both learning systems.

Key findings

  • On the first free-choice trial of session 2, 13 of 18 subjects chose the optimal action and 5 chose the other option (p < 0.05, sign-test, one-tailed), indicating model-based state knowledge acquired during the non-rewarded first session.
  • The HYBRID learner fit behavior better than SARSA alone (χ²(2) = 21.32, p = 2.35×10⁻⁵) and better than FORWARD alone (χ²(2) = 224.94, p = 0); the actual session 1 trial sequence explained choices better than 99.6% of random permutations (p = 0.004).
  • fMRI showed SPE correlates bilaterally in posterior intraparietal sulcus and lateral prefrontal cortex, while RPE correlated in ventral striatum; no SPE effect appeared in ventral striatum even at p < 0.001 uncorrected, and no systematic RPE relationship appeared in pIPS/latPFC.
  • The SPE parameter estimate in right pIPS correlated with percent correct choices (r = 0.57, p = 0.013), but lateral PFC did not (left: r = 0.28, p = 0.27; right: r = 0.38, p = 0.12); session 1 SPE effects were confirmed in bilateral latPFC and right pIPS/angular gyrus.

Methods

  • Twenty subjects were tested; 2 were excluded for not exceeding the 95th percentile of rewards from 10,000 randomly behaving agents, leaving 18 analyzed (8 females; mean age 24 years ± 7.57 SD). fMRI was performed on a 3T Siemens Trio scanner during a probabilistic Markov decision task with two scanning sessions of 80 trials each: session 1 had instructed choices and no rewards, session 2 had free choices and monetary outcomes (0c, 10c, 25c).
  • Three computational models were compared: a model-free SARSA learner generating RPEs, a model-based FORWARD learner generating SPEs, and a HYBRID learner combining their action values with an exponentially decaying weight. Parameters were fit by minimizing negative log-likelihood; fMRI analysis used SPM5, single-subject parametric modulators, and group repeated-measures ANOVA with whole-brain FWE correction at p < 0.05.

Limitations

  • Stated: the experimental design does not allow the authors to rule out definitively that the SPE signal reflects a general attention or salience signal, though they argue against this using abs(RPE) comparisons and the instructed, non-rewarded session 1.
  • Stated: the SPE signal in lateral PFC was not predictive of subsequent choice behavior, unlike right pIPS, suggesting latPFC may integrate other learning signals.
  • Evident: the fMRI results are correlational in 18 subjects, and a single set of model parameters was estimated across all participants, which may obscure individual differences and cannot establish causal roles for these regions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.neuron.2010.04.016

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 14 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In humans solving a probabilistic Markov decision task, BOLD activation in the ventral striatum correlates with a model-free reward prediction error (RPE).

    “in addition to the previously well-characterized RPE in the ventral striatum.”
    • Also stated in: § Neural signatures of RPE and SPE · plain
      “Conversely, when we tested for a correlation between BOLD activation and the RPE we found a significant effect in the ventral striatum (vStr) (Figure 3c), consistent with previous accounts (McClure et al., 2003; O’Doherty et al., 2003), but no effects for an SPE even at p < 0.001 uncorrected.”
    • Also stated in: § DISCUSSION · plain
      “whereas a model-free RPE correlated with the BOLD signal in the ventral striatum.”
    Quote in the paperSubstantial claim
  • Findingown data

    In humans solving a probabilistic Markov decision task, BOLD activation correlates with a model-based state prediction error (SPE) in the posterior intraparietal sulcus (pIPS) and lateral prefrontal cortex (latPFC).

    “Using functional magnetic resonance imaging in humans solving a probabilistic Markov decision task we found the neural signature of an SPE in the intraparietal sulcus and lateral prefrontal cortex, in addition to the previously well-characterized RPE in the ventral striatum.”
    • Also stated in: § Neural signatures of RPE and SPE · plain
      “we pooled across both sessions and found significant effects bilaterally in the posterior intraparietal sulcus (pIPS) reaching on the left side into the superior parietal lobule and on the right side into the angular gyrus, and in the lateral prefrontal cortex (latPFC) (dorsal bank of the posterior inferior frontal gyrus (pIFG), see circled areas in Figure 3a and 3b and Table 2).”
    • Also stated in: § DISCUSSION · plain
      “In the imaging data we found trial-by-trial correlations of the model-based SPE in the pIPS and latPFC, whereas a model-free RPE correlated with the BOLD signal in the ventral striatum.”
    Quote in the paperSubstantial claim
  • Findingown data

    In session 2, the mean correlation of the prediction error signals from the model-free and model-based models was −0.37 (±0.09 s.d.) across all subjects.

    “In the second session, the mean correlation of these prediction error signals from both models was −0.37 (±0.09 s.d.) across all subjects.”
    Quote in the paperSubstantial claim
  • Findingown data

    The model-based SPE provided a significantly better account for the BOLD signal in pIPS and latPFC than an unsigned RPE.

    “Our finding that our model-based SPE provided a significantly better account for the BOLD signal in pIPS and latPFC than an unsigned RPE (Figure S4) provides direct evidence for the distinction between these two signals.”
    Quote in the paperSubstantial claim
  • Findingown data

    The absolute value of the signed RPE correlated with a number of brain regions including an IPS locus anterior to where SPE correlates were found at p < 0.001 uncorrected.

    “This abs(RPE) signal correlated with a number of brain regions including an IPS locus anterior to where we found SPE correlates at p < 0.001 uncorrected (Figure S3).”
    Quote in the paperNot a substantial claim
  • May combine two claims. 15
  • Classified as probably not a substantial claim. 11

  • May combine two claims.

    Findingown data

    The parameter estimate for SPE in right pIPS correlated significantly with percent correct choices (r = 0.57, p = 0.013), but not in left or right lateral PFC.

    “We observed a significant correlation between the neural and the behavioral data of r = 0.57 (p = 0.013) in the right pIPS, but not in lateral PFC (left: r = 0.28, p = 0.27; right; r = 0.38, p = 0.12).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A direct comparison between SPE and abs(RPE) revealed a region of both posterior IPS that was significantly better explained by the SPE than by the abs(RPE) signal at p < 0.05 corrected, as well as a region of lateral PFC that showed a difference at p < 0.001 uncorrected.

    “However, a direct comparison between SPE and abs(RPE) revealed a region of both posterior IPS that was significantly better explained by the SPE than by the abs(RPE) signal at p < 0.05 corrected, as well as a region of lateral PFC that showed a difference at p < 0.001 uncorrected (Figure S4).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The model-free RL system's basic learning mechanisms are driven by a reward prediction error, as confirmed here.

    “The simpler of these two, associated with habits and model-free RL, has attracted a huge wealth of work, and there are ample studies (also confirmed here) elucidating its basic learning mechanisms driven by a reward prediction error.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The RPE is present predominantly in sub-cortical structures such as the striatum, and the SPE is present instead in dorsal cortical areas, in the parietal and frontal lobes.

    “Whereas the RPE is present predominantly in sub-cortical structures such as the striatum, appropriate to the rich input into this area from mid-brain dopaminergic neurons (Haber, 2003) known to broadcast this signal (Schultz, 1998; Schultz et al., 1997), the SPE was present instead in dorsal cortical areas, in the parietal and frontal lobes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In pIPS and latPFC, BOLD activation increased linearly with increasing SPEs, except in left IPS where the increase occurred only for the highest SPE; there was no systematic relationship between BOLD activation and RPE in these regions.

    “This reveals a linear increase in BOLD activation across trials with increasing SPEs, except for the left IPS, in which the increase in BOLD activation occurs only for trials with the highest SPE. In contrast, there is no such systematic relationship between BOLD activation and the RPE.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the first trial of session 2, 13 of 18 subjects chose the optimal action R in state 1, significantly more than chance (p < 0.05, sign-test, one-tailed).

    “Indeed, of all 18 subjects, 13 chose R (the optimal choice) and 5 chose L in state 1 in the very first trial of session 2 (p < 0.05, sign-test, one-tailed), indicating that their choice behavior cannot be completely explained by traditional model-free reward learning theory.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Evidence at least in rats indicates that both model-free and model-based mechanisms exist and adapt simultaneously.

    “with evidence at least in rats that both mechanisms exist and adapt simultaneously (Dickinson and Balleine, 2002)”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Dickinson and Balleine, 2002

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The dissociable SPE and RPE signals support the existence of two unique forms of learning signal in humans that may form the basis of distinct computational strategies for guiding behavior.

    “This finding supports the existence of two unique forms of learning signal in humans, which may form the basis of distinct computational strategies for guiding behavior.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Model-free learning focuses exclusively on predicting rewards without building a model of the environment and therefore learns nothing during a session with no rewarding outcomes.

    “this would be possible only with model-based, but not model-free learning, because the latter focuses exclusively on predicting rewards without building a model of the environment and therefore learns nothing during session 1.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The fMRI BOLD signal reflects in part the input into an area and intrinsic computations within it.

    “Under the view that the fMRI BOLD signal reflects in part the input into an area and intrinsic computations within it (Logothetis et al., 2001)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Logothetis et al., 2001 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    There is an impressive agreement from a wide variety of animal and human paradigms for the involvement of at least two systems in decision-making and control.

    “In conclusion, there is an impressive agreement from a wide variety of animal and human paradigms for the involvement of at least two systems in decision-making and control.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    We showed that the two error signals are computed in partially distinct brain areas and illustrated how human choice behavior may emerge through the combination of the systems.

    “In particular, we showed that the two error signals are computed in partially distinct brain areas and illustrated how human choice behavior may emerge through the combination of the systems.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The relative weight accorded to the two functions in determining the hybrid state-action valuations (and thus choice behavior) can change over the course of the free-choice scanning session (session 2).
    • The authors state this as an assumption: The two sets of state-action value estimates are combined according to a weighted average.
    • The authors state this as an assumption: Participants select actions stochastically according to probabilities determined by their state-action values through a softmax distribution.

  • May combine two claims.

    Interpretationown data

    SPE signal in right pIPS is predictive of subsequent successful choice behavior, whereas the same signal in the left lateral PFC is not.

    “Furthermore, it is interesting to note that SPE signal in right pIPS is predictive of subsequent successful choice behavior, whereas the same signal in the left lateral PFC is not.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Unregularized maximum likelihood estimators tend to be very noisy in individual subjects leading to very different and sometimes even outlying parameter estimates.

    “We estimated a single set of parameters for all participants because the unregularized maximum likelihood estimators tend to be very noisy in individual subjects leading to very different and sometimes even outlying parameter estimates.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    For the analysis about the consistency of SPE signal in pIPS and latPFC, independence of voxel selection was ensured by first identifying the cluster peaks in these regions for the SPE signal in the rewarded session 2.

    “For the analysis about the consistency of SPE signal in pIPS and latPFC we insure independence of voxel selection by first identifying the cluster peaks in these regions for the SPE signal in the rewarded session 2.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    A single set of parameters imposes a simple, but efficient regularization, which stabilizes the estimated model parameters.

    “A single set of parameters as frequently employed in our recent work (Daw et al., 2006; Gershman et al., 2009; Glascher et al., 2009) imposes a simple, but efficient regularization, which stabilizes the estimated model parameters.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Daw et al., 2006 source

      Not checked in the free run.

    • cites Gershman et al., 2009 source

      Not checked in the free run.

    • cites Glascher et al., 2009

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The statistical threshold was set to p < 0.05 family-wise error (FWE) corrected for the entire brain volume.

    “We set our statistical threshold to p < 0.05 family-wise error (FWE) corrected for the entire brain volume.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Low-level perceptual learning resulting in altered activation due to improvement in stimulus detection and discriminability was most likely minimized in our study and cannot account for the SPE signal found in pIPS.

    “Because our stimuli (fractal images, see Figure 1) were selected for maximal discriminability in terms of color scheme and shape, low-level perceptual learning resulting in altered activation due to improvement in stimulus detection and discriminability was most likely minimized in our study and cannot account for the SPE signal found in pIPS.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The data presented here, in particular the weighted combination of both models in the hybrid account, suggests that both learning mechanisms act together to produce effective action selection rather than dualistic processing of two learning modules that exert separate control over choice behavior.

    “However, the data presented here, in particular the weighted combination of both models in the hybrid account, suggests that both learning mechanisms act together to produce effective action selection rather than dualistic processing of two learning modules that exert separate control over choice behavior.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The relative weight accorded to the two functions in determining the hybrid state-action valuations (and thus choice behavior) can change over the course of the free-choice scanning session (session 2).
    • The authors state this as an assumption: The two sets of state-action value estimates are combined according to a weighted average.
    • The authors state this as an assumption: Participants select actions stochastically according to probabilities determined by their state-action values through a softmax distribution.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The lateral PFC has been proposed to contribute toward the implementation of model-based RL, possibly in the form of Bayesian belief states.

    “On these grounds the lateral PFC has been proposed to contribute toward the implementation of model-based RL, possibly in the form of Bayesian belief states (Samejima and Doya, 2007), an account consistent with our proposed role of this area in model-based reinforcement learning.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Samejima and Doya, 2007

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The resulting regressors for this kind of 'model-based fMRI' data analysis tend to perform poorly.

    “In addition, the resulting regressors for this kind of “model-based fMRI” data analysis tend to perform poorly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The RPE signals in the ventral striatum are consistent with many previous accounts.

    “This is consistent with many previous accounts (McClure et al., 2003; O’Doherty et al., 2003; Seymour et al., 2004).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites McClure et al., 2003 source

      Not checked in the free run.

    • cites O’Doherty et al., 2003

      Not checked in the free run.

    • cites Seymour et al., 2004 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Neural signals marking gross violations of expectations respond to incorrect prediction of affectively neutral stimuli, unlike dopamine-related prediction error signals which are largely reward-focused and associated with model-free RL.

    “Unlike the prediction error signals associated with dopamine activity, which are largely reward-focused and associated with model-free RL (Holroyd and Coles, 2002), these respond to incorrect prediction of affectively neutral stimuli.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Holroyd and Coles, 2002 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The presence of SPE signals in pIPS while subjects learn state transitions in the complete absence of reward suggests that pIPS is involved in pure state-learning, and not just in encoding value-based representations.

    “The finding that SPE signals are present in pIPS while subjects are learning state transitions even in the complete absence of reward (session 1 of our task), suggests that this region is involved also in pure state-learning, and not just in encoding value-based representations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The lateral prefrontal cortex, despite maintaining a similar representation of the SPE, may be concerned with integrating other learning signals too, and therefore does not exhibit the same clear link to subsequent choice behavior.

    “This underlines the importance of the state space representation that is built in the parietal cortex, and suggests that the lateral prefrontal cortex, despite maintaining a similar representation of the SPE, may be concerned with integrating other learning signals too, and therefore does not exhibit the same clear link to subsequent choice behavior.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The data for the plots of percent signal change in Figures 3 and 4 were extracted using a cross-validation leave-one-out procedure.

    “We extracted the data for the plots of percent signal change in Figures 3 and 4 using a cross-validation leave-one-out procedure: we re-estimated our second level analysis (repeated measures ANOVA, see above) 18 times always leaving out one subject.”
    Quote in the paperNot a substantial claim
26 of 26

  • The paper relies on this without support.

    A full characterization of the different learning properties of model-based and model-free mechanisms and how their outputs are integrated is missing.

    “a full characterization of their different learning properties and the way that their outputs are integrated to achieve better control is as yet missing.”

  • The authors state this as an assumption.

    Beliefs about transition probabilities are updated by error-driven model-based learning with a fixed learning rate, as assumed in the FORWARD model.

    “If participants’ beliefs about the transition probabilities were updated by error-driven model-based learning (with a fixed learning rate, as assumed in FORWARD)”

  • The authors state this as an assumption.

    Attention would be grabbed equally by unexpected reward delivery or omission, and more by either than by unexpected presentations of the less behaviorally salient visual state stimuli.

    “However, we might expect that attention would be grabbed equally by the delivery of unexpected rewards or omissions of reward, and certainly more by either of these than the unexpected presentations of the somewhat less behaviorally salient visual stimuli, which denote the different states in our task.”

  • The authors state this as an assumption.

    The relative weight accorded to the two functions in determining the hybrid state-action valuations (and thus choice behavior) can change over the course of the free-choice scanning session (session 2).

    “We assume that the relative weight accorded to the two functions in determining the hybrid state-action valuations (and thus choice behavior) can change over the course of the free-choice scanning session (session 2).”

  • The authors state this as an assumption.

    The two sets of state-action value estimates are combined according to a weighted average.

    “The model assumes that the two sets of state-action value estimates are combined according to a weighted average.”

  • The authors state this as an assumption.

    Participants select actions stochastically according to probabilities determined by their state-action values through a softmax distribution.

    “Each of the models additionally assumes that participants select actions stochastically according to probabilities determined by their state-action values through a softmax distribution: P(s,a)=exp(τ×Q(s,a))∑b=1nexp(τ×Q(s,b))”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Rewarded experience updates the model for stimulus-response-outcome associations when using a model-based learning strategy.

    “As conceptualized by computational studies2,41,46, rewarded experience updates the model for stimulus-response-outcome associations when using a model-based learning strategy.”

    From Dissociable roles of prefrontal plasticity in decision-making strategy and execution of habitual behavior (2026) · cites this paper as [41]

    The Experimental Procedures (FORWARD Learner; page unavailable) say that experience with state transitions updates a model of state–action–next-state probabilities. But the rewards used to evaluate actions were taken as given, not learned from rewarded experience; the Discussion also reports model-based learning during a non-rewarded session. The paper therefore supports updating a transition model, but not the claim that rewarded experience updates stimulus–response–outcome associations. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Two parallel forms of reinforcement learning, model-based and model-free (or value-free), correspond to goal-directed and habitual strategies and develop concurrently during behavior acquisition.

    “Supporting this, computational studies have proposed two parallel forms of reinforcement learning, model-based and model-free (or value-free) learning corresponding to goal-directed and habitual strategies, respectively, that develop concurrently during behavior acquisition2,41,42.”

    From Dissociable roles of prefrontal plasticity in decision-making strategy and execution of habitual behavior (2026) · cites this paper as [41]

    The Abstract describes model-based and model-free as two strategies and says both mechanisms can adapt simultaneously; the Discussion associates model-free learning with habits, while the Results report both processes influencing behavior. But the paper does not call model-free learning “value-free”: it explicitly says that model-free learning acquires action values. Its Results also specify that only the model-based system could usefully learn during the initial, unrewarded session, so concurrent learning does not describe every phase of acquisition. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • The dorsolateral striatum is more involved in stimulus-response learning and habitual control, the dorsomedial striatum is linked to action-outcome learning and goal-directed action, and ventral striatum to Pavlovian values.

    “For example, the dorsolateral striatum is thought to be more involved in stimulus-response learning and habitual control, the dorsomedial striatum has been linked to action-outcome learning and goal-directed action, and ventral striatum to Pavlovian values (Glascher et al., 2010, Liljeholm and O’Doherty, 2012, Mannella et al., 2013, O’Doherty et al., 2004).”

    From Active Inference, homeostatic regulation and adaptive behavioural control (2015) · cites this paper as Glascher et al., 2010

    The Discussion (page not available) tentatively links a goal-directed system to parts of the anterior medial striatum and a habitual system to dopaminergic-striatal pathways. It does not specifically assign action-outcome learning to dorsomedial striatum or stimulus-response learning and habits to dorsolateral striatum. The Results (page not available) find a reward-prediction-error signal in ventral striatum, not Pavlovian values. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • In the Gläscher et al. latent-learning task, appropriate initial behavior after utilities were revealed could only arise from model-based control.

    “Appropriate initial behavior in the task once the utilities were revealed could only arise from model-based control.”

    From Goals and Habits in the Brain (2013) · cites this paper as Gläscher et al., 2010

  • During the interregnum in the Gläscher et al. task, behavior was best fit by a hybrid model in which both systems exerted some control.

    “During the interregnum, behavior was best fit by a hybrid model in which both systems exerted some control.”

    From Goals and Habits in the Brain (2013) · cites this paper as Gläscher et al., 2010

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • FORWARD value equation uses an action instead of a successor value

    Minor

    The equation uses argmax, which returns the maximizing action, in the sum with reward. That conflicts with the stated computation of an expectation over the successor state's value; the equation would need the maximum value rather than the action that maximizes it.

  • Likelihood-ratio tests use the wrong degrees of freedom

    Minor

    The HYBRID model has five free parameters and each model it is compared with has two, so each likelihood-ratio test should have three degrees of freedom, not two. The reported p-value for the HYBRID-versus-SARSA test corresponds to the stated two degrees of freedom.

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P.
Affiliations
  • California Institute of Technology (US)
  • New York University (US)
  • Gatsby Computational Neuroscience Unit (GB)
Corresponding author
Gläscher, Jan
ORCID
Venue
Neuron
Publisher
Elsevier BV
Published
2010-05
Volume
66
Issue
4
Pages
585-595
ISSN
0896-6273 (ISSN-L)
DOI
10.1016/j.neuron.2010.04.016
PMC
PMC2895323
OpenAlex
W2149565728
Semantic Scholar
2910ee0ac40abb61dbdd0d38861129a54eae5070
Cited by
1,166
References
57
Citations by year
2026: 45 · 2025: 65 · 2024: 70 · 2023: 92 · 2022: 71 · 2021: 83 · 2020: 95 · 2019: 96 · 2018: 80 · 2017: 89 · 2016: 69 · 2015: 78 · 2014: 60 · 2013: 68 · 2012: 51
FWCI
33.46
Type
journal article
Language
en
Open access
Licence
Locations
  • Neuron (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • CaltechAUTHORS (California Institute of Technology) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • MPG.PuRe (Max Planck Society) (submittedVersion)
Funders
National Institute of Mental Health (R03MH075763)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
Keywords
model-based reinforcement learning, model-free reinforcement learning, reward prediction error, learning signals, intraparietal sulcus, lateral prefrontal cortex, ventral striatum, state transitions, human decision-making
MeSH terms
Adolescent, Adult, Choice Behavior, Female, Forecasting, Humans, Learning, Male, Models, Neurological, Psychomotor Performance, Reinforcement, Psychology, Research Design, Reward, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review