Goal-Directed Decision Making with Spiking Neurons
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Behavioral and neuroscientific data on reward-based decision making point to a fundamental distinction between habitual and goal-directed action selection. The formation of habits, which requires simple updating of cached values, has been studied in great detail, and the reward prediction error theory of dopamine function has enjoyed prominent success in accounting for its neural bases. In contrast, the neural circuit mechanisms of goal-directed decision making, requiring extended iterative computations to estimate values online, are still unknown. Here we present a spiking neural network that provably solves the difficult online value estimation problem underlying goal-directed decision making in a near-optimal way and reproduces behavioral as well as neurophysiological experimental data on tasks ranging from simple binary choice to sequential decision making. Our model uses local plasticity rules to learn the synaptic weights of a simple neural network to achieve optimal performance and solves one-step decision-making tasks, commonly considered in neuroeconomics, as well as more challenging sequential decision-making tasks within 1 s. These decision times, and their parametric dependence on task parameters, as well as the final choice probabilities match behavioral data, whereas the evolution of neural activities in the network closely mimics neural responses recorded in frontal cortices during the execution of such tasks. Our theory provides a principled framework to understand the neural underpinning of goal-directed decision making and makes novel predictions for sequential decision-making tasks with multiple rewards. SIGNIFICANCE STATEMENT Goal-directed actions requiring prospective planning pervade decision making, but their circuit-level mechanisms remain elusive. We show how a model circuit of biologically realistic spiking neurons can solve this computationally challenging problem in a novel way. The synaptic weights of our network can be learned using local plasticity rules such that its dynamics devise a near-optimal plan of action. By systematically comparing our model results to experimental data, we show that it reproduces behavioral decision times and choice probabilities as well as neural responses in a rich set of tasks. Our results thus offer the first biologically realistic account for complex goal-directed decision making at a computational, algorithmic, and implementational level.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · cited 1×
“Research in animal learning and behavioral neuroscience has given rise to the view that reward-based decision making is governed by (at least) two distinct strategies: a habit system, which relies on cached associations between actions or situations and their long-run future values, and a goal-directed system, which i…”
BackgroundThe time course of perceptual choice: The leaky, competing accumulator model.
Usher, M., & McClelland, J. L. (2001) · Psychological Review · doi:10.1037/0033-295x.108.3.550 · Cited by 2,152 · cited 1×
“Previous models relied on stochasticity in the inputs to the network, even for non-stochastic stimuli, and the fact that this stochasticity needs to be integrated out over time (Usher and McClelland, 2001; Smith and Ratcliff, 2004).”
MethodDissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 1×
“In particular, temporal-difference algorithms of model-free learning account for both behavioral and neuroimaging data regarding habitbased decision making (O’Doherty et al., 2004; Seymour et al., 2004).”
MethodFunctional role of the supplementary and pre-supplementary motor areas
Nachev, P., Kennard, C., & Husain, M. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2478 · Cited by 1,928
Reverse replay of behavioural sequences in hippocampal place cells during the awake state
Foster, D. J., & Wilson, M. A. (2006) · Nature · doi:10.1038/nature04587 · Cited by 1,727 · cited 2×
“Reverse replay has only been observed in the hippocampus (Foster and Wilson, 2006; Diba and Buzsa´ki, 2007), and it remains to be tested whether, for example, it also exists in PFC or whether there are anti-Hebbian forms of plasticity operating there.”
BackgroundThe importance of mixed selectivity in complex cognitive tasks
Rigotti, M., Barak, O., Warden, M. R., Wang, X.-J., Daw, N. D., Miller, E. K., & Fusi, S. (2013) · Nature · doi:10.1038/nature12160 · Cited by 1,639 · cited 1×
“We consider neurons with mixed selectivity, a phenomenon widely observed in prefrontal cortex (PFC) and important for solving complex cognitive tasks (Rigotti et al., 2013).”
BackgroundNeurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598 · cited 10×
“Theoveralltimecoursesof neuronsweresimilartothosefoundinOFC,withonekeydiffer-ence:neuronscodingfortheunchosenoptionBdidnotshowan initialincreaseinactivity(Fig.3 C , purple lines).”
BackgroundResultInfluentialSelf-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564 · cited 1×
“…goal-directeddecisionmaking,suchastheventralstriatum(Pen-nartzetal.,2004;Lansinketal.,2008),thePFC(Eustonetal., 2007;Peyracheetal.,2009),andthehippocampus(Fosterand Wilson,2006;Guptaetal.,2010),mayalsoservetoreinforceand consolidateinternalmodelsoftheenvironment(Lin,1992;Mnih etal.,2015).”
Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition
Gerstner, W., Kistler, W. M., Naud, R., & Paninski, L. (2014) · doi:10.1017/cbo9781107447615 · Cited by 1,515 · cited 2×
“For biological plausibility, we used a canonical, phenomenological single-neuron model, the spike–response model (Gerstner et al., 2014), or, equivalently, the generalized linear model (Pillow et al., 2008; which has been demonstrated to be an accurate predictor of neural responses in a wide…”
MethodThe Complexity of Markov Decision Processes
Papadimitriou, C. H., & Tsitsiklis, J. N. (1987) · Mathematics of Operations Research · doi:10.1287/moor.12.3.441 · Cited by 1,329 · cited 1×
“Thus, there is a strongly nonlinear, recursive relationship between the optimal values of different actions (formalized by the so-called Bellman optimality equation; Bellman, 1957), which makes the computation of optimal values challenging (Papadimitriou and Tsitsiklis, 1987).”
BackgroundForward and reverse hippocampal place-cell sequences during ripples
Diba, K., & Buzsáki, G. (2007) · Nature Neuroscience · doi:10.1038/nn1961 · Cited by 1,205
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
Psychology and neurobiology of simple decisions
Smith, P. L., & Ratcliff, R. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.01.006 · Cited by 1,051
A Recurrent Network Mechanism of Time Integration in Perceptual Decisions
Wong, K.-F., & Wang, X.-J. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.3733-05.2006 · Cited by 1,029 · cited 1×
“Inearliermodels,lateralinhibition occurred at the level of the accumulated decision variables (Bo-gacz et al., 2006; Wong and Wang, 2006).”
BackgroundSpontaneous Cortical Activity Reveals Hallmarks of an Optimal Internal Model of the Environment
Berkes, P., Orbán, G., Lengyel, M., & Fiser, J. (2011) · Science · doi:10.1126/science.1195870 · Cited by 782
Self-citationReplay of rule-learning related neural patterns in the prefrontal cortex during sleep
Peyrache, A., Khamassi, M., Benchenane, K., Wiener, S. I., & Battaglia, F. P. (2009) · Nature Neuroscience · doi:10.1038/nn.2337 · Cited by 767
Hippocampal replay is not a simple function of experience
Gupta, A. S., van der Meer, M. A. A., Touretzky, D. S., & Redish, A. D. (2010) · Neuron · doi:10.1016/j.neuron.2010.01.034 · Cited by 733
Changes of mind in decision-making
Resulaj, A., Kiani, R., Wolpert, D. M., & Shadlen, M. N. (2009) · Nature · doi:10.1038/nature08275 · Cited by 678
Statistically optimal perception and learning: from behavior to neural representations
Fiser, J., Berkes, P., Orbán, G., & Lengyel, M. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.01.003 · Cited by 672 · cited 2×
“…and algorithmically attractive, especially in light of the converging evidence that cortical circuits may naturally perform probabilistic inference (Fiser et al., 2010), the neural instantiations suggested so far relied on two particularly speculative assumptions (Solway and Botvinick, 2012): they…”
BackgroundSelf-citationTemporal difference models describe higher-order learning in humans
Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S. (2004) · Nature · doi:10.1038/nature02581 · Cited by 605 · cited 1×
“In particular, temporal-difference algorithms of model-free learning account for both behavioral and neuroimaging data regarding habitbased decision making (O’Doherty et al., 2004; Seymour et al., 2004).”
MethodFast-Forward Playback of Recent Memory Sequences in Prefrontal Cortex During Sleep
Euston, D. R., Tatsuno, M., & McNaughton, B. L. (2007) · Science · doi:10.1126/science.1148979 · Cited by 586 · cited 1×
“…goal-directeddecisionmaking,suchastheventralstriatum(Pen-nartzetal.,2004;Lansinketal.,2008),thePFC(Eustonetal., 2007;Peyracheetal.,2009),andthehippocampus(Fosterand Wilson,2006;Guptaetal.,2010),mayalsoservetoreinforceand…”
Confidence in value-based choice
De Martino, B., Fleming, S. M., Garrett, N., & Dolan, R. J. (2013) · Nature Neuroscience · doi:10.1038/nn.3279 · Cited by 534 · cited 1×
“…hypothesize that the accumulation of spike count differences underlying the final decision takes place in ventromedial prefrontal cortex, which receives input from OFC (Rushworth et al., 2012) and has been implicated in the comparison of options (Wunderlich et al., 2012; De Martino et al., 2013).”
BackgroundCoordinated Reactivation of Distributed Memory Traces in Primate Neocortex
Hoffman, K. L., & McNaughton, B. L. (2002) · Science · doi:10.1126/science.1073538 · Cited by 420
Buesing, L., Bill, J., Nessler, B., & Maass, W. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002211 · Cited by 384 · cited 1×
“Whereas belief propagation has been often evoked as the algorithmic basis of how the cortex performs probabilistic inference (Lochmann and Deneve, 2011), recent results suggest that the cortex may, in fact, implement other kinds of inference algorithms (Fiser et al., 2010; Berkes et al., 2011; Buesing et al., 2011; He…”
BackgroundSpeed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346
Mapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310
Valuation and decision-making in frontal cortex: one or many serial or parallel systems?
Rushworth, M. F., Kolling, N., Sallet, J., & Mars, R. B. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.04.011 · Cited by 309
Reward value comparison via mutual inhibition in ventromedial prefrontal cortex
Strait, C. E., Blanchard, T. C., & Hayden, B. Y. (2014) · Neuron · doi:10.1016/j.neuron.2014.04.032 · Cited by 294 · cited 1×
“This representation was achieved by eventually only one neuron remaining active in each group representing a state (because the optimal action was unique), through competition mediated by lateral inhibition within the group (Strait et al., 2014).”
BackgroundPennartz, C. M. A., Lee, E., Verheul, J., Lipa, P., Barnes, C. A., & McNaughton, B. L. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.0575-04.2004 · Cited by 282 · cited 1×
“Off-line replay of experience, during periods of rest or sleep, as observed throughout the neocortex (Hoffman and McNaughton, 2002) and, in particular, in multiple brain areas implicated in goal-directed decision making, such as the ventral striatum (Pennartz et al., 2004; Lansink et al., 2008), the PFC (Euston et al.”
BackgroundA normative perspective on motivation
NIV, Y., JOEL, D., & DAYAN, P. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.06.010 · Cited by 251
Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans
Mushiake, H., Saito, N., Sakamoto, K., Itoyama, Y., & Tanji, J. (2006) · Neuron · doi:10.1016/j.neuron.2006.03.045 · Cited by 249 · cited 4×
“Specifically, performing a state–action transition activates the corresponding neurons in the network (Mushiake et al., 2006) and sets the rate of the external input, making this information available to the synapse in the form of presynaptic and postsynaptic activities.”
BackgroundInfluentialPreferential Reactivation of Motivationally Relevant Information in the Ventral Striatum
Lansink, C. S., Goltstein, P. M., Lankelma, J. V., Joosten, R. N. J. M. A., McNaughton, B. L., & Pennartz, C. M. A. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1054-08.2008 · Cited by 168
Purposive behavior and cognitive mapping: a neural network model
Schmajuk, N. A., & Thieme, A. D. (1992) · Biological Cybernetics · doi:10.1007/bf00201023 · Cited by 91
Cited by
Learning task-state representations
Niv, Y. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0470-8 · Cited by 356
With or without you: predictive coding and Bayesian inference in the brain
Aitchison, L., & Lengyel, M. (2017) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.08.010 · Cited by 308
“Similarly, the computational goal of reinforcement learning can be achieved by several algorithms that do not compute and represent prediction errors per se [51, 52], and whether prediction error-based or these other algorithms should be used depends on environmental and neural constraints [53].”
BackgroundSelf-citationOrbitofrontal Cortex: A Neural Circuit for Economic Decisions
Padoa-Schioppa, C., & Conen, K. E. (2017) · Neuron · doi:10.1016/j.neuron.2017.09.031 · Cited by 306
A review of inflammatory mechanism in airway diseases
Aghasafari, P., George, U., & Pidaparti, R. (2019) · Inflammation Research · doi:10.1007/s00011-018-1191-2 · Cited by 300
Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits
Payeur, A., Guerguiev, J., Zenke, F., Richards, B. A., & Naud, R. (2021) · Nature Neuroscience · doi:10.1038/s41593-021-00857-x · Cited by 292
Metadata
- Authors
- Friedrich, J., & Lengyel, M.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2016-02-03
- Volume
- 36
- Issue
- 5
- Pages
- 1529-1546
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W2255822949
- Cited by
- 57
- References
- 33
- FWCI
- 4.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.