Adaptive learning and decision-making under uncertainty by metaplastic synapses guided by a surprise detection system
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Recent experiments have shown that animals and humans have a remarkable ability to adapt their learning rate according to the volatility of the environment. Yet the neural mechanism responsible for such adaptive learning has remained unclear. To fill this gap, we investigated a biophysically inspired, metaplastic synaptic model within the context of a well-studied decision-making network, in which synapses can change their rate of plasticity in addition to their efficacy according to a reward-based learning rule. We found that our model, which assumes that synaptic plasticity is guided by a novel surprise detection system, captures a wide range of key experimental findings and performs as well as a Bayes optimal model, with remarkably little parameter tuning. Our results further demonstrate the computational power of synaptic plasticity, and provide insights into the circuit-level computation which underlies adaptive decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · cited 3×
“…has long been studied in relation to the modulation of learning rates in reinforcement learning theory and Bayesian optimal learning theory ( Pearce and Hall, 1980 ; Adams and MacKay, 2007 ; Dayan et al., 2000 ; Gallistel et al., 2001 ; Courville et al., 2006 ; Yu and Dayan, 2005 ; Behrens…”
BackgroundSynaptic Plasticity and Memory: An Evaluation of the Hypothesis
Martin, S. J., Grimwood, P. D., & Morris, R. G. M. (2000) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.23.1.649 · Cited by 2,794 · cited 1×
“…and experimental studies strongly suggest that activity-dependent synaptic plasticity plays a crucial role in learning and adaptation in general ( Martin et al., 2000 ; Kandel et al., 2000 ; Dayan and Abbott, 2001 ), the neural mechanisms behind flexible learning, especially in the case of…”
BackgroundA theory of attention: Variations in the associability of stimuli with reinforcement
Mackintosh, N. J. (1975) · Psychological Review · doi:10.1037/h0076778 · Cited by 2,733
Synaptic Plasticity: Multiple Forms, Functions, and Mechanisms
Citri, A., & Malenka, R. C. (2008) · Neuropsychopharmacology · doi:10.1038/sj.npp.1301559 · Cited by 2,323 · cited 1×
“It is however known that there are various chemical cascade processes taking place in synapses that affect synaptic plasticity ( Citri and Malenka, 2008 ; Kotaleski and Blackwell, 2010 ).”
BackgroundRELATIVE AND ABSOLUTE STRENGTH OF RESPONSE AS A FUNCTION OF FREQUENCY OF REINFORCEMENT 1, 2
Herrnstein, R. J. (1961) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1961.4-267 · Cited by 2,281
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Uncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · cited 5×
“We also related our computation to the notions of unexpected and expected uncertainties, which have been suggested to be correlated with NE and Acetylcholine (Ach) release, respectively ( Yu and Dayan, 2005 ).”
BackgroundInfluentialThe importance of mixed selectivity in complex cognitive tasks
Rigotti, M., Barak, O., Warden, M. R., Wang, X.-J., Daw, N. D., Miller, E. K., & Fusi, S. (2013) · Nature · doi:10.1038/nature12160 · Cited by 1,639 · cited 2×
“In fact, it has been shown that introducing neurons that are randomly connected to neurons in the decision making network can solve context dependent decision-making tasks ( Rigotti et al., 2010 ; Barak et al., 2013 ).”
BackgroundProbabilistic decision making by slow reverberation in cortical circuits
Wang, X.-J. (2002) · Neuron · doi:10.1016/s0896-6273(02)01092-9 · Cited by 1,242
Tonic dopamine: Opportunity costs and the control of response vigor
Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0502-4 · Cited by 1,133 · cited 1×
“…well known that the phasic activity of dopaminergic neurons reflects a reward prediction error ( Schultz et al., 1997 ), while tonic dopamine levels may reflect reward rates ( Niv et al., 2007 ); these signals could also play crucial roles in our multiple timescales of reward integration process.”
BackgroundMatching Behavior and the Representation of Value in the Parietal Cortex
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004) · Science · doi:10.1126/science.1094765 · Cited by 1,030
Choice, uncertainty and value in prefrontal and cingulate cortex
Rushworth, M. F. S., & Behrens, T. E. J. (2008) · Nature Neuroscience · doi:10.1038/nn2066 · Cited by 866
The Cognitive Neuroscience of Human Memory Since H.M.
Squire, L. R., & Wixted, J. T. (2011) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-061010-113720 · Cited by 749 · cited 1×
“…areas may also allow flexible access of memory on different timescales, or hierarchical structure of contexts, if the rates of synaptic plasticity are similarly distributed amongst different brain areas, with memory information being transferred from one area to another ( Squire and Wixted, 2011 ).”
BackgroundRational regulation of learning dynamics by pupil-linked arousal systems
Nassar, M. R., Rumsey, K. M., Wilson, R. C., Parikh, K., Heasly, B., & Gold, J. I. (2012) · Nature Neuroscience · doi:10.1038/nn.3130 · Cited by 722 · cited 3×
“Other studies show that animals can change the integration timescale, or the learning rate, depending on the environment ( Behrens et al., 2007 ; Nassar et al., 2010 ; Nassar et al., 2012 ).”
BackgroundMethodDecision Making in Recurrent Neuronal Circuits
Wang, X.-J. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.034 · Cited by 679 · cited 3×
“We therefore assume that the goal of subjects in this case is to implement the matching law, which has previously been shown to be produced by the model under study ( Soltani and Wang, 2006 ; Fusi et al., 2007 ; Wang, 2008 ; Iigaya and Fusi, 2013 ).”
BackgroundMethodBayesian Online Changepoint Detection
Adams, R. P., & MacKay, D. J. C. (2007) · arXiv · doi:10.48550/arxiv.0710.3742 · Cited by 601
Cascade Models of Synaptically Stored Memories
Fusi, S., Drew, P. J., & Abbott, L. F. (2005) · Neuron · doi:10.1016/j.neuron.2005.02.001 · Cited by 569 · cited 8×
“While the original cascade model ( Fusi et al., 2005 ) is likely to be able to naturally encode multiple timescales of reward information ( Corrado et al., 2005 ; Fusi et al., 2007 ; Bernacchia et al., 2011 ; Iigaya et al., 2013 ; Iigaya, 2013 ), such active adaptation may also require external…”
BackgroundInfluentialWixted, J. T., & Ebbesen, E. B. (1991) · Psychological Science · doi:10.1111/j.1467-9280.1991.tb00175.x · Cited by 513 · cited 1×
“The resulting model can efficiently capture the widely-observed power-law forgetting curve ( Wixted and Ebbesen, 1991 ).”
BackgroundLearning and selective attention
Dayan, P., Kakade, S., & Montague, P. R. (2000) · Nature Neuroscience · doi:10.1038/81504 · Cited by 495 · cited 2×
“This is consistent with both experimental findings and the predictions of optimal Bayesian models ( Nassar et al., 2010 ; Dayan et al., 2000 ).”
BackgroundResultDYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS
Lau, B., & Glimcher, P. W. (2005) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.2005.110-04 · Cited by 485
Context, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386 · cited 1×
“Second, although we assumed that the surprise signal would reset most of the accumulated evidence when reward-harvesting performance deterio-rates, in many cases it would be better to keep accumulated evidence, such as to form distinct ’con-texts’ ( Gershman et al., 2010 ; Lloyd and Leslie, 2013 ).”
BackgroundRescorla, R. A. (2004) · Learning & Memory · doi:10.1101/lm.77504 · Cited by 380
Hayden, B. Y., Heilbronner, S. R., Pearson, J. M., & Platt, M. L. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.4652-10.2011 · Cited by 379 · cited 2×
“…multiple timescales of reward information ( Corrado et al., 2005 ; Fusi et al., 2007 ; Bernacchia et al., 2011 ; Iigaya et al., 2013 ; Iigaya, 2013 ), such active adaptation may also require external guidance, such as in the form of a surprise signal ( Hayden et al., 2011 ; Garvert et al., 2015 ).”
BackgroundNorepinephrine triggers release of glial ATP to increase postsynaptic efficacy
Gordon, G. R. J., Baimoukhametova, D. V., Hewitt, S. A., Rajapaksha, W. R. A. K. J. S., Fisher, T. E., & Bains, J. S. (2005) · Nature Neuroscience · doi:10.1038/nn1498 · Cited by 374 · cited 1×
“Also, there is a large amount of experimental evidence that Ach can enhance synaptic plasticity ( Gordon et al., 2005 ; Mitsushima et al., 2013 ).”
BackgroundA reservoir of time constants for memory traces in cortical neurons
Bernacchia, A., Seo, H., Lee, D., & Wang, X.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2752 · Cited by 292 · cited 2×
“Experimental studies suggest that they integrate reward history on multiple timescales rather than a single timescale ( Corrado et al., 2005 ; Fusi et al., 2007 ; Bernacchia et al., 2011 ).”
BackgroundNeocortical Pyramidal Cells Respond as Integrate-and-Fire Neurons to In Vivo–Like Input Currents
Rauch, A., La Camera, G., Lüscher, H.-R., Senn, W., & Fusi, S. (2003) · Journal of Neurophysiology · doi:10.1152/jn.00293.2003 · Cited by 264 · cited 2×
“…or collection of multiple exponents, rather than a single exponent ( Thorson and Biederman-Thorson, 1974 ; Ulanovsky et al., 2004 ; Corrado et al., 2005 ; Fusi et al., 2007 ; Kording et al., 2007 ; Wark et al., 2009 ; Lundstrom et al., 2010 ; Rauch et al., 2003 ; Pozzorini et al., 2013 ).”
BackgroundLINEAR-NONLINEAR-POISSON MODELS OF PRIMATE CHOICE DYNAMICS
Corrado, G. S., Sugrue, L. P., Seung, H. S., & Newsome, W. T. (2005) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.2005.23-05 · Cited by 206
Fusi, S., Asaad, W. F., Miller, E. K., & Wang, X.-J. (2007) · Neuron · doi:10.1016/j.neuron.2007.03.017 · Cited by 199 · cited 9×
“…or collection of multiple exponents, rather than a single exponent ( Thorson and Biederman-Thorson, 1974 ; Ulanovsky et al., 2004 ; Corrado et al., 2005 ; Fusi et al., 2007 ; Kording et al., 2007 ; Wark et al., 2009 ; Lundstrom et al., 2010 ; Rauch et al., 2003 ; Pozzorini et al., 2013 ).”
BackgroundMethodInfluentialSynaptic evidence for the efficacy of spaced learning
Kramár, E. A., Babayan, A. H., Gavin, C. F., Cox, C. D., Jafari, M., Gall, C. M., Rumbaugh, G., & Lynch, G. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1120700109 · Cited by 192 · cited 1×
“Those processes, in general, operate on a wide range of timescales ( Zhang et al., 2012 ; Kramar et al., 2012 ).”
BackgroundRigotti, M. (2010) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2010.00024 · Cited by 184 · cited 1×
“In fact, it has been shown that introducing neurons that are randomly connected to neurons in the decision making network can solve context dependent decision-making tasks ( Rigotti et al., 2010 ; Barak et al., 2013 ).”
BackgroundComputational design of enhanced learning protocols
Zhang, Y., Liu, R.-Y., Heberton, G. A., Smolen, P., Baxter, D. A., Cleary, L. J., & Byrne, J. H. (2012) · Nature Neuroscience · doi:10.1038/nn.2990 · Cited by 80 · cited 1×
“Those processes, in general, operate on a wide range of timescales ( Zhang et al., 2012 ; Kramar et al., 2012 ).”
BackgroundNeural mechanism for stochastic behaviour during a competitive game
Soltani, A., Lee, D., & Wang, X.-J. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.05.044 · Cited by 77 · cited 11×
“For instance, animals can react rapidly to abrupt step-like changes in environments ( Behrens et al., 2007 ; Rushworth and Behrens, 2008 ; Soltani et al., 2006 ; Nassar et al., 2010 ; Nassar et al., 2012 ; Neiman and Loewenstein, 2013 ; McGuire et al., 2014 ), or change their strategies dynamically…”
BackgroundMethodInfluential
Cited by
Prefrontal cortex as a meta-reinforcement learning system
Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., & Botvinick, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0147-8 · Cited by 596
Adaptive learning under expected and unexpected uncertainty
Soltani, A., & Izquierdo, A. (2019) · Nature reviews. Neuroscience · doi:10.1038/s41583-019-0180-y · Cited by 286
Iigaya, K., Fonseca, M. S., Murakami, M., Mainen, Z. F., & Dayan, P. (2018) · Nature Communications · doi:10.1038/s41467-018-04840-2 · Cited by 120
Self-citationBalancing new against old information: the role of puzzlement surprise in learning
Faraji, M., Preuschoff, K., & Gerstner, W. (2018) · Neural Computation · doi:10.1162/neco_a_01025 · Cited by 68
“…Li, Daw, & Schoenbaumn, 2012), risk prediction errors (Preuschoff & Bossaerts, 2007; Preuschoff, Quartz, & Bossaerts, 2008), uncertainty-based jump detection (Nassar, Wilson, Heasly, & Gold, 2010; Payzan-Nestour & Bossaerts, 2011), or multi-timescale reward estimator comparison (Iigaya, 2016).”
Metadata
- Authors
- Iigaya, K.
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2016-08-09
- Volume
- 5
- ISSN
- 2050-084X (ISSN-L)
- OpenAlex
- W2514548734
- Cited by
- 43
- References
- 32
- FWCI
- 3.13
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.