Metaplasticity as a Neural Substrate for Adaptive Learning and Choice under Uncertainty
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Summary Value-based decision making often involves integration of reward outcomes over time, but this becomes considerably more challenging if reward assignments on alternative options are probabilistic and non-stationary. Despite the existence of various models for optimally integrating reward under uncertainty, the underlying neural mechanisms are still unknown. Here we propose that reward-dependent metaplasticity (RDMP) can provide a plausible mechanism for both integration of reward under uncertainty and estimation of uncertainty itself. We show that a model based on RDMP can robustly perform the probabilistic reversal learning task via dynamic adjustment of learning based on reward feedback, while changes in its activity signal unexpected uncertainty. The model predicts time-dependent and choice-specific learning rates which strongly depend on reward history. Key predictions from this model were confirmed with behavioral data from non-human primates. Overall, our results suggest that metaplasticity can provide a neural substrate for adaptive learning and choice under uncertainty.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Aston-Jones, G., & Cohen, J. D. (2005) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.28.061604.135709 · Cited by 4,472
Getting formal with dopamine and reward
Schultz, W. (2002) · Neuron · doi:10.1016/s0896-6273(02)00967-4 · Cited by 2,574
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Uncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · cited 8×
“…on different timescales (Behrens et al., 2007; Mathys et al., 2011; McGuire et al., 2014; PayzanLeNestour and Bossaerts, 2011; Wilson et al., 2013; Yu and Dayan, 2005) or assume the correct structure of the task at hand (Costa et al., 2015; Daw et al., 2002; Hampton et al., 2006; Jang et al.,…”
InfluentialMultiple dopamine functions at different time courses
Schultz, W. (2007) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.28.061604.135722 · Cited by 1,415 · cited 1×
“Second, DA affects neural processes at multiple timescales (Schultz, 2007).”
Synaptic theory of working memory
Mongillo, G., Barak, O., & Tsodyks, M. (2008) · Science · doi:10.1126/science.1150769 · Cited by 1,253 · cited 1×
“Together, our work highlights the overlooked power of metaplastic synaptic mechanisms for solving complex cognitive problems (Mongillo et al., 2008).”
Optimal decision making and the anterior cingulate cortex
Kennerley, S. W., Walton, M. E., Behrens, T. E. J., Buckley, M. J., & Rushworth, M. F. S. (2006) · Nature Neuroscience · doi:10.1038/nn1724 · Cited by 948 · cited 3×
“…has shown that ACC-lesioned animals were unable to sustain a response that yielded reward and displayed a reduction in the time constant of reward integration leading to impaired learning, especially when reward probabilities were low (Kennerley et al., 2006; Rushworth and Behrens, 2008).”
Cools, R. (2001) · Cerebral Cortex · doi:10.1093/cercor/11.12.1136 · Cited by 906
Orbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885
Choice, uncertainty and value in prefrontal and cingulate cortex
Rushworth, M. F. S., & Behrens, T. E. J. (2008) · Nature Neuroscience · doi:10.1038/nn2066 · Cited by 866
Rational regulation of learning dynamics by pupil-linked arousal systems
Nassar, M. R., Rumsey, K. M., Wilson, R. C., Parikh, K., Heasly, B., & Gold, J. I. (2012) · Nature Neuroscience · doi:10.1038/nn.3130 · Cited by 722 · cited 5×
“Other models deal with reward uncertainty by detecting changes in the environment to adapt learning (Costa et al., 2015; Jang et al., 2015; McGuire et al., 2014; Nassar et al., 2010, 2012).”
InfluentialA Bayesian foundation for individual learning under uncertainty
Mathys, C. (2011) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2011.00039 · Cited by 655
Cascade Models of Synaptically Stored Memories
Fusi, S., Drew, P. J., & Abbott, L. F. (2005) · Neuron · doi:10.1016/j.neuron.2005.02.001 · Cited by 569 · cited 3×
“Generally speaking, any model that exhibits synaptic changes without changes in synaptic efficacy can capture a form of metaplasticity (Fusi et al., 2005).”
InfluentialHampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498
Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain
Niv, Y., Edlund, J. A., Dayan, P., & O'Doherty, J. P. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.5498-10.2012 · Cited by 356
A reservoir of time constants for memory traces in cortical neurons
Bernacchia, A., Seo, H., Lee, D., & Wang, X.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2752 · Cited by 292 · cited 2×
“The multiple time constants for updates and hierarchical structures in aforementioned models might rely on the observed reservoir of reward memory time constants (Bernacchia et al., 2011) or could be a reflection of metaplasticity proposed here, since metaplastic synapses contain different internal timescales.”
Self-citationRisk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings
Payzan-LeNestour, E., & Bossaerts, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001048 · Cited by 267 · cited 5×
“…properly incorporate reward feedback (Bayesian models) (Behrens et al., 2007; Courville et al., 2006; Daw et al., 2005, 2006; Hampton et al., 2006; Payzan-LeNestour and Bossaerts, 2011; Yu and Dayan, 2005); and (3) to detect changes in the environment to adapt learning accordingly (approximate…”
InfluentialMeta-learning in Reinforcement Learning
Schweighofer, N., & Doya, K. (2003) · Neural Networks · doi:10.1016/s0893-6080(02)00228-9 · Cited by 261 · cited 3×
“reward feedback: (1) to determine the optimal learning rates based on optimization or on the level of uncertainty in a given environment (RL models) (Doya, 2002; Schweighofer and Doya, 2003; Preuschoff and Bossaerts, 2007); (2) to identify the correct model of the world, which includes the amount of uncertainty to pro…”
Krugel, L. K., Biele, G., Mohr, P. N. C., Li, S.-C., & Heekeren, H. R. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0905191106 · Cited by 229 · cited 1×
“Indeed, it has been experimentally demonstrated that humans are able to adjust their learning or learning rates based on the reward uncertainty and volatility in a given environment (Behrens et al., 2007; Krugel et al., 2009; Nassar et al., 2010).”
Reversal learning and dopamine: a Bayesian perspective
Costa, V. D., Tran, V. L., Turchi, J., & Averbeck, B. B. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.1989-14.2015 · Cited by 170
Rygula, R., Walker, S. C., Clarke, H. F., Robbins, T. W., & Roberts, A. C. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.2631-10.2010 · Cited by 136 · cited 2×
“To test our model, we simulated choice behavior during a dynamic learning and decision-making task known as probabilistic reversal learning (PRL), which has been extensively used to study adaptive learning and decision making in health and disease (Cools et al., 2001; Costa et al., 2015; Rudebeck et al., 2013; Rygula…”
Adding prediction risk to the theory of reward learning
PREUSCHOFF, K., & BOSSAERTS, P. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.005 · Cited by 133 · cited 2×
“reward feedback: (1) to determine the optimal learning rates based on optimization or on the level of uncertainty in a given environment (RL models) (Doya, 2002; Schweighofer and Doya, 2003; Preuschoff and Bossaerts, 2007); (2) to identify the correct model of the world, which includes the amount of uncertainty to pro…”
Neural mechanism for stochastic behaviour during a competitive game
Soltani, A., Lee, D., & Wang, X.-J. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.05.044 · Cited by 77 · cited 2×
“As we have shown before, the decision on every trial only depends on the overall difference in the output of the two value-encoding pools (Soltani andWang, 2006, 2010; Soltani et al., 2006).”
Self-citationScaling prediction errors to reward variability benefits error-driven learning in humans
Diederen, K. M. J., & Schultz, W. (2015) · Journal of Neurophysiology · doi:10.1152/jn.00483.2015 · Cited by 69 · cited 1×
“A few studies have shown that the learning rates sharply increase and then decay when a change point in reward statistics occurs (Nassar et al., 2010; Diederen and Schultz, 2015).”
Cited by
Adaptive learning under expected and unexpected uncertainty
Soltani, A., & Izquierdo, A. (2019) · Nature reviews. Neuroscience · doi:10.1038/s41583-019-0180-y · Cited by 286
Self-citationMedial and orbital frontal cortex in decision-making and flexible behavior
Klein-Flügge, M. C., Bongioanni, A., & Rushworth, M. F. S. (2022) · Neuron · doi:10.1016/j.neuron.2022.05.022 · Cited by 244
A model for learning based on the joint estimation of stochasticity and volatility
Piray, P., & Daw, N. D. (2021) · Nature Communications · doi:10.1038/s41467-021-26731-9 · Cited by 153
Anterior Cingulate Cortex and the Control of Dynamic Behavior in Primates
Monosov, I. E., Haber, S. N., Leuthardt, E. C., & Jezzini, A. (2020) · Current Biology · doi:10.1016/j.cub.2020.10.009 · Cited by 118
Flexible combination of reward information across primates
Farashahi, S., Donahue, C. H., Hayden, B. Y., Lee, D., & Soltani, A. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0714-3 · Cited by 112
Self-citation
Metadata
- Authors
- Farashahi, S., Donahue, C. H., Khorsand, P., Seo, H., Lee, D., & Soltani, A.
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2017-04
- Volume
- 94
- Issue
- 2
- Pages
- 401-414.e6
- ISSN
- 0896-6273 (ISSN-L)
- OpenAlex
- W2606776585
- Cited by
- 101
- References
- 25
- FWCI
- 6.44
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.