CoChat
Sign Up
Cited by 153Memory and Neural MechanismsNeural and Behavioral Psychology StudiesNeural dynamics and brain function

A model for learning based on the joint estimation of stochasticity and volatility

Open access
Piray, Payam, Daw, Nathaniel D.·2021·Nature Communications
DOI 10.1038/s41467-021-26731-9PMC PMC8592992

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Previous research has stressed the importance of uncertainty for controlling the speed of learning, and how such control depends on the learner inferring the noise properties of the environment, especially volatility: the speed of change. However, learning rates are jointly determined by the comparison between volatility and a second factor, moment-to-moment stochasticity. Yet much previous research has focused on simplified cases corresponding to estimation of either factor alone. Here, we introduce a learning model, in which both factors are learned simultaneously from experience, and use the model to simulate human and animal data across many seemingly disparate neuroscientific and behavioral phenomena. By considering the full problem of joint estimation, we highlight a set of previously unappreciated issues, arising from the mutual interdependence of inference about volatility and stochasticity. This interdependence complicates and enriches the interpretation of previous results, such as pathological learning in individuals with anxiety and following amygdala damage.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-021-26731-9

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Real-world agents need to detect and distinguish dynamic changes in both volatility and observation noise at the same time.

    “While existing modelling approaches have focused on either accounting for changes in volatility (Behrens et al., 2007; Mathys et al., 2011; Piray & Daw, 2020) or block-wise changes in stochasticity or noise (Lee et al., 2020; Nassar et al., 2010, 2012, 2016), it has recently been pointed out that real-world agents need to able to detect (and distinguish) dynamic changes in both at the same time (”

    From The generalized Hierarchical Gaussian Filter (2026) · cites this paper as Piray & Daw, 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 67

  1. [1]Dayan, P. & Long, T. Statistical Models of Conditioning. In Advances in Neural Information Processing Systems 10 (eds, Jordan, M., Kearns, M. & Solla, S.) 117–123 (MIT Press, 1998).
  2. [2]Dayan P Kakade S Montague PR Learning and selective attention Nat. Neurosci. 2000 3 1218 1223 11127841
  3. [3]Courville AC Daw ND Touretzky DS Bayesian theories of conditioning in a changing world Trends Cogn. Sci. (Regul. Ed.) 2006 10 294 300 16793323
  4. [4]Daunizeau J Observing the observer (I): meta-bayesian models of learning and decision-making PLoS ONE 2010 5 e15554 21179480 PMC3001878
  5. [7]Nassar MR Rational regulation of learning dynamics by pupil-linked arousal systems Nat. Neurosci. 2012 15 1040 1046 22660479 PMC3386464
  6. [8]Iglesias S Hierarchical prediction errors in midbrain and basal forebrain during sensory learning Neuron 2013 80 519 530 24139048
  7. [9]McGuire JT Nassar MR Gold JI Kable JW Functionally dissociable influences on learning rate in a dynamic environment Neuron 2014 84 870 881 25459409 PMC4437663
  8. [11]Mathys C Daunizeau J Friston KJ Stephan KE A bayesian foundation for individual learning under uncertainty Front Hum. Neurosci. 2011 5 39 21629826 PMC3096853
  9. [12]Piray P Daw ND A simple model for learning in volatile environments PLoS Comput. Biol. 2020 16 e1007963 32609755 PMC7329063
  10. [13]Browning M Behrens TE Jocham G O’Reilly JX Bishop SJ Anxious individuals have difficulty learning the causal statistics of aversive environments Nat. Neurosci. 2015 18 590 596 25730669 PMC4644067
  11. [14]Brazil IA Mathys CD Popma A Hoppenbrouwers SS Cohn MD Representational uncertainty in the brain during threat conditioning and the link with psychopathic traits Biol. Psychiatry Cogn. Neurosci. Neuroimaging 2017 2 689 695 29560903
  12. [15]Farashahi S Metaplasticity as a neural substrate for adaptive learning and choice under uncertainty Neuron 2017 94 401 414.e6 28426971 PMC5515734
  13. [17]Powers AR Mathys C Corlett PR Pavlovian conditioning-induced hallucinations result from overweighting of perceptual priors Science 2017 357 596 600 28798131 PMC5802347
  14. [18]Katthagen T Modeling subjective relevance in schizophrenia and its relation to aberrant salience PLoS Comput. Biol. 2018 14 e1006319 30096179 PMC6105009
  15. [19]Paliwal S Subjective estimates of uncertainty during gambling and impulsivity after subthalamic deep brain stimulation for Parkinson’s disease Sci. Rep. 2019 9 14795 31616015 PMC6794275
  16. [20]Piray P Ly V Roelofs K Cools R Toni I Emotionally aversive cues suppress neural systems underlying optimal learning in socially anxious individuals J. Neurosci. 2019 39 1445 1456 30559152 PMC6381249
  17. [21]Cole DM Atypical processing of uncertainty in individuals at risk for psychosis NeuroImage: Clin. 2020 26 102239 32182575 PMC7076146
  18. [22]Deserno L Volatility estimates increase choice switching and relate to prefrontal activity in Schizophrenia Biol. Psychiatry.: Cogn. Neurosci. Neuroimaging 2020 5 173 183 31937449
  19. [23]Diaconescu AO Wellstein KV Kasper L Mathys C Stephan KE Hierarchical Bayesian models of social inference for probing persecutory delusional ideation J. Abnorm. Psychol. 2020 129 556 569 32757600
  20. [25]Nassar MR Wilson RC Heasly B Gold JI An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment J. Neurosci. 2010 30 12366 12378 20844132 PMC2945906
  21. [27]Diaconescu, A. O. et al. Inferring on the Intentions of Others by Hierarchical Bayesian Learning. PLoS Comput. Biol.10, e1003810 (2014). PMC4154656 25187943
  22. [28]Reed, E. J. et al. Paranoia as a deficit in non-social belief updating. eLife9, (2020). PMC7326495 32452769
  23. [31]Aylward J Altered learning under uncertainty in unmedicated mood and anxiety disorders Nat. Hum. Behav. 2019 3 1116 1123 31209369 PMC6790140
  24. [33]Harlé KM Guo D Zhang S Paulus MP Yu AJ Anhedonia and anxiety underlying depressive symptomatology have distinct effects on reward-based decision-making PLOS ONE 2017 12 e0186473 29059254 PMC5653291
  25. [34]Huang H Thompson W Paulus MP Computational dysfunctions in anxiety: failure to differentiate signal from noise Biol. Psychiatry 2017 82 440 446 28838468 PMC5576575
  26. [36]Luhmann CC Ishida K Hajcak G Intolerance of uncertainty and decisions about delayed, probabilistic rewards Behav. Ther. 2011 42 378 386 21658521
  27. [38]Pulcu E Browning M Affective bias as a rational response to the statistics of rewards and punishments eLife 2017 6 e27879 28976304 PMC5633345
  28. [41]Kalman RE A New Approach to Linear Filtering and Prediction Problems Trans. ASME–J. Basic Eng. 1960 82 35 45
  29. [42]Kakade S Dayan P Acquisition and extinction in autoshaping Psychol. Rev. 2002 109 533 544 12088244
  30. [43]Moens, V. & Zénon, A. Learning and forgetting using reinforced Bayesian change detection. PLoS Comput. Biol.15, e1006713 (2019). PMC6488101 30995214
  31. [44]Silvetti M Vassena E Abrahamse E Verguts T Dorsal anterior cingulate-brainstem ensemble as a reinforcement meta-learner PLOS Computational Biol. 2018 14 e1006370 PMC6126878 30142152
  32. [45]Wilson RC Nassar MR Gold JI Bayesian on-line learning of the hazard rate in change-point problems Neural Comput 2010 22 2452 2476 20569174 PMC2966286
  33. [46]Griffiths, T. L., Navarro, D. J. & Sanborn, A. N. A More Rational Model of Categorization. Proceedings of the Annual Meeting of the Cognitive Science Society28, (2006).
  34. [47]Daw, N. D. & Courville, A. C. The rat as particle filter. in Advances in Neural Information Processing Systems 20 (eds. Platt, J. C., Koller, D., Singer, Y. & Roweis, S. T.) 369–376 (Curran Associates, Inc., 2008).
  35. [48]Brown SD Steyvers M Detecting and predicting changes Cogn. Psychol. 2009 58 49 67 18976988
  36. [49]Doucet, A., Freitas, N. de, Murphy, K. P. & Russell, S. J. Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks. in Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence 176–183 (Morgan Kaufmann Publishers Inc., 2000).
  37. [50]Doucet, A. & Johansen, A. M. A tutorial on particle filtering and smoothing: fifteen years later (Oxford University Press, 2011).
  38. [52]Nassar, M. R. et al. Age differences in learning emerge from an insufficient representation of uncertainty in older adults. Nat Commun7, 11609 (2016). PMC4906358 27282467
  39. [56]Haselgrove M Aydin A Pearce JM A partial reinforcement extinction effect despite equal rates of reinforcement during Pavlovian conditioning J. Exp. Psychol. Anim. Behav. Process 2004 30 240 250 15279514
  40. [57]Li J Schiller D Schoenbaum G Phelps EA Daw ND Differential roles of human striatum and amygdala in associative learning Nat. Neurosci. 2011 14 1250 1252 21909088 PMC3268261
  41. [58]Gallistel CR Gibbon J Time, rate, and conditioning Psychol. Rev. 2000 107 289 344 10789198
  42. [59]Phelps EA Lempert KM Sokol-Hessner P Emotion and decision making: multiple modulatory neural circuits Annu. Rev. Neurosci. 2014 37 263 287 24905597
  43. [60]Averbeck BB Costa VD Motivational neural circuits underlying reinforcement learning Nat. Neurosci. 2017 20 505 512 28352111
  44. [61]Holland PC Gallagher M Amygdala circuitry in attentional and representational processes Trends Cogn. Sci. (Regul. Ed.) 1999 3 65 73 10234229
  45. [62]Roesch MR Esber GR Li J Daw ND Schoenbaum G Surprise! neural correlates of Pearce-Hall and Rescorla-Wagner Coexist within the Brain Eur. J. Neurosci. 2012 35 1190 1200 22487047 PMC3325511
  46. [63]Costa VD Dal Monte O Lucas DR Murray EA Averbeck BB Amygdala and Ventral Striatum Make Distinct Contributions to Reinforcement Learning Neuron 2016 92 505 517 27720488 PMC5074688
  47. [64]Homan P Neural computations of threat in the aftermath of combat trauma Nat. Neurosci. 2019 22 470 476 30664770 PMC6829910
  48. [65]Holland PC Gallagher M Amygdala central nucleus lesions disrupt increments, but not decrements, in conditioned stimulus processing Behav. Neurosci. 1993 107 246 253 8484890
  49. [66]Holland PC Schiffino FL Mini-review: prediction errors, attention and associative learning Neurobiol. Learn Mem. 2016 131 207 215 26948122 PMC4862921
  50. [67]Hampton AN Adolphs R Tyszka MJ O’Doherty JP Contributions of the amygdala to reward expectancy and choice signals in human prefrontal cortex Neuron 2007 55 545 555 17698008
  51. [69]Khorsand P Soltani A Optimal structure of metaplasticity for adaptive learning PLoS Comput. Biol. 2017 13 e1005630 28658247 PMC5509349
  52. [70]Diederen KMJ Schultz W Scaling prediction errors to reward variability benefits error-driven learning in humans J. Neurophysiol. 2015 114 1628 1640 26180123 PMC4563025
  53. [71]Wilson RC Nassar MR Gold JI A Mixture of Delta-Rules Approximation to Bayesian Inference in Change-Point Problems PLOS Computational Biol. 2013 9 e1003150 PMC3723502 23935472
  54. [74]Hartley CA Phelps EA Anxiety and decision-making Biol. Psychiatry 2012 72 113 118 22325982 PMC3864559
  55. [75]Beck, A. T. Depression: Causes and Treatment. (University of Pennsylvania Press, 1970).
  56. [78]Bouton ME Context and behavioral processes in extinction Learn Mem. 2004 11 485 494 15466298
  57. [79]Redish AD Jensen S Johnson A Kurth-Nelson Z Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling Psychol. Rev. 2007 114 784 805 17638506
  58. [80]Stephan KE Baldeweg T Friston KJ Synaptic plasticity and dysconnection in schizophrenia Biol. Psychiatry 2006 59 929 939 16427028
  59. [81]Baker SC Konova AB Daw ND Horga G A distinct inferential mechanism for delusions in schizophrenia Brain 2019 142 1797 1812 30895299 PMC6644849
  60. [83]Wengler K Goldberg A Chahine G Horga G Hallucinations and Delusions Relate to Distinct Hierarchical Alterations in Intrinsic Neural Timescales Biol. Psychiatry 2020 87 S179 S180 PMC7591251 33107431
  61. [84]Le Pelley ME The role of associative history in models of associative learning: a selective review and a hybrid model Q J. Exp. Psychol. B 2004 57 193 243 15204108
  62. [85]Haselgrove M Esber GR Pearce JM Jones PM Two kinds of attention in Pavlovian conditioning: evidence for a hybrid model of learning J. Exp. Psychol. Anim. Behav. Process 2010 36 456 470 20718552
  63. [86]Pearce, J. M. & Mackintosh, N. J. Two theories of attention: a review and a possible integration. in Attention and Associative Learning: From Brain to Behaviour (eds. Mitchell, C. & Le Pelley, M. E.) 11–39 (Oxford University Press, 2010).
  64. [89]Aponte, E. et al. TAPAS - Translational Algorithms for Psychiatry-Advancing Science. Front. Psychiatry. 10.3389/fpsyt.2021.680811 (2020). · DOI · Paper page
  65. [91]Daw ND O’Doherty JP Dayan P Seymour B Dolan RJ Cortical substrates for exploratory decisions in humans Nature 2006 441 876 879 16778890 PMC2635947
  66. [92]Wilson RC Takahashi YK Schoenbaum G Niv Y Orbitofrontal cortex as a cognitive map of task space Neuron 2014 81 267 279 24462094 PMC4001869
  67. [95]Piray, P. & Daw, N. D. A model for learning based on the joint estimation of stochasticity and volatility. Zenodo, 10.5281/zenodo.5526668 (2021). PMC8592992 34782597 · DOI

Cited by

Sorted by citations

Authors
Piray, P., & Daw, N. D.
Affiliations
Princeton University (US)
Corresponding author
Piray, Payam
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2021-11-15
Volume
12
Issue
1
Pages
6587-6587
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-021-26731-9
PMC
PMC8592992
OpenAlex
W3211391129
Cited by
153
References
17
Citations by year
2026: 26 · 2025: 51 · 2024: 31 · 2023: 26 · 2022: 14 · 2021: 5
FWCI
9.99
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Science Foundation (1822571)
  • John Templeton Foundation (61454)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
Keywords
learning rate, volatility estimation, uncertainty, Bayesian inference, simultaneous estimation, learning models, anxiety, amygdala damage
MeSH terms
Algorithms, Animals, Haplorhini, Anxiety, Anxiety Disorders, Decision Making, Humans, Learning, Volatilization, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review