Uncertainty-driven regulation of learning and exploration in adolescents: A computational account
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Abstract Healthy adults flexibly adapt their learning strategies to ongoing changes in uncertainty, a key feature of adaptive behaviour. However, the developmental trajectory of this ability is yet unknown, as developmental studies have not incorporated trial-to-trial variation in uncertainty in their analyses or models. To address this issue, we compared adolescents’ and adults’ trial-to-trial dynamics of uncertainty, learning rate, and exploration in two tasks that assess learning in noisy but otherwise stable environments. In an estimation task—which provides direct indices of trial-specific learning rate—both age groups reduced their learning rate over time, as self-reported uncertainty decreased. Accordingly, the estimation data in both groups was better explained by a Bayesian model with dynamic learning rate (Kalman filter) than by conventional reinforcement-learning models. Furthermore, adolescents’ learning rates asymptoted at a higher level, reflecting an over-weighting of the most recent outcome, and the estimated Kalman-filter parameters suggested that this was due to an overestimation of environmental volatility. In a choice task, both age groups became more likely to choose the higher-valued option over time, but this increase in choice accuracy was smaller in the adolescents. In contrast to the estimation task, we found no evidence for a Bayesian expectation-updating process in the choice task, suggesting that estimation and choice tasks engage different learning processes. However, our modeling results of the choice task suggested that both age groups reduced their degree of exploration over time, and that the adolescents explored overall more than the adults. Finally, age-related differences in exploration parameters from fits to the choice data were mediated by participants’ volatility parameter from fits to the estimation data. Together, these results suggest that adolescents overestimate the rate of environmental change, resulting in elevated learning rates and increased exploration, which may help understand developmental changes in learning and decision-making. Author Summary To successfully learn the value of stimuli and actions, people should take into account their current (un)certainty about these values: Learning rates and exploration should be high when one’s value estimates are highly uncertain (in the beginning of learning), and decrease over time as evidence accumulates and uncertainty decreases. Recent studies have shown that healthy adults flexibly adapt their learning strategies based on ongoing changes in uncertainty, consistent with normative learning. However, the development of this ability prior to adulthood is yet unknown, as developmental learning studies have not considered trial-to-trial changes in uncertainty. Here, we show that adolescents, as compared to adults, showed a smaller decrease in both learning rate and exploration over time. Computational modeling revealed that both of these effects were due to adolescents overestimating the amount of environmental volatility, which made them more sensitive to recent relative to older evidence. The overestimation of volatility during adolescence may represent the rapidly changing environmental demands during this developmental period, and can help understand the surge in real-life risk taking and exploratory behaviours characteristic of adolescents.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120
Gelman, A. (2006) · Bayesian Analysis · doi:10.1214/06-ba117a · Cited by 4,050 · cited 1×
“These priors and hyperpriors were chosen for their 982 uninformative nature [64].”
BackgroundPearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · cited 1×
“This model also has a dynamic learning rate, but the learning rate on each trial is 343 defined in a non-Bayesian way, by scaling it by a dynamic ‘associability’ term which reflects 344 the unexpectedness of previous outcomes [28].”
AIC model selection using Akaike weights
Wagenmakers, E.-J., & Farrell, S. (2004) · Psychonomic Bulletin & Review · doi:10.3758/bf03206482 · Cited by 2,684
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588 · cited 1×
“In 610 Bayesian learning models, this corresponds to weaker priors—less precise belief 611 distributions—resulting in faster belief updating based on new information [34, 35].”
BackgroundUnderstanding adolescence as a period of social–affective engagement and goal flexibility
Crone, E. A., & Dahl, R. E. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3313 · Cited by 2,507 · cited 1×
“29 sensation-seeking behaviours during adolescence [39-41].”
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
A better lemon squeezer? maximum-likelihood regression with beta-distributed dependent variables.
Smithson, M., & Verkuilen, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.1.54 · Cited by 1,788 · cited 1×
“Beta distributions are 969 typically defined by two shape parameters (a and b), but we reparameterized these in terms of 970 a group-level mean (a/[a+b]) and group-level precision (a+b) [61].”
Uncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · cited 3×
“625 Volatility is one form of uncertainty that has been dissociated—both computationally 626 and in terms of relevant brain systems—from noise (stochasticity inherent in the 627 environment) and estimation uncertainty (the imprecision of the learner’s current estimate of 628 the environment, due to limited knowledge,…”
BackgroundCohen, J. D., McClure, S. M., & Yu, A. J. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2098 · Cited by 1,121
Risk taking in adolescence: What changes, and why?
STEINBERG, L. (2004) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1308.005 · Cited by 963
Self-Evaluation of Decision-Making: A General Bayesian Framework for Metacognitive Computation
Fleming, S. M., & Daw, N. D. (2017) · Psychological Review · doi:10.1037/rev0000045 · Cited by 584
Brain mediators of cardiovascular responses to social threat
Wager, T. D., Waugh, C. E., Lindquist, M., Noll, D. C., Fredrickson, B. L., & Taylor, S. F. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.05.043 · Cited by 402
Adolescents’ risk-taking behavior is driven by tolerance to ambiguity
Tymula, A., Rosenberg Belmaker, L. A., Roy, A. K., Ruderman, L., Manson, K., Glimcher, P. W., & Levy, I. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1207144109 · Cited by 387
Le Pelley, M. E. (2004) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/02724990344000141 · Cited by 360 · cited 1×
“The associability term 923 is incorporated in the expectation updating algorithm from the standard reinforcement 924 learning model [29-31]: 925”
MethodKnowing how much you don't know: a neural organization of uncertainty estimates
Bach, D. R., & Dolan, R. J. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3289 · Cited by 339 · cited 1×
“Specifically, the optimal rates of 68 learning and exploration depend on various sources of uncertainty, including imperfect 69 knowledge about the current state of the environment (estimation uncertainty, or ambiguity), 70 the amount of random outcome variability (noise, or risk), and the degree of environmental 71 c…”
BackgroundAdaptive learning under expected and unexpected uncertainty
Soltani, A., & Izquierdo, A. (2019) · Nature reviews. Neuroscience · doi:10.1038/s41583-019-0180-y · Cited by 286 · cited 1×
“Specifically, the optimal rates of 68 learning and exploration depend on various sources of uncertainty, including imperfect 69 knowledge about the current state of the environment (estimation uncertainty, or ambiguity), 70 the amount of random outcome variability (noise, or risk), and the degree of environmental 71 c…”
BackgroundA unique adolescent response to reward prediction errors
Cohen, J. R., Asarnow, R. F., Sabb, F. W., Bilder, R. M., Bookheimer, S. Y., Knowlton, B. J., & Poldrack, R. A. (2010) · Nature Neuroscience · doi:10.1038/nn.2558 · Cited by 285 · cited 1×
“Recent developmental studies that used computational models have provided 62 evidence that age-related differences in reinforcement-learning performance—usually 63 defined as choice accuracy in multi-armed bandit tasks—are related to developmental 64 changes in learning rate [3, 8-10], sensitivity to prediction errors…”
BackgroundGronau, Q. F., Sarafoglou, A., Matzke, D., Ly, A., Boehm, U., Marsman, M., Leslie, D. S., Forster, J. J., Wagenmakers, E.-J., & Steingroever, H. (2017) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2017.09.005 · Cited by 284
The neural representation of unexpected uncertainty during value-based decision making
Payzan-LeNestour, E., Dunne, S., Bossaerts, P., & O’Doherty, J. P. (2013) · Neuron · doi:10.1016/j.neuron.2013.04.037 · Cited by 257
Lucas, C. G., Bridgers, S., Griffiths, T. L., & Gopnik, A. (2014) · Cognition · doi:10.1016/j.cognition.2013.12.010 · Cited by 240
Hauser, T. U., Iannaccone, R., Walitza, S., Brandeis, D., & Brem, S. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.09.018 · Cited by 203 · cited 2×
“Recent developmental studies that used computational models have provided 62 evidence that age-related differences in reinforcement-learning performance—usually 63 defined as choice accuracy in multi-armed bandit tasks—are related to developmental 64 changes in learning rate [3, 8-10], sensitivity to prediction errors…”
BackgroundThe Sense of Confidence during Probabilistic Learning: A Normative Account
Meyniel, F., Schlunegger, D., & Dehaene, S. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004305 · Cited by 163 · cited 1×
“32 compare adolescents’ and adults’ performance on learning tasks involving continuous [48, 697 49] versus discrete [50, 51] change.”
BackgroundDiederen, K. M. J., Spencer, T., Vestergaard, M. D., Fletcher, P. C., & Schultz, W. (2016) · Neuron · doi:10.1016/j.neuron.2016.04.019 · Cited by 102 · cited 1×
“Indeed, 678 estimation performance on a more computationally demanding estimation task in a previous 679 study—adult participants learned about two stimuli during each task block—was best 680 explained by a non-Bayesian learning model that resembled our best-fitting model for the 681 adults’ choice data [47].”
MethodCatecholaminergic Regulation of Learning Rate in a Dynamic Environment
Jepma, M., Murphy, P. R., Nassar, M. R., Rangel-Gomez, M., Meeter, M., & Nieuwenhuis, S. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005171 · Cited by 91
Self-citation
Cited by
Lloyd, A., McKay, R., Sebastian, C. L., & Balsters, J. H. (2021) · Developmental Science · doi:10.1111/desc.13075 · Cited by 49
Marzuki, A. A., Tomić, I., Ip, S. H. Y., Gottwald, J., Kanen, J. W., Kaser, M., Sule, A., Conway-Morris, A., Sahakian, B. J., & Robbins, T. W. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.36195 · Cited by 32
“001 BAI score, median (IQR)d 46 (44-50) 64 (59-69) −18 (−22 to −14) <.”
(2025) · doi:10.1371/journal.pcbi.1012872 · Cited by 3
Metadata
- Authors
- Jepma, M., Schaaf, J. V., Visser, I., & Huizenga, H. M.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-09-30
- Volume
- 16
- Issue
- 9
- Pages
- e1008276
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W3091391680
- Cited by
- 53
- References
- 25
- FWCI
- 3.38
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.