Confidence modulates exploration and exploitation in value-based learning
Open accessPublished version of a preprint
The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Uncertainty is ubiquitous in cognitive processing. In this study, we aim to investigate the ability agents possess to track and report the noise inherent in their mental operations, often in the form of confidence judgments. Here, we argue that humans can use uncertainty inherent in their representations of value beliefs to arbitrate between exploration and exploitation. Such uncertainty is reflected in explicit confidence judgments. Using a novel variant of a multi-armed bandit paradigm, we studied how beliefs were formed and how uncertainty in the encoding of these value beliefs (belief confidence) evolved over time. We found that people used uncertainty to arbitrate between exploration and exploitation, reflected in a higher tendency toward exploration when their confidence in their value representations was low. We furthermore found that value uncertainty can be linked to frameworks of metacognition in decision making in two ways. First, belief confidence drives decision confidence, i.e. people's evaluation of their own choices. Second, individuals with higher metacognitive insight into their choices were also better at tracing the uncertainty in their environment. Together, these findings argue that such uncertainty representations play a key role in the context of cognitive control.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
Metamemory: A Theoretical Framework and New Findings
Nelson, T. O. (1990) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60053-5 · Cited by 2,583 · cited 1×
“Confidence-guided exploration takes its place alongside a range of other findings that suggest that confidence forms a cornerstone of cognitive control (for reviews see Yeung & Summerfield, 2012, 2014; Shea et al., 2014; Femandez-Duque et al., 2000; Nelson & Narens, 1990).”
BackgroundUncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774
Probabilistic mental models: a Brunswikian theory of confidence
Gigerenzer, G., Hoffrage, U., & Kleinbölting, H. (1991) · Psychological Review · doi:10.1037/0033-295x.98.4.506 · Cited by 1,506 · cited 1×
“Much of this work had focused perception and memory (Koriat et al. 1980; Nelson and Narens 1990; Gigerenzer et al. 1991; Baranski and Petrusic 1995; Hertzog and Hultsch 2000; Pleskac and Busemeyer 2010).”
BackgroundRisko, E. F., & Gilbert, S. J. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.07.002 · Cited by 1,502
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477
Bayesian inference with probabilistic population codes
Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006) · Nature Neuroscience · doi:10.1038/nn1790 · Cited by 1,457
Fleming, S. M., & Lau, H. C. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00443 · Cited by 1,408
Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning
Koriat, A. (1997) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.126.4.349 · Cited by 1,363
Cohen, J. D., McClure, S. M., & Yu, A. J. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2098 · Cited by 1,121
Research works printed but not linked: 38
- Audley, 1960, A stochastic model for individual choice behavior, Psychol Rev, 67, 1, 10.1037/h0046438 · DOI
- Badre, 2012, Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration, Neuron, 73, 595, 10.1016/j.neuron.2011.12.025 · DOI · Paper page
- Baranski, 1994, The calibration and resolution of confidence in perceptual judgments, Percept Psychophys, 55, 412, 10.3758/BF03205299 · DOI · Paper page
- Baranski, 1995, On the calibration of knowledge and perception, Can J Exp Psychol, 49, 397, 10.1037/1196-1961.49.3.397 · DOI · Paper page
- Boorman, 2011, Counterfactual choice and learning in a Neural Network centered on human lateral frontopolar cortex, PLoS Biol, 9, 10.1371/journal.pbio.1001093 · DOI · Paper page
- Butlin, 2010, The psychology and neuroscience of desire and action, Neuron, 88, 1
- Brim, 1955, Attitude content-intensity and probability expectations, Am Sociol Rev, 20, 68., 10.2307/2088202 · DOI
- Daw, 2006, Cortical substrates for exploratory decisions in humans, Nature, 441, 876, 10.1038/nature04766 · DOI · Paper page
- Dayan, 1996, Exploration Bonuses and Dual Control, 22, 5
- De Martino, 2017, Social information is integrated into value and confidence judgments according to its reliability, J Neurosci, 37, 6066, 10.1523/JNEUROSCI.3880-16.2017 · DOI · Paper page
- Domenech, 2015, Executive control and decision-making in the prefrontal cortex, Curr Opin Behav Sci, 1, 101, 10.1016/j.cobeha.2014.10.007 · DOI · Paper page
- Dutilh, 2012, Testing theories of post-error slowing, Attent Percept Psychophys, 74, 454, 10.3758/s13414-011-0243-2 · DOI · Paper page
- Fernandez-Duque, 2000, Executive attention and metacognitive regulation, Conscious Cogn, 9, 288, 10.1006/ccog.2000.0447 · DOI · Paper page
- Fleming, 2017, HMeta-d: hierarchical Bayesian estimation of metacognitive efficiency from confidence ratings, Neurosci Conscious, 3, 1
- Frank, 2009, Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation, Nat Neurosci, 12, 1062., 10.1038/nn.2342 · DOI · Paper page
- Fullerton, 1892, One the Perception of Small Differences
- Gittins, 1974, Statistics, 241
- Griffing, 1895, On Sensations from Pressure and Impact
- Guggenmos, 2016, Mesolimbic confidence signals guide perceptual learning in the absence of external feedback, eLife, 5, 1, 10.7554/eLife.13388 · DOI · Paper page
- Hanczakowski, 2014, Feeling of knowing and restudy choices, Psychon Bull Rev, 21, 1617, 10.3758/s13423-014-0619-0 · DOI
- Hertzog, 2000, The Handbook of Aging and Cognition, 417
- Kleiner, 2007, What’s new in Psychtoolbox-3?, Perception 36 ECVP Abstract Supplement
- Kolling, 2014, Multiple neural mechanisms of decision making and their competition under changing risk pressure, Neuron, 81, 1190, 10.1016/j.neuron.2014.01.033 · DOI · Paper page
- Koriat, 2013, Confidence in personal preferences, J Behav Decis Mak, 26, 247, 10.1002/bdm.1758 · DOI
- Maniscalco, 2012, A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings, Conscious Cogn, 21, 422., 10.1016/j.concog.2011.09.021 · DOI · Paper page
- Metcalfe, 2008, Evidence that judgments of learning are causally related to study choice, Psychon Bull Rev, 15, 174, 10.3758/PBR.15.1.174 · DOI · Paper page
- Meyniel, 2015, Confidence as Bayesian probability: from neural origins to behavior, Neuron, 88, 78, 10.1016/j.neuron.2015.09.039 · DOI · Paper page
- Moreno-Bote, 2010, Decision confidence and uncertainty in diffusion models with partially correlated neuronal integrators, Neural Comput, 22, 1786, 10.1162/neco.2010.12-08-930 · DOI · Paper page
- Nelson, 1988, Allocation of self-paced study time and the “labor-in-vain effect”, J Exp Psychol Learn Mem Cogn, 14, 676, 10.1037/0278-7393.14.4.676 · DOI · Paper page
- Nelson, 1990, The Psychology of Learning and Motivation: Advances in Research and Theory, 125
- Peirce, 1884, On small differences in sensation, Mem Natl Acad Sci, 3, 73
- Rabbitt, 1966, Errors and error correction in choice-response tasks, J Exp Psychol, 71, 264, 10.1037/h0022853 · DOI · Paper page
- Schaalje, 2002, Adequacy of approximations to distributions of test statistics in complex mixed linear models, J Agric Biol Environ Stat, 7, 512, 10.1198/108571102726 · DOI · Paper page
- Schumpeter, 1934, The Theory of Economic Development
- Smith, 1988, Cognitive response to advertising and trial: belief strength, belief confidence and product curiosity, J Advert, 17, 3, 10.1080/00913367.1988.10673118 · DOI · Paper page
- Sutton, 1998, Reinforcement Learning: An Introduction
- Van Bergen, 2015, Sensory uncertainty decoded from visual cortex predicts behavior, Nat Neurosci, 18, 1728, 10.1038/nn.4150 · DOI · Paper page
- Yeung, 2014, The Cognitive Neuroscience of Metacognition, 147, 10.1007/978-3-642-45190-4_7 · DOI
Cited by
A three-dimensional taxonomy of achievement emotions.
Pekrun, R., Marsh, H. W., Elliot, A. J., Stockinger, K., Perry, R. P., Vogl, E., Goetz, T., van Tilburg, W. A. P., Lüdtke, O., & Vispoel, W. P. (2023) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000448 · Cited by 301
Visual attention modulates the integration of goal-relevant evidence and not value
Sepulveda, P., Usher, M., Davies, N., Benson, A. A., Ortoleva, P., & De Martino, B. (2020) · eLife · doi:10.7554/elife.60705 · Cited by 121
Confidence guides spontaneous cognitive offloading
Boldt, A., & Gilbert, S. J. (2019) · Cognitive Research: Principles and Implications · doi:10.1186/s41235-019-0195-y · Cited by 105
Self-citationA postdecisional neural marker of confidence predicts information-seeking in decision-making
Desender, K., Murphy, P., Boldt, A., Verguts, T., & Yeung, N. (2019) · The Journal of Neuroscience · doi:10.1523/jneurosci.2620-18.2019 · Cited by 88
Self-citationUnderstanding neural signals of post-decisional performance monitoring: An integrative review
Desender, K., Ridderinkhof, K. R., & Murphy, P. R. (2021) · eLife · doi:10.7554/elife.67556 · Cited by 77
Metadata
- Authors
- Boldt, A., Blundell, C., & De Martino, B.
- Venue
- Neuroscience of Consciousness
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-01-01
- Volume
- 2019
- Issue
- 1
- Pages
- niz004-niz004
- ISSN
- 2057-2107 (ISSN-L)
- OpenAlex
- W2951603152
- Cited by
- 121
- References
- 31
- FWCI
- 9.76
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- Sir Henry Dale Fellowship
- Wellcome Trust
- Sir Henry Wellcome Postdoctoral Fellowship
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Confidence modulates exploration and exploitation in value-based learning. https://app.cochat.ai/papers/10.1093/nc/niz004
- The paper
- Boldt, A., Blundell, C., & De Martino, B. (2019). Confidence modulates exploration and exploitation in value-based learning. Neuroscience of Consciousness. https://doi.org/10.1093/nc/niz004
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.