Confidence modulates exploration and exploitation in value-based learning
Open accessPublished version of a preprint
The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Uncertainty is ubiquitous in cognitive processing. In this study, we aim to investigate the ability agents possess to track and report the noise inherent in their mental operations, often in the form of confidence judgments. Here, we argue that humans can use uncertainty inherent in their representations of value beliefs to arbitrate between exploration and exploitation. Such uncertainty is reflected in explicit confidence judgments. Using a novel variant of a multi-armed bandit paradigm, we studied how beliefs were formed and how uncertainty in the encoding of these value beliefs (belief confidence) evolved over time. We found that people used uncertainty to arbitrate between exploration and exploitation, reflected in a higher tendency toward exploration when their confidence in their value representations was low. We furthermore found that value uncertainty can be linked to frameworks of metacognition in decision making in two ways. First, belief confidence drives decision confidence, i.e. people's evaluation of their own choices. Second, individuals with higher metacognitive insight into their choices were also better at tracing the uncertainty in their environment. Together, these findings argue that such uncertainty representations play a key role in the context of cognitive control.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
Metamemory: A Theoretical Framework and New Findings
Nelson, T. O. (1990) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60053-5 · Cited by 2,583 · cited 1×
“Confidence-guided exploration takes its place alongside a range of other findings that suggest that confidence forms a cornerstone of cognitive control (for reviews see Yeung & Summerfield, 2012, 2014; Shea et al., 2014; Femandez-Duque et al., 2000; Nelson & Narens, 1990).”
BackgroundUncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774
Probabilistic mental models: a Brunswikian theory of confidence
Gigerenzer, G., Hoffrage, U., & Kleinbölting, H. (1991) · Psychological Review · doi:10.1037/0033-295x.98.4.506 · Cited by 1,506 · cited 1×
“Much of this work had focused perception and memory (Koriat et al. 1980; Nelson and Narens 1990; Gigerenzer et al. 1991; Baranski and Petrusic 1995; Hertzog and Hultsch 2000; Pleskac and Busemeyer 2010).”
BackgroundRisko, E. F., & Gilbert, S. J. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.07.002 · Cited by 1,502
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477
Bayesian inference with probabilistic population codes
Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006) · Nature Neuroscience · doi:10.1038/nn1790 · Cited by 1,457
Fleming, S. M., & Lau, H. C. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00443 · Cited by 1,408
Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning
Koriat, A. (1997) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.126.4.349 · Cited by 1,363
Cohen, J. D., McClure, S. M., & Yu, A. J. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2098 · Cited by 1,121
Kang, M. J., Hsu, M., Krajbich, I. M., Loewenstein, G., McClure, S. M., Wang, J. T.-Y., & Camerer, C. F. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02402.x · Cited by 882 · cited 1×
“This finding is seemingly at odds with studies and theories that suggest confidence is related to curiosity in an inverse u-form shape (Kang et al. 2009; for a review of these findings see Butlin 2010).”
ResultRelating Introspective Accuracy to Individual Differences in Brain Structure
Fleming, S. M., Weil, R. S., Nagy, Z., Dolan, R. J., & Rees, G. (2010) · Science · doi:10.1126/science.1191883 · Cited by 809
The neural basis of metacognitive ability
Fleming, S. M., & Dolan, R. J. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2011.0417 · Cited by 747
Bahrami, B., Olsen, K., Latham, P. E., Roepstorff, A., Rees, G., & Frith, C. D. (2010) · Science · doi:10.1126/science.1185718 · Cited by 738
Two-stage dynamic signal detection: A theory of choice, decision time, and confidence.
Pleskac, T. J., & Busemeyer, J. R. (2010) · Psychological Review · doi:10.1037/a0019737 · Cited by 722
Metacognition in human decision-making: confidence and error monitoring
Yeung, N., & Summerfield, C. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2011.0416 · Cited by 667
Kolling, N., Behrens, T. E. J., Mars, R. B., & Rushworth, M. F. S. (2012) · Science · doi:10.1126/science.1216930 · Cited by 629
Confidence in value-based choice
De Martino, B., Fleming, S. M., Garrett, N., & Dolan, R. J. (2013) · Nature Neuroscience · doi:10.1038/nn.3279 · Cited by 534
Confidence and certainty: Distinct probabilistic quantities for different goals
Pouget, A., Drugowitsch, J., & Kepecs, A. (2016) · Nature Neuroscience · doi:10.1038/nn.4240 · Cited by 528
Choice Certainty Is Informed by Both Evidence and Decision Time
Kiani, R., Corthell, L., & Shadlen, M. N. (2014) · Neuron · doi:10.1016/j.neuron.2014.12.015 · Cited by 470
Prefrontal Contributions to Metacognition in Perceptual Decision Making
Fleming, S. M., Huijgen, J., & Dolan, R. J. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.6489-11.2012 · Cited by 391
Elapsed Decision Time Affects the Weighting of Prior Probability in a Perceptual Decision Task
Hanks, T. D., Mazurek, M. E., Kiani, R., Hopp, E., & Shadlen, M. N. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.5613-10.2011 · Cited by 378
Supra-personal cognitive control and metacognition
Shea, N., Boldt, A., Bang, D., Yeung, N., Heyes, C., & Frith, C. D. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.01.006 · Cited by 368 · cited 1×
“A wealth of empirical work has found that humans can accurately trace uncertainty and report the level of confidence in their judgments (Peirce and Jastrow 1884; Henmon 1911; Audley 1960; Shea et al. 2014; Meyniel et al. 2015b; Guggenmos et al. 2016).”
BackgroundSelf-citationKnowing how much you don't know: a neural organization of uncertainty estimates
Bach, D. R., & Dolan, R. J. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3289 · Cited by 339
Neural Coding of Uncertainty and Probability
Ma, W. J., & Jazayeri, M. (2014) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071013-014017 · Cited by 290 · cited 1×
“Recent research has supported the view of a signalinherent representation of uncertainty, suggesting a coding scheme in which the reliability of a signal is represented together with its average strength (e.g. Ma et al. 2006; Ma and Jazayeri 2014; Van Bergen et al. 2015).”
BackgroundAutomatic integration of confidence in the brain valuation signal
Lebreton, M., Abitbol, R., Daunizeau, J., & Pessiglione, M. (2015) · Nature Neuroscience · doi:10.1038/nn.4064 · Cited by 265
Distinct encoding of decision confidence in human medial prefrontal cortex
Bang, D., & Fleming, S. M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1800795115 · Cited by 207
The relation of the time of a judgment to its accuracy
Henmon, V. A. C. (1911) · Psychological Review · doi:10.1037/h0074579 · Cited by 190
The Sense of Confidence during Probabilistic Learning: A Normative Account
Meyniel, F., Schlunegger, D., & Dehaene, S. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004305 · Cited by 163
Vickers, D., & Packer, J. (1982) · Acta Psychologica · doi:10.1016/0001-6918(82)90006-3 · Cited by 139
The impact of evidence reliability on sensitivity and bias in decision confidence.
Boldt, A., de Gardelle, V., & Yeung, N. (2017) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/xhp0000404 · Cited by 110
Self-citation
Research works printed but not linked: 38
- Audley, 1960, A stochastic model for individual choice behavior, Psychol Rev, 67, 1, 10.1037/h0046438 · DOI
- Badre, 2012, Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration, Neuron, 73, 595, 10.1016/j.neuron.2011.12.025 · DOI · Paper page
- Baranski, 1994, The calibration and resolution of confidence in perceptual judgments, Percept Psychophys, 55, 412, 10.3758/BF03205299 · DOI · Paper page
- Baranski, 1995, On the calibration of knowledge and perception, Can J Exp Psychol, 49, 397, 10.1037/1196-1961.49.3.397 · DOI · Paper page
- Boorman, 2011, Counterfactual choice and learning in a Neural Network centered on human lateral frontopolar cortex, PLoS Biol, 9, 10.1371/journal.pbio.1001093 · DOI · Paper page
- Butlin, 2010, The psychology and neuroscience of desire and action, Neuron, 88, 1
- Brim, 1955, Attitude content-intensity and probability expectations, Am Sociol Rev, 20, 68., 10.2307/2088202 · DOI
- Daw, 2006, Cortical substrates for exploratory decisions in humans, Nature, 441, 876, 10.1038/nature04766 · DOI · Paper page
- Dayan, 1996, Exploration Bonuses and Dual Control, 22, 5
- De Martino, 2017, Social information is integrated into value and confidence judgments according to its reliability, J Neurosci, 37, 6066, 10.1523/JNEUROSCI.3880-16.2017 · DOI · Paper page
- Domenech, 2015, Executive control and decision-making in the prefrontal cortex, Curr Opin Behav Sci, 1, 101, 10.1016/j.cobeha.2014.10.007 · DOI · Paper page
- Dutilh, 2012, Testing theories of post-error slowing, Attent Percept Psychophys, 74, 454, 10.3758/s13414-011-0243-2 · DOI · Paper page
- Fernandez-Duque, 2000, Executive attention and metacognitive regulation, Conscious Cogn, 9, 288, 10.1006/ccog.2000.0447 · DOI · Paper page
- Fleming, 2017, HMeta-d: hierarchical Bayesian estimation of metacognitive efficiency from confidence ratings, Neurosci Conscious, 3, 1
- Frank, 2009, Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation, Nat Neurosci, 12, 1062., 10.1038/nn.2342 · DOI · Paper page
- Fullerton, 1892, One the Perception of Small Differences
- Gittins, 1974, Statistics, 241
- Griffing, 1895, On Sensations from Pressure and Impact
- Guggenmos, 2016, Mesolimbic confidence signals guide perceptual learning in the absence of external feedback, eLife, 5, 1, 10.7554/eLife.13388 · DOI · Paper page
- Hanczakowski, 2014, Feeling of knowing and restudy choices, Psychon Bull Rev, 21, 1617, 10.3758/s13423-014-0619-0 · DOI
- Hertzog, 2000, The Handbook of Aging and Cognition, 417
- Kleiner, 2007, What’s new in Psychtoolbox-3?, Perception 36 ECVP Abstract Supplement
- Kolling, 2014, Multiple neural mechanisms of decision making and their competition under changing risk pressure, Neuron, 81, 1190, 10.1016/j.neuron.2014.01.033 · DOI · Paper page
- Koriat, 2013, Confidence in personal preferences, J Behav Decis Mak, 26, 247, 10.1002/bdm.1758 · DOI
- Maniscalco, 2012, A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings, Conscious Cogn, 21, 422., 10.1016/j.concog.2011.09.021 · DOI · Paper page
- Metcalfe, 2008, Evidence that judgments of learning are causally related to study choice, Psychon Bull Rev, 15, 174, 10.3758/PBR.15.1.174 · DOI · Paper page
- Meyniel, 2015, Confidence as Bayesian probability: from neural origins to behavior, Neuron, 88, 78, 10.1016/j.neuron.2015.09.039 · DOI · Paper page
- Moreno-Bote, 2010, Decision confidence and uncertainty in diffusion models with partially correlated neuronal integrators, Neural Comput, 22, 1786, 10.1162/neco.2010.12-08-930 · DOI · Paper page
- Nelson, 1988, Allocation of self-paced study time and the “labor-in-vain effect”, J Exp Psychol Learn Mem Cogn, 14, 676, 10.1037/0278-7393.14.4.676 · DOI · Paper page
- Nelson, 1990, The Psychology of Learning and Motivation: Advances in Research and Theory, 125
- Peirce, 1884, On small differences in sensation, Mem Natl Acad Sci, 3, 73
- Rabbitt, 1966, Errors and error correction in choice-response tasks, J Exp Psychol, 71, 264, 10.1037/h0022853 · DOI · Paper page
- Schaalje, 2002, Adequacy of approximations to distributions of test statistics in complex mixed linear models, J Agric Biol Environ Stat, 7, 512, 10.1198/108571102726 · DOI · Paper page
- Schumpeter, 1934, The Theory of Economic Development
- Smith, 1988, Cognitive response to advertising and trial: belief strength, belief confidence and product curiosity, J Advert, 17, 3, 10.1080/00913367.1988.10673118 · DOI · Paper page
- Sutton, 1998, Reinforcement Learning: An Introduction
- Van Bergen, 2015, Sensory uncertainty decoded from visual cortex predicts behavior, Nat Neurosci, 18, 1728, 10.1038/nn.4150 · DOI · Paper page
- Yeung, 2014, The Cognitive Neuroscience of Metacognition, 147, 10.1007/978-3-642-45190-4_7 · DOI
Cited by
A three-dimensional taxonomy of achievement emotions.
Pekrun, R., Marsh, H. W., Elliot, A. J., Stockinger, K., Perry, R. P., Vogl, E., Goetz, T., van Tilburg, W. A. P., Lüdtke, O., & Vispoel, W. P. (2023) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000448 · Cited by 301
Visual attention modulates the integration of goal-relevant evidence and not value
Sepulveda, P., Usher, M., Davies, N., Benson, A. A., Ortoleva, P., & De Martino, B. (2020) · eLife · doi:10.7554/elife.60705 · Cited by 121
Confidence guides spontaneous cognitive offloading
Boldt, A., & Gilbert, S. J. (2019) · Cognitive Research: Principles and Implications · doi:10.1186/s41235-019-0195-y · Cited by 105
Self-citationA postdecisional neural marker of confidence predicts information-seeking in decision-making
Desender, K., Murphy, P., Boldt, A., Verguts, T., & Yeung, N. (2019) · The Journal of Neuroscience · doi:10.1523/jneurosci.2620-18.2019 · Cited by 88
Self-citationUnderstanding neural signals of post-decisional performance monitoring: An integrative review
Desender, K., Ridderinkhof, K. R., & Murphy, P. R. (2021) · eLife · doi:10.7554/elife.67556 · Cited by 77
Metadata
- Authors
- Boldt, A., Blundell, C., & De Martino, B.
- Venue
- Neuroscience of Consciousness
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-01-01
- Volume
- 2019
- Issue
- 1
- Pages
- niz004-niz004
- ISSN
- 2057-2107 (ISSN-L)
- OpenAlex
- W2951603152
- Cited by
- 121
- References
- 31
- FWCI
- 9.76
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- Sir Henry Dale Fellowship
- Wellcome Trust
- Sir Henry Wellcome Postdoctoral Fellowship
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.