CoChat

Confidence modulates exploration and exploitation in value-based learning

Open access
Boldt, Annika, Blundell, Charles, De Martino, Benedetto · 2019 · Neuroscience of Consciousness

Published version of a preprint

Cited by 121

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.

Abstract

Uncertainty is ubiquitous in cognitive processing. In this study, we aim to investigate the ability agents possess to track and report the noise inherent in their mental operations, often in the form of confidence judgments. Here, we argue that humans can use uncertainty inherent in their representations of value beliefs to arbitrate between exploration and exploitation. Such uncertainty is reflected in explicit confidence judgments. Using a novel variant of a multi-armed bandit paradigm, we studied how beliefs were formed and how uncertainty in the encoding of these value beliefs (belief confidence) evolved over time. We found that people used uncertainty to arbitrate between exploration and exploitation, reflected in a higher tendency toward exploration when their confidence in their value representations was low. We furthermore found that value uncertainty can be linked to frameworks of metacognition in decision making in two ways. First, belief confidence drives decision confidence, i.e. people's evaluation of their own choices. Second, individuals with higher metacognitive insight into their choices were also better at tracing the uncertainty in their environment. Together, these findings argue that such uncertainty representations play a key role in the context of cognitive control.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/nc/niz004

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 31 references

Research works printed but not linked: 38

  1. Audley, 1960, A stochastic model for individual choice behavior, Psychol Rev, 67, 1, 10.1037/h0046438 · DOI
  2. Badre, 2012, Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration, Neuron, 73, 595, 10.1016/j.neuron.2011.12.025 · DOI · Paper page
  3. Baranski, 1994, The calibration and resolution of confidence in perceptual judgments, Percept Psychophys, 55, 412, 10.3758/BF03205299 · DOI · Paper page
  4. Baranski, 1995, On the calibration of knowledge and perception, Can J Exp Psychol, 49, 397, 10.1037/1196-1961.49.3.397 · DOI · Paper page
  5. Boorman, 2011, Counterfactual choice and learning in a Neural Network centered on human lateral frontopolar cortex, PLoS Biol, 9, 10.1371/journal.pbio.1001093 · DOI · Paper page
  6. Butlin, 2010, The psychology and neuroscience of desire and action, Neuron, 88, 1
  7. Brim, 1955, Attitude content-intensity and probability expectations, Am Sociol Rev, 20, 68., 10.2307/2088202 · DOI
  8. Daw, 2006, Cortical substrates for exploratory decisions in humans, Nature, 441, 876, 10.1038/nature04766 · DOI · Paper page
  9. Dayan, 1996, Exploration Bonuses and Dual Control, 22, 5
  10. De Martino, 2017, Social information is integrated into value and confidence judgments according to its reliability, J Neurosci, 37, 6066, 10.1523/JNEUROSCI.3880-16.2017 · DOI · Paper page
  11. Domenech, 2015, Executive control and decision-making in the prefrontal cortex, Curr Opin Behav Sci, 1, 101, 10.1016/j.cobeha.2014.10.007 · DOI · Paper page
  12. Dutilh, 2012, Testing theories of post-error slowing, Attent Percept Psychophys, 74, 454, 10.3758/s13414-011-0243-2 · DOI · Paper page
  13. Fernandez-Duque, 2000, Executive attention and metacognitive regulation, Conscious Cogn, 9, 288, 10.1006/ccog.2000.0447 · DOI · Paper page
  14. Fleming, 2017, HMeta-d: hierarchical Bayesian estimation of metacognitive efficiency from confidence ratings, Neurosci Conscious, 3, 1
  15. Frank, 2009, Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation, Nat Neurosci, 12, 1062., 10.1038/nn.2342 · DOI · Paper page
  16. Fullerton, 1892, One the Perception of Small Differences
  17. Gittins, 1974, Statistics, 241
  18. Griffing, 1895, On Sensations from Pressure and Impact
  19. Guggenmos, 2016, Mesolimbic confidence signals guide perceptual learning in the absence of external feedback, eLife, 5, 1, 10.7554/eLife.13388 · DOI · Paper page
  20. Hanczakowski, 2014, Feeling of knowing and restudy choices, Psychon Bull Rev, 21, 1617, 10.3758/s13423-014-0619-0 · DOI
  21. Hertzog, 2000, The Handbook of Aging and Cognition, 417
  22. Kleiner, 2007, What’s new in Psychtoolbox-3?, Perception 36 ECVP Abstract Supplement
  23. Kolling, 2014, Multiple neural mechanisms of decision making and their competition under changing risk pressure, Neuron, 81, 1190, 10.1016/j.neuron.2014.01.033 · DOI · Paper page
  24. Koriat, 2013, Confidence in personal preferences, J Behav Decis Mak, 26, 247, 10.1002/bdm.1758 · DOI
  25. Maniscalco, 2012, A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings, Conscious Cogn, 21, 422., 10.1016/j.concog.2011.09.021 · DOI · Paper page
  26. Metcalfe, 2008, Evidence that judgments of learning are causally related to study choice, Psychon Bull Rev, 15, 174, 10.3758/PBR.15.1.174 · DOI · Paper page
  27. Meyniel, 2015, Confidence as Bayesian probability: from neural origins to behavior, Neuron, 88, 78, 10.1016/j.neuron.2015.09.039 · DOI · Paper page
  28. Moreno-Bote, 2010, Decision confidence and uncertainty in diffusion models with partially correlated neuronal integrators, Neural Comput, 22, 1786, 10.1162/neco.2010.12-08-930 · DOI · Paper page
  29. Nelson, 1988, Allocation of self-paced study time and the “labor-in-vain effect”, J Exp Psychol Learn Mem Cogn, 14, 676, 10.1037/0278-7393.14.4.676 · DOI · Paper page
  30. Nelson, 1990, The Psychology of Learning and Motivation: Advances in Research and Theory, 125
  31. Peirce, 1884, On small differences in sensation, Mem Natl Acad Sci, 3, 73
  32. Rabbitt, 1966, Errors and error correction in choice-response tasks, J Exp Psychol, 71, 264, 10.1037/h0022853 · DOI · Paper page
  33. Schaalje, 2002, Adequacy of approximations to distributions of test statistics in complex mixed linear models, J Agric Biol Environ Stat, 7, 512, 10.1198/108571102726 · DOI · Paper page
  34. Schumpeter, 1934, The Theory of Economic Development
  35. Smith, 1988, Cognitive response to advertising and trial: belief strength, belief confidence and product curiosity, J Advert, 17, 3, 10.1080/00913367.1988.10673118 · DOI · Paper page
  36. Sutton, 1998, Reinforcement Learning: An Introduction
  37. Van Bergen, 2015, Sensory uncertainty decoded from visual cortex predicts behavior, Nat Neurosci, 18, 1728, 10.1038/nn.4150 · DOI · Paper page
  38. Yeung, 2014, The Cognitive Neuroscience of Metacognition, 147, 10.1007/978-3-642-45190-4_7 · DOI

Cited by

See all 121

Metadata

Authors
Boldt, A., Blundell, C., & De Martino, B.
Venue
Neuroscience of Consciousness
Published
2019-01-01
Volume
2019
Issue
1
Pages
niz004-niz004
ISSN
2057-2107 (ISSN-L)
OpenAlex
W2951603152
Cited by
121
References
31
FWCI
9.76
Type
journal article
Language
en
Open access
gold
Funders
  • Sir Henry Dale Fellowship
  • Wellcome Trust
  • Sir Henry Wellcome Postdoctoral Fellowship

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Confidence modulates exploration and exploitation in value-based learning. https://app.cochat.ai/papers/10.1093/nc/niz004
The paper
Boldt, A., Blundell, C., & De Martino, B. (2019). Confidence modulates exploration and exploitation in value-based learning. Neuroscience of Consciousness. https://doi.org/10.1093/nc/niz004

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.