The DOI matches its Crossref record; 45 of the 46 references it cites were found in a registry or an index.
Deconstructing the human algorithms for exploration
Open accessAbstract
The dilemma between information gathering (exploration) and reward seeking (exploitation) is a fundamental problem for reinforcement learning agents. How humans resolve this dilemma is still an open question, because experiments have provided equivocal evidence about the underlying algorithms used by humans. We show that two families of algorithms can be distinguished in terms of how uncertainty affects exploration. Algorithms based on uncertainty bonuses predict a change in response bias as a function of uncertainty, whereas algorithms based on sampling predict a change in response slope. Two experiments provide evidence for both bias and slope changes, and computational modeling confirms that a hybrid model is the best quantitative account of the data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“UCB exploration bonuses are linked to Bayesian posterior uncertainty: as samples accumulate, the posterior distribution over an option’s value narrows, reducing the exploration bonus20.”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
In the citing paper
“The model has some generalizability: it predicted behavior in a different explore–exploit task20 (Fig. 5b).”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
In the citing paper
“Remarkably, our symbolic functions also outperformed the UCB model in predicting participants’ choices in this different context (Wilcoxon signed-rank test: W = 15.0, n = 89, median difference = −0.07, P = 4.31 × 10−16).”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
On the Rate of Gain of Information
Hick, W. E. (1952) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470215208416600 · Cited by 2,279
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224
Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. (2012) · IEEE Transactions on Information Theory · doi:10.48550/arxiv.0912.3995 · Cited by 2,100
Stimulus information as a determinant of reaction time.
Hyman, R. (1953) · Journal of Experimental Psychology · doi:10.1037/h0056940 · Cited by 1,443 · cited 1×
“More generally, many studies have found that response time increases with uncertainty (e.g., Gershman, Tenenbaum, & Jäkel, 2016; Grossman, 1953; Hick, 1952; Hyman, 1953).”
BackgroundBayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423
Cohen, J. D., McClure, S. M., & Yu, A. J. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2098 · Cited by 1,121
Research works printed but not linked: 1
- Srinivas, N., Krause, A., Seeger, M., & Kakade, S. M. (2010). Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th international conference on machine learning (pp. 1015–1022).
Cited by
Sorted by citations
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269
Computational mechanisms of curiosity and goal-directed exploration
Schwartenbeck, P., Passecker, J., Hauser, T. U., FitzGerald, T. H. B., Kronbichler, M., & Friston, K. J. (2019) · eLife · doi:10.7554/elife.41703 · Cited by 278
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236
Self-citationWhen Is More Uncertainty Better? A Model of Uncertainty Regulation and Effectiveness
Griffin, M. A., & Grote, G. (2020) · Academy of Management Review · doi:10.5465/amr.2018.0271 · Cited by 223
Balancing exploration and exploitation with information and randomization
Wilson, R. C., Bonawitz, E. B., Costa, V. D., & Ebitz, R. B. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.10.001 · Cited by 236
Metadata
- Authors
- Gershman, S. J.
- Affiliations
- Harvard University (US)
- Corresponding author
- Gershman, Samuel J.
- ORCID
- Gershman, Samuel J.
- Venue
- Cognition
- Publisher
- Elsevier BV
- Published
- 2018-04
- Volume
- 173
- Pages
- 34-42
- ISSN
- 0010-0277 (ISSN-L)
- PMC
- PMC5801139
- OpenAlex
- W2778689844
- Cited by
- 267
- References
- 46
- Citations by year
- 2026: 38 · 2025: 24 · 2024: 49 · 2023: 35 · 2022: 33 · 2021: 27 · 2020: 29 · 2019: 19 · 2018: 11
- FWCI
- 9.79
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Cognition (acceptedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- NSF Collaborative Research in Computational Neuroscience (IIS-120 7833)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- exploration-exploitation balance, human decision-making, uncertainty, exploration algorithms, response bias, reinforcement learning, computational modeling, hybrid model
- MeSH terms
- Adult, Algorithms, Choice Behavior, Exploratory Behavior, Humans, Psychomotor Performance, Reinforcement, Psychology, Reward, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
