The DOI matches its Crossref record; 45 of the 46 references it cites were found in a registry or an index.
Deconstructing the human algorithms for exploration
Open accessAbstract
The dilemma between information gathering (exploration) and reward seeking (exploitation) is a fundamental problem for reinforcement learning agents. How humans resolve this dilemma is still an open question, because experiments have provided equivocal evidence about the underlying algorithms used by humans. We show that two families of algorithms can be distinguished in terms of how uncertainty affects exploration. Algorithms based on uncertainty bonuses predict a change in response bias as a function of uncertainty, whereas algorithms based on sampling predict a change in response slope. Two experiments provide evidence for both bias and slope changes, and computational modeling confirms that a hybrid model is the best quantitative account of the data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“UCB exploration bonuses are linked to Bayesian posterior uncertainty: as samples accumulate, the posterior distribution over an option’s value narrows, reducing the exploration bonus20.”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
In the citing paper
“The model has some generalizability: it predicted behavior in a different explore–exploit task20 (Fig. 5b).”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
In the citing paper
“Remarkably, our symbolic functions also outperformed the UCB model in predicting participants’ choices in this different context (Wilcoxon signed-rank test: W = 15.0, n = 89, median difference = −0.07, P = 4.31 × 10−16).”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
On the Rate of Gain of Information
Hick, W. E. (1952) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470215208416600 · Cited by 2,279
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224
Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. (2012) · IEEE Transactions on Information Theory · doi:10.48550/arxiv.0912.3995 · Cited by 2,100
Stimulus information as a determinant of reaction time.
Hyman, R. (1953) · Journal of Experimental Psychology · doi:10.1037/h0056940 · Cited by 1,443 · cited 1×
“More generally, many studies have found that response time increases with uncertainty (e.g., Gershman, Tenenbaum, & Jäkel, 2016; Grossman, 1953; Hick, 1952; Hyman, 1953).”
BackgroundBayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423
Cohen, J. D., McClure, S. M., & Yu, A. J. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2098 · Cited by 1,121
(2011) · Cited by 1,000
Small feedback‐based decisions and their limited correspondence to description‐based decisions
Barron, G., & Erev, I. (2003) · Journal of Behavioral Decision Making · doi:10.1002/bdm.443 · Cited by 742
Bayesian model selection for group studies — Revisited
Rigoux, L., Stephan, K. E., Friston, K. J., & Jean Daunizeau, A. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.08.065 · Cited by 659
Humans use directed and random exploration to solve the explore–exploit dilemma.
Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A., & Cohen, J. D. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0038199 · Cited by 580
Frank, M. J., Doll, B. B., Oas-Terpstra, J., & Moreno, F. (2009) · Nature Neuroscience · doi:10.1038/nn.2342 · Cited by 479
The Oxford Handbook of Causal Reasoning
(2017) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199399550.001.0001 · Cited by 421
Unpacking the exploration–exploitation tradeoff: A synthesis of human and animal literatures.
Mehlhorn, K., Newell, B. R., Todd, P. M., Lee, M. D., Morgan, K. V., Braithwaite, V. A., Hausmann, D., Fiedler, K., & González, C. (2015) · Decision · doi:10.1037/dec0000033 · Cited by 419
Milosavljevic, M., Malmaud, J., Huth, A. G., Koch, C., & Rangel, A. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500001285 · Cited by 333
Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings
Payzan-LeNestour, E., & Bossaerts, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001048 · Cited by 267
Instance-based learning: Integrating sampling and repeated decisions from experience.
Gonzalez, C., & Dutt, V. (2011) · Psychological Review · doi:10.1037/a0024558 · Cited by 255
Optimal policy for value-based decision-making
Tajima, S., Drugowitsch, J., & Pouget, A. (2016) · Nature Communications · doi:10.1038/ncomms12400 · Cited by 226
Retrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217
Self-citationOptimistic bayesian sampling in contextual-bandit problems
May, B. C., Korda, N., Lee, A., & Leslie, D. S. (2012) · Lancaster EPrints (Lancaster University) · doi:10.5555/2188385.2343711 · Cited by 213
Ratcliff, R., & Frank, M. J. (2012) · Neural Computation · doi:10.1162/neco_a_00270 · Cited by 211
The algorithmic anatomy of model-based evaluation
Daw, N. D., & Dayan, P. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0478 · Cited by 195
Pearson, J. M., Hayden, B. Y., Raghavachari, S., & Platt, M. L. (2009) · Current Biology · doi:10.1016/j.cub.2009.07.048 · Cited by 191
Uncertainty and Exploration in a Restless Bandit Problem
Speekenbrink, M., & Konstantinidis, E. (2015) · Topics in Cognitive Science · doi:10.1111/tops.12145 · Cited by 177
A causal role for right frontopolar cortex in directed, but not random, exploration
Zajkowski, W. K., Kossut, M. M., & Wilson, R. C. (2017) · eLife · doi:10.7554/elife.27430 · Cited by 169
Summerfield, C., & Tsetsos, K. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00070 · Cited by 167
Charting the expansion of strategic exploratory behavior during adolescence.
Somerville, L. H., Sasse, S. F., Garrad, M. C., Drysdale, A. T., Akar, N. A., Insel, C., & Wilson, R. C. (2016) · Journal of Experimental Psychology General · doi:10.1037/xge0000250 · Cited by 160
Entropy and Choice Time: The Effect of Frequency Unbalance on Choice-Response
Crossman, E. R. F. W. (1953) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470215308416625 · Cited by 142
Instance‐based Learning: A General Model of Repeated Binary Choice
Lejarraga, T., Dutt, V., & González, C. (2010) · Journal of Behavioral Decision Making · doi:10.1002/bdm.722 · Cited by 126
Sequential Choice Under Ambiguity: Intuitive Solutions to the Armed-Bandit Problem
Meyer, R. J., & Shi, Y. (1995) · Management Science · doi:10.1287/mnsc.41.5.817 · Cited by 112
Novelty and Inductive Generalization in Human Reinforcement Learning
Gershman, S. J., & Niv, Y. (2015) · Topics in Cognitive Science · doi:10.1111/tops.12138 · Cited by 99
Self-citationPsychological models of human and optimal performance in bandit problems
Lee, M. D., Zhang, S., Munro, M., & Steyvers, M. (2010) · Cognitive Systems Research · doi:10.1016/j.cogsys.2010.07.007 · Cited by 87
The Nature of Belief-Directed Exploratory Choice in Human Decision-Making
Knox, W. B., Otto, A. R., Stone, P., & Love, B. C. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00398 · Cited by 85
The effect of atomoxetine on random and directed exploration in humans
Warren, C. M., Wilson, R. C., van der Wee, N. J., Giltay, E. J., van Noorden, M. S., Cohen, J. D., & Nieuwenhuis, S. (2017) · PLoS ONE · doi:10.1371/journal.pone.0176034 · Cited by 68
Structure Learning in Human Sequential Decision-Making
Acuna, D. E., & Schrater, P. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001003 · Cited by 60
Forgetful Bayes and myopic planning: Human learning and decision-making in a bandit setting
Zhang, S., & Yu, A. J. (2013) · Cited by 59
Payzan-LeNestour, É., & Bossaerts, P. L. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00150 · Cited by 54
Reinforcement Learning and Causal Models
Gershman, S. J. (2017) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199399550.013.20 · Cited by 51
Self-citationDiscovering hierarchical motion structure
Gershman, S. J., Tenenbaum, J. B., & Jäkel, F. (2015) · Vision Research · doi:10.1016/j.visres.2015.03.004 · Cited by 47
Self-citationStrategies for exploration in the domain of losses
Krueger, P. M., Wilson, R. C., & Cohen, J. D. (2017) · Judgment and Decision Making · doi:10.1017/s1930297500005659 · Cited by 38
Physiological and behavioral signatures of reflective exploratory choice
Otto, A. R., Knox, W. B., Markman, A. B., & Love, B. C. (2014) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-014-0260-4 · Cited by 28
Learning and decisions in contextual multi-armed bandit tasks
Schulz, E., Konstantinidis, E., & Speekenbrink, M. (2015) · MPG.PuRe (Max Planck Society) · Cited by 20
Flagged first, then by citations
1 of the 46 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Srinivas, N., Krause, A., Seeger, M., & Kakade, S. M. (2010). Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th international conference on machine learning (pp. 1015–1022).
Cited by
Sorted by citations
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269
Computational mechanisms of curiosity and goal-directed exploration
Schwartenbeck, P., Passecker, J., Hauser, T. U., FitzGerald, T. H. B., Kronbichler, M., & Friston, K. J. (2019) · eLife · doi:10.7554/elife.41703 · Cited by 278
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236
Self-citationWhen Is More Uncertainty Better? A Model of Uncertainty Regulation and Effectiveness
Griffin, M. A., & Grote, G. (2020) · Academy of Management Review · doi:10.5465/amr.2018.0271 · Cited by 223
Balancing exploration and exploitation with information and randomization
Wilson, R. C., Bonawitz, E. B., Costa, V. D., & Ebitz, R. B. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.10.001 · Cited by 236
Metadata
- Authors
- Gershman, S. J.
- Affiliations
- Harvard University (US)
- Corresponding author
- Gershman, Samuel J.
- ORCID
- Gershman, Samuel J.
- Venue
- Cognition
- Publisher
- Elsevier BV
- Published
- 2018-04
- Volume
- 173
- Pages
- 34-42
- ISSN
- 0010-0277 (ISSN-L)
- PMC
- PMC5801139
- OpenAlex
- W2778689844
- Cited by
- 267
- References
- 46
- Citations by year
- 2026: 38 · 2025: 24 · 2024: 49 · 2023: 35 · 2022: 33 · 2021: 27 · 2020: 29 · 2019: 19 · 2018: 11
- FWCI
- 9.79
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Cognition (acceptedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- NSF Collaborative Research in Computational Neuroscience (IIS-120 7833)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- exploration-exploitation balance, human decision-making, uncertainty, exploration algorithms, response bias, reinforcement learning, computational modeling, hybrid model
- MeSH terms
- Adult, Algorithms, Choice Behavior, Exploratory Behavior, Humans, Psychomotor Performance, Reinforcement, Psychology, Reward, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
