The DOI matches its Crossref record; 45 of the 46 references it cites were found in a registry or an index.
Deconstructing the human algorithms for exploration
Open accessAbstract
The dilemma between information gathering (exploration) and reward seeking (exploitation) is a fundamental problem for reinforcement learning agents. How humans resolve this dilemma is still an open question, because experiments have provided equivocal evidence about the underlying algorithms used by humans. We show that two families of algorithms can be distinguished in terms of how uncertainty affects exploration. Algorithms based on uncertainty bonuses predict a change in response bias as a function of uncertainty, whereas algorithms based on sampling predict a change in response slope. Two experiments provide evidence for both bias and slope changes, and computational modeling confirms that a hybrid model is the best quantitative account of the data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“UCB exploration bonuses are linked to Bayesian posterior uncertainty: as samples accumulate, the posterior distribution over an option’s value narrows, reducing the exploration bonus20.”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
In the citing paper
“The model has some generalizability: it predicted behavior in a different explore–exploit task20 (Fig. 5b).”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
In the citing paper
“Remarkably, our symbolic functions also outperformed the UCB model in predicting participants’ choices in this different context (Wilcoxon signed-rank test: W = 15.0, n = 89, median difference = −0.07, P = 4.31 × 10−16).”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
On the Rate of Gain of Information
Hick, W. E. (1952) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470215208416600 · Cited by 2,279
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224
Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. (2012) · IEEE Transactions on Information Theory · doi:10.48550/arxiv.0912.3995 · Cited by 2,100
Stimulus information as a determinant of reaction time.
Hyman, R. (1953) · Journal of Experimental Psychology · doi:10.1037/h0056940 · Cited by 1,443 · cited 1×
“More generally, many studies have found that response time increases with uncertainty (e.g., Gershman, Tenenbaum, & Jäkel, 2016; Grossman, 1953; Hick, 1952; Hyman, 1953).”
BackgroundBayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423
Cohen, J. D., McClure, S. M., & Yu, A. J. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2098 · Cited by 1,121
Flagged first, then by citations
1 of the 46 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Srinivas, N., Krause, A., Seeger, M., & Kakade, S. M. (2010). Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th international conference on machine learning (pp. 1015–1022).
Cited by
Sorted by citations
Items 201–266 of 266
Human Strategy Adaptation in Reinforcement Learning Resembles Policy Gradient AscentPreprint
Xiong, H.-D., Li, J., Wilson, R. C., & Mattar, M. G. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.28.667308 · Cited by 1
Compression Efficiency and Structural Learning as a Computational Model of DLN Cognitive StagesPreprint
Wu, A. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.01.703168 · Cited by 1
D’Ambrogio, S., Grohn, J., Khalighinejad, N., Mattar, M. G., Hunt, L., & Rushworth, M. F. S. (2026) · Nature Neuroscience · doi:10.1038/s41593-026-02342-9 · Cited by 1
Uncertainty quantification and exploration–exploitation trade-off in humans
Candelieri, A., Ponti, A., & Archetti, F. (2021) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-021-03547-5 · Cited by 9
Binz, M. (2002) · CRC Press eBooks · doi:10.1201/9780203164587.ch3
Where Do Human Heuristics Come From?Preprint
Binz, M., & Endres, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07580
Impulsivity and risk-seeking as Bayesian inference under dopaminergic controlPreprint
Mikhael, J. G., & Gershman, S. J. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.10.06.327775
Self-citationValue maximization versus directed exploration in human learningPreprint
Dan, O., & Loewenstein, Y. (2020) · doi:10.31234/osf.io/2pxg5
Exploration With a Finite BrainPreprint
Binz, M., & Schulz, E. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/qx4h5
The development of creative search strategiesPreprint
Alison Gopnik, C., Hart, Y., Liquin, E. G., Kosoy, E., Leonard, J., & Mackey, A. P. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/dhwmn
Positive affect modulates memory by regulating the influence of reward prediction errorsPreprint
Qasim, S. E., Deswal, A., Sáez, I., & Gu, X. (2022) · doi:10.31234/osf.io/3972g
Cognitive heuristics are generalisation processes that reduce uncertaintyPreprint
Houser, T. M. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/wjqpb
Basal Ganglia Stimulation Ameliorates Schizophrenia Exploration AnomaliesPreprint
Asch, N., Rahamim, N., Werner-Reiss, U., Israel, Z., & Bergman, H. (2023) · bioRxiv · doi:10.1101/2023.08.13.553111
Coordinating several mental strategies favors integration: Evidence from human judgmentPreprint
Hoffmȧnn, J., Albrecht, R., & von Helversen, B. (2023) · doi:10.31234/osf.io/9yv57
Irrelevant comparisons affect people's exploration strategies in different waysPreprint
Zhang, S., Haofei, W. U., Hu, K., Liu, Q., & Wu, H. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/74qvb
Information spillover effect on human exploratory behavior in contextual multi-armed bandit problem
Nakazato, S., Yang, B., & Shimokawa, T. (2023) · World Symposium on Software Engineering (WSSE) · doi:10.1145/3631991.3632010
Zhang, S., Tian, Y., Liu, Q., & Wu, H. (2024) · eLife · doi:10.7554/elife.92892.1
Multiple and subject-specific roles of uncertainty in reward-guided decision-makingPreprint
Paunov, A. M., L’Hôtellier, M., Guo, D., He, Z., Yu, A. J., & Meyniel, F. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.03.27.587016
Shock to Thrill: Linking Sensation and Information SeekingPreprint
Wong, E., Hauser, T. U., Pietrini, P., & Wu, C. M. (2024) · doi:10.31234/osf.io/xa6yk
Goetmaeckers, F., Goris, J., Wiersema, J. R., Verguts, T., & Braem, S. (2024) · doi:10.31234/osf.io/3mvxk
Goetmaeckers, F., Goris, J., Wiersema, J. R., Verguts, T., & Braem, S. (2024) · Research Square · doi:10.21203/rs.3.rs-4926144/v1
Zhang, S., Tian, Y., Liu, Q., & Wu, H. (2025) · eLife · doi:10.7554/elife.92892.3
Individual differences in sequential decision-makingPreprint
Abbaszadeh, M., Ozanick, E., Magen, N., Darrow, D., Yan, X., Grissom, N., Herman, A. B., & Ebitz, R. B. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.04.04.647306
Estimation-uncertainty affects decisions with and without learning opportunities
Aberg, K. C., Antle, L., & Paz, R. (2025) · Nature Communications · doi:10.1038/s41467-025-61960-2
Composing trajectories for rapid inference of navigational goals
Abdelrahman, N. Y., Jiang, W.-C., Coddington, L. T., Gong, S., Dudman, J. T., & Hermundstad, A. M. (2026) · Neuron · doi:10.1016/j.neuron.2026.05.030 · Cited by 4
Forest-Inspired Reinforcement Learning Based On Nature Ecosystem Feedback Mechanisms
Maskeliūnas, R., & Damaševičius, R. (2025) · Annals of Computer Science and Information Systems · doi:10.15439/2025f7818
Németh, E., Chartouny, A., Jedlovszky, K., Freire, I. T., & Khamassi, M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.10.31.685837
A model-centered introduction to curiosityPreprint
Chartouny, A., Girard, B., & Khamassi, M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.11.19.689250 · Cited by 1
Mochizuki, Y., Aggarwal, M., Harasawa, N., & Chen, C. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-26629-4.00124-6
Habit learning is associated with efficiently controlled network dynamics in naive macaque monkeys
Brynildsen, J. K., Fotiadis, P., Szymula, K. P., Kim, J. Z., Pasqualetti, F., Graybiel, A. M., Desrochers, T. M., & Bassett, D. S. (2026) · npj Complexity · doi:10.1038/s44260-025-00066-8 · Cited by 1
Uncertainty handling in human-AmI interaction for opportunistic software composition
Delcourt, K., Trouilhet, S., & Arcangeli, J.-P. (2026) · Personal and Ubiquitous Computing · doi:10.1007/s00779-026-01853-3
Social learning and exploration–exploitation dilemma in decision-making
Morishita, G., & Suzuki, S. (2026) · Frontiers in Neural Circuits · doi:10.3389/fncir.2026.1781811
Separating random and deterministic sources of computational noise in explore-exploit decisions
Wang, S., & Wilson, R. C. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014026
Egocentric belief attribution enables higher level Theory of Mind computation in group interactionsPreprint
Zhang, S., Wang, H., & Báez-Mendoza, R. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.27.714726
Computational learning noise in human decision-making
Findling, C. (2018) · theses.fr (ABES) · doi:10.70675/32e62279zea97z45b1z9326z06a449dae472
Impact of aging on human decision-making in ecological conditions
Aubert, J. D. S. (2023) · theses.fr (ABES) · doi:10.70675/f7143113zd412z4b5fza432z1032ef6ea209
Aoun-Durand, E. (2024) · theses.fr (ABES) · doi:10.70675/8d2fff4az90d9z4963z9cdfzde888cc5e180
Evaluative Switching in Cognitive Systems under Feasibility ConstraintsPreprint
Toshio, S. A. T. O. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6584981
Predictive processing in volatile environments
Treviño, M., & Márquez, I. (2026) · Behavioural Processes · doi:10.1016/j.beproc.2026.105384
Findling, C., Lee, J. K., Bakermans, J. J. W., Pouget, A., & Wyart, V. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.06.722922
Bell, V., Raihani, N., & Hertz, U. (2022) · OSF Preprints (OSF Preprints)
Boundedly Rational Meta-learning in Sequential Consumer ChoicePreprint
Khosravi, M., Max Kleiman-weiner, A., & Yoganarasimhan, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6755202
Stochastic choice drives variability in patch foraging decisions in humans and rats
Scholey, E. V., Apps, M. A. J., & Humphries, M. D. (2026) · Communications Psychology · doi:10.1038/s44271-026-00465-0
Strategy effects and value directed recall of sub- and supra-span word lists
Babu, H. J., Jantz, T. K., Xie, K. Y., & Reuter‐Lorenz, P. A. (2026) · Memory & Cognition · doi:10.3758/s13421-025-01828-4
[These authors contributed equally: Anna P. Giron, Simon Ciranka.]
Giron, A., Ciranka, S., Schulz, E., van den Bos, W., Ruggeri, A., Meder, B., & Wu, C. (2023) · MPG.PuRe (Max Planck Society)
Comay, N. A., Solovey, G., & Barttfeld, P. (2026) · eLife · doi:10.7554/elife.111820
Comay, N. A., Solovey, G., & Barttfeld, P. (2026) · eLife · doi:10.7554/elife.111820.1
Task-Irrelevant Information Reduces Attentional Uncertainty
Geng, J. J., Shomstein, S., Wan, Q., & McEvoy, K. E. (2026) · Annual Review of Vision Science · doi:10.1146/annurev-vision-102122-112153 · Cited by 1
Representational capacity shapes both the “whether” and “how” of social learning
Taylor-Davies, M. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014504
A Closed-Form Choice Statistic for Entrenchment under a Future-Path DistributionPreprint
Nakazato, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7163064
How Individuals Efficiently Search for Creative Concepts: A Value-Guided Search AccountPreprint
Hao, N., Tong, S., Zhang, W., & Xie, C. (2026) · Research Square · doi:10.21203/rs.3.rs-10201873/v1
Tyrer, A., Nikolova, N. E., Dubois, M., Banellis, L., Vejlø, M., Hauser, T. U., & Allen, M. (2026) · Computational Psychiatry · doi:10.5334/cpsy.164 · Cited by 1
Tyrer, A., Nikolova, N. E., Dubois, M., Banellis, L., Vejlø, M., Hauser, T., & Allen, M. (2026) · MPG.PuRe (Max Planck Society)
Al-Ma’shumah, F., Kirani, F. F., & Ratnawaty, N. (2026) · Journal of Computing Theories and Applications · doi:10.62411/jcta.16818
Problematic social media use is associated with altered directed exploration under uncertainty
Zhou, X., Liu, X., Dou, H., Zhao, H., Liu, Y., & Shen, X. (2026) · Addictive Behaviors Reports · doi:10.1016/j.abrep.2026.100737
Wang, Y., Burgeno, L. M., Cerpa, J.-C., Manohar, S., Bogacz, R., & Walton, M. E. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.08.19.745693
The cognitive scaffolding of tinkering: Directed innovation and strategic social transmission
Tian, S., & Lindström, B. (2026) · Behavioral and Brain Sciences · doi:10.1017/s0140525x2610661x
Valtierra, C., Hogeveen, J., Mace, G., Moreno, S., Ostrand, A. E., Griffith, S., Auil, M. J., McConnell, P. A., Chen, J., Mitchell, J., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.09.21.753261
Individual Differences in Dopaminergic Modulation of ExplorationExploitation Behavior
Smith, E., Theis, H., van Eimeren, T., Knauth, K., Tuzsus, D., Brands, A., Mathar, D., & Peters, J. (2026) · Journal of Neuroscience · doi:10.1523/jneurosci.0261-26.2026
Zhang, H., Zhang, C., Widjaja, A., & Ma, Y. (2026) · iScience · doi:10.1016/j.isci.2026.117887
Self-relevance dynamically biases exploration and exploitation in value-based decisions
Selvaraj, E. S., Golubickis, M., Caughey, S. E., Jalalian, P., Sharma, Y., Khasa, S., Masroujah, L. S., & Macrae, C. N. (2026) · Psychological Research · doi:10.1007/s00426-026-02402-6
Stimulation uncertainty reveals distinct mechanisms of sensation seeking and impulsivity
Wong, E., Hauser, T. U., Chandrasekaran, A., Pietrini, P., & Wu, C. M. (2026) · Communications Psychology · doi:10.1038/s44271-026-00525-5
Effort-based decision-making: A value-guided dissection and arousal-focused synthesis
Clairis, N., & Tobler, P. N. (2026) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2026.107062
Metadata
- Authors
- Gershman, S. J.
- Affiliations
- Harvard University (US)
- Corresponding author
- Gershman, Samuel J.
- ORCID
- Gershman, Samuel J.
- Venue
- Cognition
- Publisher
- Elsevier BV
- Published
- 2018-04
- Volume
- 173
- Pages
- 34-42
- ISSN
- 0010-0277 (ISSN-L)
- PMC
- PMC5801139
- OpenAlex
- W2778689844
- Cited by
- 267
- References
- 46
- Citations by year
- 2026: 38 · 2025: 24 · 2024: 49 · 2023: 35 · 2022: 33 · 2021: 27 · 2020: 29 · 2019: 19 · 2018: 11
- FWCI
- 9.79
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Cognition (acceptedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- NSF Collaborative Research in Computational Neuroscience (IIS-120 7833)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- exploration-exploitation balance, human decision-making, uncertainty, exploration algorithms, response bias, reinforcement learning, computational modeling, hybrid model
- MeSH terms
- Adult, Algorithms, Choice Behavior, Exploratory Behavior, Humans, Psychomotor Performance, Reinforcement, Psychology, Reward, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
