Deconstructing the human algorithms for exploration
Open accessAbstract
The dilemma between information gathering (exploration) and reward seeking (exploitation) is a fundamental problem for reinforcement learning agents. How humans resolve this dilemma is still an open question, because experiments have provided equivocal evidence about the underlying algorithms used by humans. We show that two families of algorithms can be distinguished in terms of how uncertainty affects exploration. Algorithms based on uncertainty bonuses predict a change in response bias as a function of uncertainty, whereas algorithms based on sampling predict a change in response slope. Two experiments provide evidence for both bias and slope changes, and computational modeling confirms that a hybrid model is the best quantitative account of the data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
UCB exploration bonuses are linked to Bayesian posterior uncertainty: as samples accumulate, the posterior distribution over an option's value narrows, reducing the exploration bonus.
“UCB exploration bonuses are linked to Bayesian posterior uncertainty: as samples accumulate, the posterior distribution over an option’s value narrows, reducing the exploration bonus20.”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
The model predicted behavior in a different explore-exploit task.
“The model has some generalizability: it predicted behavior in a different explore–exploit task20 (Fig. 5b).”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
The symbolic functions outperformed the UCB model in predicting participants' choices in an independent two-armed bandit dataset (Wilcoxon signed-rank test: W = 15.0, n = 89, median difference = −0.07, P = 4.31 × 10−16).
“Remarkably, our symbolic functions also outperformed the UCB model in predicting participants’ choices in this different context (Wilcoxon signed-rank test: W = 15.0, n = 89, median difference = −0.07, P = 4.31 × 10−16).”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824 · see citing works
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033 · see citing works
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712 · see citing works
On the Rate of Gain of Information
Hick, W. E. (1952) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470215208416600 · Cited by 2,279 · see citing works
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224 · see citing works
Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. (2012) · IEEE Transactions on Information Theory · doi:10.48550/arxiv.0912.3995 · Cited by 2,100 · see citing works
Stimulus information as a determinant of reaction time.
Hyman, R. (1953) · Journal of Experimental Psychology · doi:10.1037/h0056940 · Cited by 1,443 · see citing works
Bayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423 · see citing works
Cohen, J. D., McClure, S. M., & Yu, A. J. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2098 · Cited by 1,121 · see citing works
(2011) · Cited by 1,000 · see citing works
Small feedback‐based decisions and their limited correspondence to description‐based decisions
Barron, G., & Erev, I. (2003) · Journal of Behavioral Decision Making · doi:10.1002/bdm.443 · Cited by 725 · see citing works
Bayesian model selection for group studies — Revisited
Rigoux, L., Stephan, K. E., Friston, K. J., & Jean Daunizeau, A. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.08.065 · Cited by 659 · see citing works
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 636 · see citing works
Humans use directed and random exploration to solve the explore–exploit dilemma.
Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A., & Cohen, J. D. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0038199 · Cited by 580 · see citing works
Frank, M. J., Doll, B. B., Oas-Terpstra, J., & Moreno, F. (2009) · Nature Neuroscience · doi:10.1038/nn.2342 · Cited by 452 · see citing works
The Oxford Handbook of Causal Reasoning
(2017) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199399550.001.0001 · Cited by 412 · see citing works
Unpacking the exploration–exploitation tradeoff: A synthesis of human and animal literatures.
Mehlhorn, K., Newell, B. R., Todd, P. M., Lee, M. D., Morgan, K. V., Braithwaite, V. A., Hausmann, D., Fiedler, K., & González, C. (2015) · Decision · doi:10.1037/dec0000033 · Cited by 383 · see citing works
Milosavljevic, M., Malmaud, J., Huth, A. G., Christof Koch, A., & Rangel, A. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500001285 · Cited by 320 · see citing works
Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings
Payzan-LeNestour, E., & Bossaerts, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001048 · Cited by 267 · see citing works
Instance-based learning: Integrating sampling and repeated decisions from experience.
Gonzalez, C., & Dutt, V. (2011) · Psychological Review · doi:10.1037/a0024558 · Cited by 255 · see citing works
Retrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217 · see citing works
Self-citationOptimistic bayesian sampling in contextual-bandit problems
May, B. C., Korda, N., Lee, A., & Leslie, D. S. (2012) · Lancaster EPrints (Lancaster University) · doi:10.5555/2188385.2343711 · Cited by 213 · see citing works
Optimal policy for value-based decision-making
Tajima, S., Drugowitsch, J., & Pouget, A. (2016) · Nature Communications · doi:10.1038/ncomms12400 · Cited by 209 · see citing works
Ratcliff, R., & Frank, M. J. (2012) · Neural Computation · doi:10.1162/neco_a_00270 · Cited by 203 · see citing works
The algorithmic anatomy of model-based evaluation
Daw, N. D., & Dayan, P. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0478 · Cited by 195 · see citing works
Pearson, J. M., Hayden, B. Y., Raghavachari, S., & Platt, M. L. (2009) · Current Biology · doi:10.1016/j.cub.2009.07.048 · Cited by 185 · see citing works
Uncertainty and Exploration in a Restless Bandit Problem
Speekenbrink, M., & Konstantinidis, E. (2015) · Topics in Cognitive Science · doi:10.1111/tops.12145 · Cited by 177 · see citing works
Summerfield, C., & Tsetsos, K. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00070 · Cited by 167 · see citing works
A causal role for right frontopolar cortex in directed, but not random, exploration
Zajkowski, W. K., Kossut, M. M., & Wilson, R. C. (2017) · eLife · doi:10.7554/elife.27430 · Cited by 164 · see citing works
Charting the expansion of strategic exploratory behavior during adolescence.
Somerville, L. H., Sasse, S. F., Garrad, M. C., Drysdale, A. T., Akar, N. A., Insel, C., & Wilson, R. C. (2016) · Journal of Experimental Psychology General · doi:10.1037/xge0000250 · Cited by 147 · see citing works
Entropy and Choice Time: The Effect of Frequency Unbalance on Choice-Response
Crossman, E. R. F. W. (1953) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470215308416625 · Cited by 142 · see citing works
Instance‐based Learning: A General Model of Repeated Binary Choice
Lejarraga, T., Dutt, V., & González, C. (2010) · Journal of Behavioral Decision Making · doi:10.1002/bdm.722 · Cited by 122 · see citing works
Sequential Choice Under Ambiguity: Intuitive Solutions to the Armed-Bandit Problem
Meyer, R. J., & Shi, Y. (1995) · Management Science · doi:10.1287/mnsc.41.5.817 · Cited by 118 · see citing works
Novelty and Inductive Generalization in Human Reinforcement Learning
Gershman, S., & Niv, Y. (2015) · Topics in Cognitive Science · doi:10.1111/tops.12138 · Cited by 102 · see citing works
Self-citationPsychological models of human and optimal performance in bandit problems
Lee, M. D., Zhang, S., Munro, M., & Steyvers, M. (2010) · Cognitive Systems Research · doi:10.1016/j.cogsys.2010.07.007 · Cited by 87 · see citing works
The Nature of Belief-Directed Exploratory Choice in Human Decision-Making
Knox, W. B., Otto, A. R., Stone, P., & Love, B. C. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00398 · Cited by 77 · see citing works
Forgetful Bayes and myopic planning: Human learning and decision-making in a bandit setting
Zhang, S., & Yu, A. J. (2013) · Cited by 70 · see citing works
The effect of atomoxetine on random and directed exploration in humans
Warren, C. M., Wilson, R. C., van der Wee, N. J., Giltay, E. J., van Noorden, M. S., Cohen, J. D., & Nieuwenhuis, S. (2017) · PLoS ONE · doi:10.1371/journal.pone.0176034 · Cited by 63 · see citing works
Reinforcement Learning and Causal Models
Gershman, S. (2017) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199399550.013.20 · Cited by 63 · see citing works
Self-citationStructure Learning in Human Sequential Decision-Making
Acuna, D. E., & Schrater, P. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001003 · Cited by 60 · see citing works
Payzan-LeNestour, É., & Bossaerts, P. L. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00150 · Cited by 54 · see citing works
Discovering hierarchical motion structure
Gershman, S. J., Tenenbaum, J. B., & Jäkel, F. (2015) · Vision Research · doi:10.1016/j.visres.2015.03.004 · Cited by 46 · see citing works
Self-citationStrategies for exploration in the domain of losses
Krueger, P. M., Wilson, R. C., & Cohen, J. D. (2017) · Judgment and Decision Making · doi:10.1017/s1930297500005659 · Cited by 33 · see citing works
Physiological and behavioral signatures of reflective exploratory choice
Otto, A. R., Knox, W. B., Markman, A. B., & Love, B. C. (2014) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-014-0260-4 · Cited by 27 · see citing works
Learning and decisions in contextual multi-armed bandit tasks
Schulz, E., Konstantinidis, E., & Speekenbrink, M. (2015) · MPG.PuRe (Max Planck Society) · Cited by 20 · see citing works
Flagged first, then by citations
1 of the 46 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Srinivas N Krause A Seeger M Kakade SM 2010 Gaussian process optimization in the bandit setting: No regret and experimental design Proceedings of the 27th international conference on machine learning 1015 1022
Cited by
Sorted by citations
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269 · see citing works
Computational mechanisms of curiosity and goal-directed exploration
Schwartenbeck, P., Passecker, J., Hauser, T. U., FitzGerald, T. H. B., Kronbichler, M., & Friston, K. J. (2019) · eLife · doi:10.7554/elife.41703 · Cited by 250 · see citing works
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236 · see citing works
Self-citationWhen Is More Uncertainty Better? A Model of Uncertainty Regulation and Effectiveness
Griffin, M. A., & Grote, G. (2020) · Academy of Management Review · doi:10.5465/amr.2018.0271 · Cited by 223 · see citing works
Balancing exploration and exploitation with information and randomization
Wilson, R. C., Bonawitz, E. B., Costa, V. D., & Ebitz, R. B. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.10.001 · Cited by 215 · see citing works
Computational noise in reward-guided learning drives behavioral variability in volatile environments
Findling, C., Skvortsova, V., Dromnelle, R., Palminteri, S., & Wyart, V. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0518-9 · Cited by 196 · see citing works
The algorithmic architecture of exploration in the human brain
Schulz, E., & Gershman, S. J. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2018.11.003 · Cited by 184 · see citing works
Self-citationSearching for Rewards Like a Child Means Less Generalization and More Directed Exploration
Schulz, E., Wu, C. M., Ruggeri, A., & Meder, B. (2019) · Psychological Science · doi:10.1177/0956797619863663 · Cited by 166 · see citing works
Children are more exploratory and learn more than adults in an approach-avoid task
Liquin, E. G., & Alison Gopnik, C. (2021) · Cognition · doi:10.1016/j.cognition.2021.104940 · Cited by 150 · see citing works
Gershman, S. J. (2018) · Decision · doi:10.1037/dec0000101 · Cited by 146 · see citing works
Self-citationDopaminergic modulation of the exploration/exploitation trade-off in human decision-making
Chakroun, K., Mathar, D., Wiehler, A., Ganzer, F., & Peters, J. (2020) · eLife · doi:10.7554/elife.51260 · Cited by 121 · see citing works
The temporal dynamics of opportunity costs: A normative account of cognitive fatigue and boredom.
Agrawal, M., Mattar, M. G., Cohen, J. D., & Daw, N. D. (2021) · Psychological Review · doi:10.1037/rev0000309 · Cited by 131 · see citing works
Controllability governs the balance between Pavlovian and instrumental action selection
Dorfman, H. M., & Gershman, S. (2019) · Nature Communications · doi:10.1038/s41467-019-13737-7 · Cited by 86 · see citing works
Self-citationDevelopment of directed and random exploration in children
Meder, B., Wu, C. M., Schulz, E., & Ruggeri, A. (2021) · Developmental Science · doi:10.1111/desc.13095 · Cited by 114 · see citing works
Deep active inference as variational policy gradients
Millidge, B. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102348 · Cited by 115 · see citing works
Structured, uncertainty-driven exploration in real-world consumer choice
Schulz, E., Bhui, R., Love, B. C., Brier, B., Todd, M. T., & Gershman, S. J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1821028116 · Cited by 111 · see citing works
Self-citationLapses in perceptual decisions reflect exploration
Pisupati, S., Chartarifsky, L., Khanal, A., & Churchland, A. K. (2021) · eLife · doi:10.7554/elife.55490 · Cited by 105 · see citing works
Dissociable neural correlates of uncertainty underlie different exploration strategies
Tomov, M. S., Truong, V. Q., Hundia, R., & Gershman, S. J. (2020) · Nature Communications · doi:10.1038/s41467-020-15766-z · Cited by 101 · see citing works
Self-citationAdaptive Regulation of Motor Variability
Dhawale, A. K., Miyamoto, Y. R., Smith, M. A., & Ölveczky, B. P. (2019) · Current Biology · doi:10.1016/j.cub.2019.08.052 · Cited by 94 · see citing works
Time pressure changes how people explore and respond to uncertainty
Wu, C. M., Schulz, E., Pleskac, T. J., & Speekenbrink, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-07901-1 · Cited by 89 · see citing works
A tutorial on open-source large language models for behavioral science
Hussain, Z., Binz, M., Mata, R., & Wulff, D. U. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02455-8 · Cited by 91 · see citing works
Uncertainty as a driver of the youth mental health crisis
Schweizer, S., Lawson, R. P., & Blakemore, S.-J. (2023) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2023.101657 · Cited by 76 · see citing works
Human complex exploration strategies are enriched by noradrenaline-modulated heuristics
Dubois, M., Habicht, J., Michely, J., Moran, R., Dolan, R. J., & Hauser, T. U. (2021) · eLife · doi:10.7554/elife.59907 · Cited by 76 · see citing works
Computation noise in human learning and decision-making: origin, impact, function
Findling, C., & Wyart, V. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.02.018 · Cited by 75 · see citing works
Cockburn, J., Man, V. Y., Cunningham, W. A., & O’Doherty, J. P. (2022) · Neuron · doi:10.1016/j.neuron.2022.05.025 · Cited by 72 · see citing works
Dubourg, E., & Baumard, N. (2021) · Behavioral and Brain Sciences · doi:10.1017/s0140525x21000923 · Cited by 75 · see citing works
Value-free random exploration is linked to impulsivity
Dubois, M., & Hauser, T. U. (2022) · Nature Communications · doi:10.1038/s41467-022-31918-9 · Cited by 72 · see citing works
Developmental changes in exploration resemble stochastic optimization
Giron, A. P., Ciranka, S., Schulz, E., van den Bos, W., Ruggeri, A., Meder, B., & Wu, C. M. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01662-1 · Cited by 71 · see citing works
Similarities and differences in spatial and non-spatial cognitive maps
Wu, C. M., Schulz, E., Garvert, M. M., Meder, B., & Schuck, N. W. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008149 · Cited by 57 · see citing works
Hierarchical models of pain: Inference, information-seeking, and adaptive control.
Seymour, B., & Mancini, F. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117212 · Cited by 53 · see citing works
Finding structure in multi-armed bandits
Schulz, E., Franklin, N. T., & Gershman, S. J. (2018) · Cognitive Psychology · doi:10.1016/j.cogpsych.2019.101261 · Cited by 59 · see citing works
Self-citationOn the normative advantages of dopamine and striatal opponency for learning and choice
Jaskir, A., & Frank, M. J. (2023) · eLife · doi:10.7554/elife.85107 · Cited by 53 · see citing works
Novelty and uncertainty differentially drive exploration across development
Nussenbaum, K., Martin, R., Maulhardt, S., Yang, Y., Bizzell-Hatcher, G., Bhatt, N. S., König, M., Rosenbaum, G. M., O’Doherty, J. P., Cockburn, J., et al. (2023) · eLife · doi:10.7554/elife.84260 · Cited by 52 · see citing works
Fan, H., Gershman, S. J., & Phelps, E. A. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01455-y · Cited by 60 · see citing works
Self-citationAnnual Research Review: Developmental computational psychiatry
Hauser, T. U., Will, G., Dubois, M., & Dolan, R. J. (2018) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12964 · Cited by 49 · see citing works
It’s new, but is it good? How generalization and uncertainty guide the exploration of novel options.
Stojić, H., Schulz, E., Analytis, P. P., & Speekenbrink, M. (2020) · Journal of Experimental Psychology General · doi:10.1037/xge0000749 · Cited by 52 · see citing works
Smith, R. S., Taylor, S., Wilson, R. C., Chuning, A. E., Persich, M. R., Wang, S., & Killgore, W. D. S. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.782136 · Cited by 51 · see citing works
Dopaminergic genes are associated with both directed and random exploration
Gershman, S. J., & Tzovaras, B. G. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.10.009 · Cited by 50 · see citing works
Self-citationLloyd, A., McKay, R., Sebastian, C. L., & Balsters, J. H. (2021) · Developmental Science · doi:10.1111/desc.13075 · Cited by 49 · see citing works
Curiosity: primate neural circuits for novelty and information seeking
Monosov, I. E. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00784-9 · Cited by 44 · see citing works
Aberg, K. C., Toren, I., & Paz, R. (2021) · Molecular Psychiatry · doi:10.1038/s41380-021-01363-z · Cited by 44 · see citing works
Olschewski, S., Luckman, A., Mason, A., Ludvig, E. A., & Konstantinidis, E. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231179138 · Cited by 44 · see citing works
Brown, V. M., Hallquist, M. N., Frank, M. J., & Dombrovski, A. Y. (2022) · Cognition · doi:10.1016/j.cognition.2022.105233 · Cited by 43 · see citing works
Meta-learned models of cognition
Binz, M., Dasgupta, I., Jagadish, A. K., Botvinick, M., Wang, J. X., & Schulz, E. (2024) · Behavioral and Brain Sciences · doi:10.1017/s0140525x23003266 · Cited by 38 · see citing works
Temporal discounting correlates with directed exploration but not with random exploration
Sadeghiyeh, H., Wang, S., Alberhasky, M. R., Kyllo, H. M., Shenhav, A., & Wilson, R. C. (2020) · Scientific Reports · doi:10.1038/s41598-020-60576-4 · Cited by 35 · see citing works
Human-AI joint task performance: Learning from uncertainty in autonomous driving systems
Constantinides, P., Monteiro, E., & Mathiassen, L. (2024) · Information and Organization · doi:10.1016/j.infoandorg.2024.100502 · Cited by 36 · see citing works
Travers, E., Friedemann, M., & Haggard, P. (2020) · Cognitive Neuroscience · doi:10.1080/17588928.2020.1824176 · Cited by 38 · see citing works
Xu, T., Singh, K., & Rajivan, P. (2022) · Applied Ergonomics · doi:10.1016/j.apergo.2022.103908 · Cited by 37 · see citing works
Uncertainty in learning, choice, and visual fixation
Stojić, H., Orquin, J. L., Dayan, P., Dolan, R. J., & Speekenbrink, M. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1911348117 · Cited by 32 · see citing works
Reward maximization through discrete active inference
Da Costa, L., Sajid, N., Parr, T., Friston, K., & Smith, R. (2023) · Neural Computation · doi:10.1162/neco_a_01574 · Cited by 35 · see citing works
Neurons in human pre-supplementary motor area encode key computations for value-based choice
Aquino, T. G., Cockburn, J., Mamelak, A. N., Rutishauser, U., & O’Doherty, J. P. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01548-2 · Cited by 33 · see citing works
Reference effects on decision-making elicited by previous rewards
Rigoli, F. (2019) · Cognition · doi:10.1016/j.cognition.2019.104034 · Cited by 33 · see citing works
Searching for rewards like a child means less generalization and more directed explorationPreprint
Schulz, E., Wu, C. M., Ruggeri, A., & Meder, B. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/327593 · Cited by 33 · see citing works
Rational inattention and tonic dopamine
Mikhael, J. G., Lai, L., & Gershman, S. J. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008659 · Cited by 30 · see citing works
Self-citationSmith, R. S., Varshney, L. R., Nagayama, S., Kazama, M., Kitagawa, T., Shunsuke, M., & Ishikawa, Y. (2022) · International Journal of Wellbeing · doi:10.5502/ijw.v12i4.2659 · Cited by 31 · see citing works
Expecting the unexpected: a review of learning under uncertainty across development
Topel, S., Ma, I., Sleutels, J. J. M., van Steenbergen, H., de Bruijn, E. R. A., & van Duijvenvoorde, A. C. K. (2023) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-023-01098-0 · Cited by 34 · see citing works
Franklin, N. T., & Frank, M. J. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007720 · Cited by 33 · see citing works
Luo, Q., Kanen, J. W., Bari, A., Skandali, N., Langley, C., Knudsen, G. M., Alsiö, J., Phillips, B. U., Sahakian, B. J., Cardinal, R. N., et al. (2023) · Neuropsychopharmacology · doi:10.1038/s41386-023-01762-6 · Cited by 26 · see citing works
Unifying Principles of Generalization: Past, Present, and Future
Wu, C. M., Meder, B., & Schulz, E. (2024) · Annual Review of Psychology · doi:10.1146/annurev-psych-021524-110810 · Cited by 33 · see citing works
The dynamics of explore–exploit decisions reveal a signal-to-noise mechanism for random exploration
Feng, S. F., Wang, S., Zarnescu, S., & Wilson, R. C. (2021) · Scientific Reports · doi:10.1038/s41598-021-82530-8 · Cited by 31 · see citing works
Inference and Search on Graph-Structured Spaces
Wu, C. M., Schulz, E., & Gershman, S. J. (2020) · Computational Brain & Behavior · doi:10.1007/s42113-020-00091-x · Cited by 26 · see citing works
Self-citationGlobally Inaccurate Stereotypes Can Result From Locally Adaptive Exploration
Bai, X., Fiske, S. T., & Griffiths, T. L. (2022) · Psychological Science · doi:10.1177/09567976211045929 · Cited by 26 · see citing works
Modernizing Conceptions of Valuation and Cognitive-Control Deployment in Adolescent Risk Taking
Do, K. T., Sharp, P. B., & Telzer, E. H. (2019) · Current Directions in Psychological Science · doi:10.1177/0963721419887361 · Cited by 26 · see citing works
Empowerment contributes to exploration behaviour in a creative video game
Brändle, F., Stocks, L. J., Tenenbaum, J. B., Gershman, S. J., & Schulz, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01661-2 · Cited by 34 · see citing works
Self-citationExploration heuristics decrease during youth
Dubois, M., Bowler, A., Moses-Payne, M. E., Habicht, J., Moran, R., Steinbeis, N., & Hauser, T. U. (2022) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-022-01009-9 · Cited by 25 · see citing works
Deep exploration as a unifying account of explore-exploit behaviorPreprint
Wilson, R. C., Wang, S., Sadeghiyeh, H., & Cohen, J. D. (2020) · doi:10.31234/osf.io/uj85c · Cited by 25 · see citing works
The development of creative search strategies
Hart, Y., Kosoy, E., Liquin, E. G., Leonard, J., Mackey, A. P., & Alison Gopnik, C. (2022) · Cognition · doi:10.1016/j.cognition.2022.105102 · Cited by 24 · see citing works
Hertz, U., Bell, V., & Raihani, N. (2021) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2021.1414 · Cited by 23 · see citing works
The modulation of brain network integration and arousal during exploration
Tardiff, N., Medaglia, J. D., Bassett, D. S., & Thompson‐Schill, S. L. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118369 · Cited by 23 · see citing works
A theory of relation learning and cross-domain generalization.
Doumas, L. A. A., Puebla, G., Martin, A. E., & Hummel, J. E. (2022) · Psychological Review · doi:10.1037/rev0000346 · Cited by 25 · see citing works
Chasing Unknown Bandits: Uncertainty Guidance in Learning and Decision Making
Speekenbrink, M. (2022) · Current Directions in Psychological Science · doi:10.1177/09637214221105051 · Cited by 23 · see citing works
Impaired value-based decision-making in Parkinson’s disease apathy
Gilmour, W., Mackenzie, G. M., Feile, M., Tayler-Grint, L., Suveges, S., Macfarlane, J. A., Macleod, A. D., Marshall, V. L., Grunwald, I. Q., Steele, J. D., et al. (2024) · Brain · doi:10.1093/brain/awae025 · Cited by 23 · see citing works
Neural mechanisms of information seeking
Kobayashi, K., & Kable, J. W. (2024) · Neuron · doi:10.1016/j.neuron.2024.04.008 · Cited by 24 · see citing works
Accumbofrontal tract integrity is related to early life adversity and feedback learning
Kennedy, B. V., Hanson, J. L., Buser, N. J., van den Bos, W., Rudolph, K. D., Davidson, R. J., & Pollak, S. D. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-021-01129-9 · Cited by 20 · see citing works
Adaptive tuning of human learning and choice variability to unexpected uncertainty
Lee, J. K., Rouault, M., & Wyart, V. (2023) · Science Advances · doi:10.1126/sciadv.add0501 · Cited by 20 · see citing works
Children perform extensive information gathering when it is not costly
Bowler, A., Habicht, J., Moses-Payne, M. E., Steinbeis, N., Moutoussis, M., & Hauser, T. U. (2020) · Cognition · doi:10.1016/j.cognition.2020.104535 · Cited by 19 · see citing works
Ger, Y., Nachmani, E., Wolf, L., & Shahar, N. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011678 · Cited by 19 · see citing works
Combrisson, E., Basanisi, R., Neri, M., Auzias, G., Petri, G., Marinazzo, D., Panzeri, S., & Brovelli, A. (2025) · Nature Communications · doi:10.1038/s41467-025-62507-1 · Cited by 18 · see citing works
Exploratory preferences explain the human fascination for imaginary worlds in fictional stories
Dubourg, E., Thouzeau, V., de Dampierre, C., Mogoutov, A., & Baumard, N. (2023) · Scientific Reports · doi:10.1038/s41598-023-35151-2 · Cited by 19 · see citing works
Pupil Size Encodes Uncertainty during Exploration
Fan, H., Burke, T. D., Sambrano, D. C., Dial, E., Phelps, E. A., & Gershman, S. J. (2023) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02025 · Cited by 18 · see citing works
Self-citationFan, H., Gershman, S. J., & Phelps, E. A. (2021) · doi:10.31234/osf.io/yx6sb · Cited by 18 · see citing works
Self-citationUncertainty–guided learning with scaled prediction errors in the basal ganglia
Möller, M., Manohar, S., & Bogacz, R. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009816 · Cited by 16 · see citing works
Selective maintenance of value information helps resolve the exploration/exploitation dilemma
Hallquist, M. N., & Dombrovski, A. Y. (2018) · Cognition · doi:10.1016/j.cognition.2018.11.004 · Cited by 16 · see citing works
Distinct motivations to seek out information in healthy individuals and problem gamblers
Dezza, I. C., Noël, X., Cleeremans, A., & Yu, A. J. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01523-3 · Cited by 16 · see citing works
The influence of anxiety on exploration: A review of computational modeling studies
Chou, K.-P., Wilson, R. C., & Smith, R. S. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105940 · Cited by 15 · see citing works
Simulating News Recommendation Ecosystems for Insights and Implications
Zhang, G., Li, D., Gu, H., Lu, T., Shang, L., & Gu, N. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3381329 · Cited by 15 · see citing works
Behavior Decision of Mobile Robot With a Neurophysiologically Motivated Reinforcement Learning Model
Wang, D., Chen, S., Hu, Y., Liu, L., & Wang, H. (2020) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2020.3035778 · Cited by 14 · see citing works
The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons
Dubey, R., Griffiths, T. L., & Dayan, P. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010316 · Cited by 16 · see citing works
Lapses in perceptual decisions reflect explorationPreprint
Pisupati, S., Chartarifsky-Lynn, L., Khanal, A., & Churchland, A. K. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/613828 · Cited by 15 · see citing works
Time pressure changes how people explore and respond to uncertaintyPreprint
Wu, C. M., Schulz, E., Pleskac, T. J., & Speekenbrink, M. (2021) · doi:10.31234/osf.io/dsw7q · Cited by 14 · see citing works
Human inference in changing environments with temporal structure.
Prat-Carrabin, A., Wilson, R. C., Cohen, J. D., & Azeredo da Silveira, R. (2021) · Psychological Review · doi:10.1037/rev0000276 · Cited by 16 · see citing works
“Several studies have reported that the responses of human subjects in explore-exploit tasks appeared consistent with Thompson sampling [84, 85, 86].”
ResultThe effects of time horizon and guided choices on explore–exploit decisions in rodents.
Wang, S., Gerken, B., Wieland, J. R., Wilson, R. C., & Fellous, J. (2023) · Behavioral Neuroscience · doi:10.1037/bne0000549 · Cited by 13 · see citing works
A Comparison of Rapid Rule-Learning Strategies in Humans and Monkeys
Goudar, V., Kim, J.-W., Liu, Y., Dede, A. J. O., Jutras, M. J., Skelin, I., Ruvalcaba, M., Chang, W. K., Ram, B., Fairhall, A. L., et al. (2024) · Journal of Neuroscience · doi:10.1523/jneurosci.0231-23.2024 · Cited by 13 · see citing works
Differential impacts of healthy cognitive aging on directed and random exploration.
Mizell, J.-M., Wang, S., Frisvold, A., Alvarado, L., Farrell-Skupny, A., Keung, W., Phelps, C. E., Sundman, M., Franchetti, M.-K., Chou, Y., et al. (2024) · Psychology and Aging · doi:10.1037/pag0000791 · Cited by 14 · see citing works
Developmental changes resemble stochastic optimizationPreprint
Giron, A. P., Ciranka, S., Schulz, E., van den Bos, W., Ruggeri, A., Meder, B., & Wu, C. M. (2022) · doi:10.31234/osf.io/9f4k3 · Cited by 13 · see citing works
Bayesian Reinforcement Learning With Limited Cognitive Load
Arumugam, D., Ho, M. K., Goodman, N. D., & Roy, B. V. (2024) · Open Mind · doi:10.1162/opmi_a_00132 · Cited by 12 · see citing works
Brändle, F., Binz, M., & Schulz, E. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009026949.008 · Cited by 12 · see citing works
Horváth, L., Colcombe, S. J., Milham, M. P., Ray, S., Schwartenbeck, P., & Ostwald, D. (2021) · Computational Brain & Behavior · doi:10.1007/s42113-021-00112-3 · Cited by 12 · see citing works
Compositional Sequence Generation in the Entorhinal–Hippocampal System
McNamee, D., Stachenfeld, K., Botvinick, M., & Gershman, S. J. (2022) · Entropy · doi:10.3390/e24121791 · Cited by 9 · see citing works
Self-citationLukas, D., McCune, K. B., Blaisdell, A. P., Johnson‐Ulrich, Z., MacPherson, M. P., Seitz, B. M., Sevchik, A., & Logan, C. (2024) · Peer Community Journal · doi:10.24072/pcjournal.456 · Cited by 12 · see citing works
- Authors
- Gershman, S. J.
- Affiliations
- Harvard University (US)
- Corresponding author
- Gershman, Samuel J.
- ORCID
- Gershman, Samuel J.
- Venue
- Cognition
- Publisher
- Elsevier BV
- Published
- 2018-04
- Volume
- 173
- Pages
- 34-42
- ISSN
- 0010-0277 (ISSN-L)
- PMC
- PMC5801139
- OpenAlex
- W2778689844
- Cited by
- 267
- References
- 46
- Citations by year
- 2026: 38 · 2025: 24 · 2024: 49 · 2023: 35 · 2022: 33 · 2021: 27 · 2020: 29 · 2019: 19 · 2018: 11 · 2017: 1
- FWCI
- 9.79
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Cognition (acceptedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- NSF Collaborative Research in Computational Neuroscience (IIS-120 7833)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- exploration-exploitation balance, human decision-making, uncertainty, exploration algorithms, response bias, reinforcement learning, computational modeling, hybrid model
- MeSH terms
- Adult, Algorithms, Choice Behavior, Exploratory Behavior, Humans, Psychomotor Performance, Reinforcement, Psychology, Reward, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
