Believing in dopamine
Open accessAbstract
Midbrain dopamine signals are widely thought to report reward prediction errors that drive learning in the basal ganglia. However, dopamine has also been implicated in various probabilistic computations, such as encoding uncertainty and controlling exploration. Here, we show how these different facets of dopamine signalling can be brought together under a common reinforcement learning framework. The key idea is that multiple sources of uncertainty impinge on reinforcement learning computations: uncertainty about the state of the environment, the parameters of the value function and the optimal action policy. Each of these sources plays a distinct role in the prefrontal cortex-basal ganglia circuit for reinforcement learning and is ultimately reflected in dopamine activity. The view that dopamine plays a central role in the encoding and updating of beliefs brings the classical prediction error theory into alignment with more recent theories of Bayesian reinforcement learning.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The brain may have dedicated machinery, perhaps in prefrontal cortex, for computing belief states, which could then be provided to downstream areas, such as the basal ganglia, to perform standard RL algorithms such as TD learning.
“The brain may have dedicated machinery, perhaps in prefrontal cortex [12,13], for computing belief states, which could then be provided to downstream areas, such as the basal ganglia, to perform standard RL algorithms such as TD learning [14].”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · see citing works
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · see citing works
Scalar expectancy theory and Weber's law in animal timing.
Gibbon, J. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.279 · Cited by 2,224 · see citing works
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053 · see citing works
Decision field theory: a dynamic-cognitive approach to decision making in an uncertain environment.
Busemeyer, J., & Townsend, J. (1993) · Psychology Review · doi:10.1037/0033-295x.100.3.432 · Cited by 2,000 · see citing works
Gibbon, J., Church, R. M., & Meck, W. H. (1984) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1984.tb23417.x · Cited by 1,806 · see citing works
Bayesian inference with probabilistic population codes
Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006) · Nature Neuroscience · doi:10.1038/nn1790 · Cited by 1,457 · see citing works
Adaptive Coding of Reward Value by Dopamine Neurons
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005) · Science · doi:10.1126/science.1105370 · Cited by 1,276 · see citing works
Active inference: a process theory
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., & Pezzulo, G. (2017) · Neural Computation · doi:10.1162/neco_a_00912 · Cited by 1,156 · see citing works
Tonic dopamine: Opportunity costs and the control of response vigor
Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0502-4 · Cited by 1,133 · see citing works
Dopamine neurons report an error in the temporal prediction of reward during learning
Hollerman, J. R., & Schultz, W. (1998) · Nature Neuroscience · doi:10.1038/1124 · Cited by 1,103 · see citing works
Dopamine responses comply with basic assumptions of formal learning theory
Waelti, P., Dickinson, A., & Schultz, W. (2001) · Nature · doi:10.1038/35083500 · Cited by 1,018 · see citing works
Orbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885 · see citing works
A causal link between prediction errors, dopamine neurons and learning
Steinberg, E. E., Keiflin, R., Boivin, J. R., Witten, I. B., Deisseroth, K. A., & Janak, P. H. (2013) · Nature Neuroscience · doi:10.1038/nn.3413 · Cited by 879 · see citing works
Lubow, R. E. (1973) · Psychological Bulletin · doi:10.1037/h0034425 · Cited by 793 · see citing works
Active inference and epistemic value
Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., & Pezzulo, G. (2015) · Cognitive Neuroscience · doi:10.1080/17588928.2015.1020053 · Cited by 731 · see citing works
Humans use directed and random exploration to solve the explore–exploit dilemma.
Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A., & Cohen, J. D. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0038199 · Cited by 580 · see citing works
Frank, M. J., & Claus, E. D. (2006) · Psychological Review · doi:10.1037/0033-295x.113.2.300 · Cited by 578 · see citing works
Optimal representation of sensory information by neural populations
Jazayeri, M., & Movshon, J. A. (2006) · Nature Neuroscience · doi:10.1038/nn1691 · Cited by 530 · see citing works
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498 · see citing works
Dopamine: generalization and bonuses
Kakade, S. M., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00048-5 · Cited by 477 · see citing works
Bayesian theories of conditioning in a changing world
Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.004 · Cited by 457 · see citing works
Frank, M. J., Doll, B. B., Oas-Terpstra, J., & Moreno, F. (2009) · Nature Neuroscience · doi:10.1038/nn.2342 · Cited by 452 · see citing works
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446 · see citing works
Self-citationNeural Circuitry of Reward Prediction Error
Watabe-Uchida, M., Eshel, N., & Uchida, N. (2017) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-072116-031109 · Cited by 437 · see citing works
Self-citationPan, W.-X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.1478-05.2005 · Cited by 429 · see citing works
A Scalable Population Code for Time in the Striatum
Mello, G. B. M. E., Soares, S., & Paton, J. J. (2015) · Current Biology · doi:10.1016/j.cub.2015.02.036 · Cited by 393 · see citing works
Context, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386 · see citing works
Self-citationBuesing, L., Bill, J., Nessler, B., & Maass, W. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002211 · Cited by 384 · see citing works
Dopamine, affordance and active inference
Friston, K. J., Shiner, T., FitzGerald, T., Galea, J. M., Adams, R., Brown, H., Dolan, R. J., Moran, R., Stephan, K. E., & Bestmann, S. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002327 · Cited by 381 · see citing works
Redish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346 · see citing works
Schultz, W., & Romo, R. (1990) · Journal of Neurophysiology · doi:10.1152/jn.1990.63.3.607 · Cited by 334 · see citing works
Hierarchical active inference: A theory of motivated control
Pezzulo, G., Rigoli, F., & Friston, K. J. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.01.009 · Cited by 334 · see citing works
Orbitofrontal Cortex Supports Behavior and Learning Using Inferred But Not Cached Values
Jones, J. L., Esber, G. R., McDannald, M. A., Gruber, A. J., Hernández, A., Mirenzi, A., & Schoenbaum, G. M. (2012) · Science · doi:10.1126/science.1227489 · Cited by 332 · see citing works
Dopamine Does Double Duty in Motivating Cognitive Effort
Westbrook, A., & Braver, T. S. (2016) · Neuron · doi:10.1016/j.neuron.2015.12.029 · Cited by 313 · see citing works
The temporal precision of reward prediction in dopamine neurons
Fiorillo, C. D., Newsome, W. T., & Schultz, W. (2008) · Nature Neuroscience · doi:10.1038/nn.2159 · Cited by 310 · see citing works
Botvinick, M., & Toussaint, M. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2012.08.006 · Cited by 295 · see citing works
Neural Coding of Uncertainty and Probability
Ma, W. J., & Jazayeri, M. (2014) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071013-014017 · Cited by 290 · see citing works
Dopamine enhances signal-to-noise ratio in cortical-brainstem encoding of aversive stimuli
Weele, C. M. V., Siciliano, C. A., Matthews, G. A., Namburi, P., Izadmehr, E. M., Espinel, I. C., Nieh, E. H., Schut, E. H. S., Padilla-Coreano, N., Burgos-Robles, A., et al. (2018) · Nature · doi:10.1038/s41586-018-0682-1 · Cited by 288 · see citing works
Bayesian optimization of time perception
Shi, Z., Church, R. M., & Meck, W. H. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.09.009 · Cited by 278 · see citing works
Dopamine Neurons Can Represent Context-Dependent Prediction Error
Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004) · Neuron · doi:10.1016/s0896-6273(03)00869-9 · Cited by 276 · see citing works
Dopamine Does Double Duty in Motivating Cognitive Effort.
Westbrook, A., & Braver, T. (2016) · Neuron · doi:10.1016/j.neuron.2016.07.020 · Cited by 271 · see citing works
Deconstructing the human algorithms for exploration
Gershman, S. J. (2018) · Cognition · doi:10.1016/j.cognition.2017.12.014 · Cited by 267 · see citing works
Self-citationMenegas, W., Babayan, B. M., Uchida, N., & Watabe‐Uchida, M. (2017) · eLife · doi:10.7554/elife.21886 · Cited by 265 · see citing works
Self-citationLatent inhibition of conditioned dopamine release in rat nucleus accumbens
Young, A. M. J., Joseph, M. H., & Gray, J. A. (1993) · Neuroscience · doi:10.1016/0306-4522(93)90378-s · Cited by 261 · see citing works
Neural Variability and Sampling-Based Probabilistic Representations in the Visual Cortex
Orbán, G., Berkes, P., Fiser, J., & Lengyel, M. (2016) · Neuron · doi:10.1016/j.neuron.2016.09.038 · Cited by 258 · see citing works
Network Resets in Medial Prefrontal Cortex Mark the Onset of Behavioral Uncertainty
Karlsson, M. P., Tervo, D. G. R., & Karpova, A. Y. (2012) · Science · doi:10.1126/science.1226518 · Cited by 236 · see citing works
Wiener, S. I. (1993) · Journal of Neuroscience · doi:10.1523/jneurosci.13-09-03802.1993 · Cited by 233 · see citing works
Gallistel, C. R., Mark, T. A., King, A. P., & Latham, P. E. (2001) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037//0097-7403.27.4.354 · Cited by 225 · see citing works
Neural Mechanisms of Belief Inference during Cooperative Games
Yoshida, W., Seymour, B., Friston, K. J., & Dolan, R. J. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.5895-09.2010 · Cited by 212 · see citing works
Perceptual Decision-Making as Probabilistic Inference by Neural Sampling
Haefner, R. M., Berkes, P., & Fiser, J. (2016) · Neuron · doi:10.1016/j.neuron.2016.03.020 · Cited by 207 · see citing works
Dopamine reward prediction errors reflect hidden-state inference across time
Starkweather, C. K., Babayan, B. M., Uchida, N., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4520 · Cited by 206 · see citing works
Self-citationDopaminergic control of the exploration-exploitation trade-off via the basal ganglia
Humphries, M. D. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00009 · Cited by 204 · see citing works
Striatal dynamics explain duration judgments
Gouvêa, T. S., Monteiro, T., Motiwala, A., Soares, S., Machens, C. K., & Paton, J. J. (2015) · eLife · doi:10.7554/elife.11386 · Cited by 203 · see citing works
Resolution of Uncertainty in Prefrontal Cortex
Yoshida, W., & Ishii, S. (2006) · Neuron · doi:10.1016/j.neuron.2006.05.006 · Cited by 196 · see citing works
The successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191 · see citing works
Self-citationRao, R. P. N. (2010) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2010.00146 · Cited by 190 · see citing works
Spatial, movement- and reward-sensitive discharge by medial ventral striatum neurons of rats
Lavoie, A. M., & Mizumori, S. J. Y. (1994) · Brain Research · doi:10.1016/0006-8993(94)90645-9 · Cited by 185 · see citing works
A Pallidus-Habenula-Dopamine Pathway Signals Inferred Stimulus Values
Bromberg-Martin, E. S., Matsumoto, M., Hong, S., & Hikosaka, O. (2010) · Journal of Neurophysiology · doi:10.1152/jn.00158.2010 · Cited by 182 · see citing works
Uncertainty and Exploration in a Restless Bandit Problem
Speekenbrink, M., & Konstantinidis, E. (2015) · Topics in Cognitive Science · doi:10.1111/tops.12145 · Cited by 177 · see citing works
Gallistel, C. R., Mark, T. A., King, A. P., et al. (2001) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.27.4.354 · Cited by 174 · see citing works
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · see citing works
The computational nature of memory modification
Gershman, S. J., Monfils, M.-H., Norman, K. A., & Niv, Y. (2017) · eLife · doi:10.7554/elife.23763 · Cited by 172 · see citing works
Self-citationMidbrain dopamine neurons signal belief in choice accuracy during a perceptual decision
Lak, A., Nomoto, K., Keramati, M., Sakagami, M., & Kepecs, A. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.026 · Cited by 170 · see citing works
Miller, R. R., & Matute, H. (1996) · Journal of Experimental Psychology General · doi:10.1037//0096-3445.125.4.370 · Cited by 167 · see citing works
Rethinking dopamine as generalized prediction error
Gardner, M., Schoenbaum, G. M., & Gershman, S. J. (2018) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2018.1645 · Cited by 162 · see citing works
Self-citationTemporally Extended Dopamine Responses to Perceptually Demanding Reward-Predictive Stimuli
Nomoto, K., Schultz, W., Watanabe, T., & Sakagami, M. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4828-09.2010 · Cited by 158 · see citing works
A Unifying Probabilistic View of Associative Learning
Gershman, S. J. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004567 · Cited by 158 · see citing works
Self-citationRodent Medial Frontal Control of Temporal Processing in the Dorsomedial Striatum
Emmons, E. B., Corte, B. J. D., Kim, Y.-C., Parker, K. L., Matell, M. S., & Narayanan, N. S. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.1376-17.2017 · Cited by 156 · see citing works
Kumaran, D., Banino, A., Blundell, C., Hassabis, D., & Dayan, P. (2016) · Neuron · doi:10.1016/j.neuron.2016.10.052 · Cited by 155 · see citing works
Neural mediators of changes of mind about perceptual decisions
Fleming, S. M., van der Putten, E. J., & Daw, N. D. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0104-6 · Cited by 155 · see citing works
Computational Dissection of Dopamine Motor and Motivational Functions in Humans
Bouc, R. L., Rigoux, L., Schmidt, L., Degos, B., Welter, M., Vidailhet, M., Jean Daunizeau, A., & Pessiglione, M. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.3078-15.2016 · Cited by 153 · see citing works
Discovering latent causes in reinforcement learning
Gershman, S. J., Norman, K. A., & Niv, Y. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.07.007 · Cited by 151 · see citing works
Self-citationWhat Is the Relationship between Dopamine and Effort?
Walton, M. E., & Bouret, S. (2018) · Trends in Neurosciences · doi:10.1016/j.tins.2018.10.001 · Cited by 150 · see citing works
Stimulus representation and the timing of reward-prediction errors in models of the dopamine system
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008) · Neural Computation · doi:10.1162/neco.2008.11-07-654 · Cited by 148 · see citing works
Differential Encoding of Time by Prefrontal and Striatal Network Dynamics
Bakhurin, K. I., Goudar, V., Shobe, J. L., Claar, L. D., Buonomano, D. V., & Masmanidis, S. C. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.1789-16.2016 · Cited by 146 · see citing works
Gershman, S. J. (2018) · Decision · doi:10.1037/dec0000101 · Cited by 146 · see citing works
Self-citationMidbrain dopamine neurons compute inferred and cached value prediction errors in a common framework
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. M. (2016) · eLife · doi:10.7554/elife.13665 · Cited by 137 · see citing works
The medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty
Starkweather, C. K., Gershman, S. J., & Uchida, N. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.036 · Cited by 136 · see citing works
Self-citationMiller, R. R., & Matute, H. (1996) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.125.4.370 · Cited by 129 · see citing works
Responses of striatal neurons in the behaving monkey. 2. Visual processing in the caudal neostriatum
Caan, W., Perrett, D. I., & Rolls, E. T. (1984) · Brain Research · doi:10.1016/0006-8993(84)90735-2 · Cited by 127 · see citing works
Confabulation: Damage to a specific inferior medial prefrontal system
Turner, M., Lisa Cipolotti, A., Yousry, T. A., & Shallice, T. (2007) · Cortex · doi:10.1016/j.cortex.2007.01.002 · Cited by 124 · see citing works
Recovery from blocking achieved by extinguishing the blocking CS
Blaisdell, A. P., Gunther, L. M., & Miller, R. R. (1999) · Animal Learning & Behavior · doi:10.3758/bf03199432 · Cited by 119 · see citing works
Takahashi, Y. K., Langdon, A., Niv, Y., & Schoenbaum, G. M. (2016) · Neuron · doi:10.1016/j.neuron.2016.05.015 · Cited by 116 · see citing works
Synaptic and Spiking Dynamics underlying Reward Reversal in the Orbitofrontal Cortex
Deco, G. (2004) · Cerebral Cortex · doi:10.1093/cercor/bhh103 · Cited by 114 · see citing works
Dopamine, reward learning, and active inference
FitzGerald, T. H. B., Dolan, R. J., & Friston, K. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00136 · Cited by 112 · see citing works
Akhlaghpour, H., Wiskerke, J., Choi, J. Y., Taliaferro, J. P., Au, J., & Witten, I. B. (2016) · eLife · doi:10.7554/elife.19507 · Cited by 110 · see citing works
Midbrain dopamine neurons signal aversion in a reward-context-dependent manner
Matsumoto, H., Tian, J., Uchida, N., & Watabe‐Uchida, M. (2016) · eLife · doi:10.7554/elife.17328 · Cited by 109 · see citing works
Self-citationPecevski, D., Buesing, L., & Maass, W. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002294 · Cited by 107 · see citing works
Dopamine Increases the Gain of the Input-Output Response of Rat Prefrontal Pyramidal Neurons
Thurley, K., Senn, W., & Lüscher, H. (2008) · Journal of Neurophysiology · doi:10.1152/jn.01098.2007 · Cited by 106 · see citing works
Evaluating the TD model of classical conditioning
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2012) · Learning & Behavior · doi:10.3758/s13420-012-0082-6 · Cited by 106 · see citing works
Acquisition and extinction in autoshaping.
Kakade, S. M., & Dayan, P. (2002) · Psychological Review · doi:10.1037/0033-295x.109.3.533 · Cited by 102 · see citing works
Belief state representation in the dopamine system
Babayan, B. M., Uchida, N., & Gershman, S. J. (2018) · Nature Communications · doi:10.1038/s41467-018-04397-0 · Cited by 101 · see citing works
Self-citationSadacca, B. F., Wied, H. M., Lopatina, N., Saini, G. K., Nemirovsky, D., & Schoenbaum, G. M. (2018) · eLife · doi:10.7554/elife.30373 · Cited by 95 · see citing works
Aguado, L., Symonds, M., & Hall, G. (1994) · Animal Learning & Behavior · doi:10.3758/bf03199919 · Cited by 93 · see citing works
One-trial discrimination reversal in the white rat.
Dufort, R. H., Guttman, N., & Kimble, G. A. (1954) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0057856 · Cited by 93 · see citing works
Responses of cells in the tail of the caudate nucleus during visual discrimination learning
Brown, V. J., Desimone, R., & Mishkin, M. (1995) · Journal of Neurophysiology · doi:10.1152/jn.1995.74.3.1083 · Cited by 90 · see citing works
Temporal Convergence of Dynamic Cell Assemblies in the Striato-Pallidal Network
Adler, A., Katabi, S., Finkes, I., Israel, Z. H., Prut, Y., & Bergman, H. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.4830-11.2012 · Cited by 89 · see citing works
Time representation in reinforcement learning models of the basal ganglia
Gershman, S. J., Moustafa, A. A., & Ludvig, E. A. (2014) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2013.00194 · Cited by 81 · see citing works
Self-citationJang, A. I., Costa, V. D., Rudebeck, P. H., Chudasama, Y., Murray, E. A., & Averbeck, B. B. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.1594-15.2015 · Cited by 75 · see citing works
Putting bandits into context: How function learning supports decision making.
Schulz, E., Konstantinidis, E., & Speekenbrink, M. (2017) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000463 · Cited by 74 · see citing works
Effects of range of payoffs as a variable in risk taking.
Myers, J. L., & Sadler, E. (1960) · Journal of Experimental Psychology · doi:10.1037/h0042499 · Cited by 72 · see citing works
Reward Contexts Extend Dopamine Signals to Unrewarded Stimuli
Kobayashi, S., & Schultz, W. (2014) · Current Biology · doi:10.1016/j.cub.2013.10.061 · Cited by 63 · see citing works
Explaining Away in Weight Space
Dayan, P., & Kakade, S. M. (2000) · ScholarlyCommons (University of Pennsylvania) · Cited by 61 · see citing works
A probabilistic approach to demixing odors
Grabska‐Barwińska, A., Barthelmé, S., Beck, J. M., Mainen, Z. F., Pouget, A., & Latham, P. E. (2016) · Nature Neuroscience · doi:10.1038/nn.4444 · Cited by 60 · see citing works
Integrating Models of Interval Timing and Reinforcement Learning
Petter, E. A., Gershman, S. J., & Meck, W. H. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.08.004 · Cited by 58 · see citing works
Self-citationContext Learning in the Rodent Hippocampus
Fuhs, M. C., & Touretzky, D. S. (2007) · Neural Computation · doi:10.1162/neco.2007.19.12.3173 · Cited by 57 · see citing works
Payzan-LeNestour, É., & Bossaerts, P. L. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00150 · Cited by 54 · see citing works
Dopamine, inference, and uncertainty
Gershman, S. J. (2017) · Neural Computation · doi:10.1162/neco_a_01023 · Cited by 40 · see citing works
Self-citationTing, C.-C., Yu, C.-C., Maloney, L. T., & Wu, S. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.3161-14.2015 · Cited by 37 · see citing works
Retrospective revaluation: The phenomenon and its theoretical implications
Miller, R. R., & Witnauer, J. E. (2015) · Behavioural Processes · doi:10.1016/j.beproc.2015.09.001 · Cited by 33 · see citing works
Sarno, S., de Lafuente, V., Romo, R., & Parga, N. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1712479114 · Cited by 30 · see citing works
Backward blocking in first-order conditioning.
Urushihara, K., & Miller, R. R. (2010) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/a0016773 · Cited by 21 · see citing works
Associative Learning Should Go Deep
Mondragón, E., Alonso, E., & Kokkola, N. H. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.06.001 · Cited by 21 · see citing works
Distinct Dynamics of Striatal and Prefrontal Neural Activity During Temporal Discrimination
Kim, J., Kim, D., & Jung, M. W. (2018) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2018.00034 · Cited by 17 · see citing works
Serial reversal learning as a function of the number of trials per reversal.
Pubols, B. H. (1962) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0041760 · Cited by 15 · see citing works
Backward Blocking in First-Order Conditioning
Urushihara, K., & Miller, R. R. (2007) · PsycEXTRA Dataset · doi:10.1037/e527342012-212 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Forgetting as a form of adaptive engram cell plasticity
Ryan, T. J., & Frankland, P. W. (2022) · Nature reviews. Neuroscience · doi:10.1038/s41583-021-00548-3 · Cited by 182 · see citing works
Anterior Cingulate Cortex and the Control of Dynamic Behavior in Primates
Monosov, I. E., Haber, S. N., Leuthardt, E. C., & Jezzini, A. (2020) · Current Biology · doi:10.1016/j.cub.2020.10.009 · Cited by 118 · see citing works
Emergence of belief-like representations through reinforcement learning
Hennig, J. A., Romero Pinto, S. A., Yamaguchi, T., Linderman, S. W., Uchida, N., & Gershman, S. J. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011067 · Cited by 41 · see citing works
Self-citationA possible evolutionary function of phenomenal conscious experience of pain
Kolodny, O., Moyal, R., & Edelman, S. (2021) · Neuroscience of Consciousness · doi:10.1093/nc/niab012 · Cited by 31 · see citing works
Pando‐Naude, V., Matthews, T. E., Højlund, A., Jakobsen, S., Østergaard, K., Johnsen, E., Garza‐Villarreal, E. A., Witek, M. A. G., Penhune, V., & Vuust, P. (2024) · European Journal of Neuroscience · doi:10.1111/ejn.16128 · Cited by 29 · see citing works
- Authors
- Gershman, S. J., & Uchida, N.
- Affiliations
- Harvard University (US)
- Corresponding author
- Gershman, Samuel J.
- Venue
- Nature Reviews Neuroscience
- Published
- 2019-09-30
- Volume
- 20
- Issue
- 11
- Pages
- 703-714
- ISSN
- 1471-003X (ISSN-L)
- 1471-0048
- PMC
- PMC7472313
- OpenAlex
- W2976141814
- Cited by
- 236
- References
- 118
- Citations by year
- 2026: 26 · 2025: 40 · 2024: 30 · 2023: 31 · 2022: 49 · 2021: 36 · 2020: 21 · 2019: 3
- FWCI
- 3.62
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Nature reviews. Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Neurological Disorders and Stroke (R01NS108740)
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- dopamine, reward prediction error, Bayesian reinforcement learning, uncertainty encoding, belief updating, policy uncertainty, reinforcement learning
- MeSH terms
- Animals, Basal Ganglia, Dopamine, Humans, Learning, Nerve Net, Prefrontal Cortex
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
