Human reinforcement learning subdivides structured action spaces by learning effector-specific values
Open accessAbstract
Humans and animals are endowed with a large number of effectors. Although this enables great behavioral flexibility, it presents an equally formidable reinforcement learning problem of discovering which actions are most valuable because of the high dimensionality of the action space. An unresolved question is how neural systems for reinforcement learning-such as prediction error signals for action valuation associated with dopamine and the striatum-can cope with this "curse of dimensionality." We propose a reinforcement learning framework that allows for learned action valuations to be decomposed into effector-specific components when appropriate to a task, and test it by studying to what extent human behavior and blood oxygen level-dependent (BOLD) activity can exploit such a decomposition in a multieffector choice task. Subjects made simultaneous decisions with their left and right hands and received separate reward feedback for each hand movement. We found that choice behavior was better described by a learning model that decomposed the values of bimanual movements into separate values for each effector, rather than a traditional model that treated the bimanual actions as unitary with a single value. A decomposition of value into effector-specific components was also observed in value-related BOLD signaling, in the form of lateralized biases in striatal correlates of prediction error and anticipatory value correlates in the intraparietal sulcus. These results suggest that the human brain can use decomposed value representations to "divide and conquer" reinforcement learning over high-dimensional action spaces.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A single set of parameters imposes a simple, but efficient regularization, which stabilizes the estimated model parameters.
“A single set of parameters as frequently employed in our recent work (Daw et al., 2006; Gershman et al., 2009; Glascher et al., 2009) imposes a simple, but efficient regularization, which stabilizes the estimated model parameters.”
From States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning (2010) · cites this paper as Gershman et al., 2009
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033 · see citing works
Brainard, D. H. (1997) · Spatial Vision · doi:10.1163/156856897x00357 · Cited by 19,308 · see citing works
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Predictive reward signal of dopamine neurons
Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.1.1 · Cited by 4,564 · see citing works
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696 · see citing works
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.2307/2291091 · Cited by 2,359 · see citing works
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224 · see citing works
Self-citationBarto, A. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,172 · see citing works
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053 · see citing works
Neural correlates of decision variables in parietal cortex
Platt, M. L., & Glimcher, P. W. (1999) · Nature · doi:10.1038/22268 · Cited by 1,721 · see citing works
Haber, S. N., Fudge, J. L., & McFarland, N. R. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-06-02369.2000 · Cited by 1,610 · see citing works
Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans
Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006) · Nature · doi:10.1038/nature05051 · Cited by 1,584 · see citing works
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · see citing works
Experience-weighted Attraction Learning in Normal Form Games
Camerer, C. F., & Ho, T. (1999) · Econometrica · doi:10.1111/1468-0262.00054 · Cited by 1,442 · see citing works
Reward representations and reward-related learning in the human brain: Insights from neuroimaging
O’Doherty, J. P. (2004) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2004.10.016 · Cited by 1,426 · see citing works
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · see citing works
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190 · see citing works
Coding of intention in the posterior parietal cortex
Snyder, L. H., Batista, A. P., & Andersen, R. A. (1997) · Nature · doi:10.1038/386167a0 · Cited by 1,103 · see citing works
O’Reilly, R. C., & Frank, M. J. (2005) · Neural Computation · doi:10.1162/089976606775093909 · Cited by 1,066 · see citing works
Matching Behavior and the Representation of Value in the Parietal Cortex
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004) · Science · doi:10.1126/science.1094765 · Cited by 1,030 · see citing works
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939 · see citing works
Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum
McClure, S. M., Berns, G. S., & Montague, P. R. (2003) · Neuron · doi:10.1016/s0896-6273(03)00154-5 · Cited by 917 · see citing works
Matsuda, W., Furuta, T., Nakamura, K. C., Hioki, H., Fujiyama, F., Arai, R., & Kaneko, T. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.4029-08.2009 · Cited by 865 · see citing works
Orbitofrontal Cortex Encodes Willingness to Pay in Everyday Economic Transactions
Plassmann, H., O'Doherty, J., & Rangel, A. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2131-07.2007 · Cited by 855 · see citing works
Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli
Brischoux, F., Chakraborty, S., Brierley, D. I., & Ungless, M. A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0811507106 · Cited by 801 · see citing works
Hare, T. A., O'Doherty, J., Camerer, C. F., Schultz, W., & Rangel, A. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1309-08.2008 · Cited by 792 · see citing works
REVIEW: The functional organization of the intraparietal sulcus in humans and monkeys
Grefkes, C., & Fink, G. R. (2005) · Journal of Anatomy · doi:10.1111/j.1469-7580.2005.00426.x · Cited by 753 · see citing works
Reward‐Related Responses in the Human Striatum
DELGADO, M. R. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.002 · Cited by 736 · see citing works
Reach Plans in Eye-Centered Coordinates
Batista, A. P., Buneo, C. A., Snyder, L. H., & Andersen, R. A. (1999) · Science · doi:10.1126/science.285.5425.257 · Cited by 707 · see citing works
A common reference frame for movement plans in the posterior parietal cortex
Cohen, Y. E., & Andersen, R. A. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn873 · Cited by 695 · see citing works
Reinforcement learning: The Good, The Bad and The Ugly
Dayan, P., & Niv, Y. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.08.003 · Cited by 690 · see citing works
Boorman, E. D., Behrens, T. E. J., Woolrich, M. W., & Rushworth, M. F. S. (2009) · Neuron · doi:10.1016/j.neuron.2009.05.014 · Cited by 665 · see citing works
Cognitive Control Signals for Neural Prosthetics
Musallam, S., Corneil, B. D., Greger, B., Scherberger, H., & Andersen, R. A. (2004) · Science · doi:10.1126/science.1097938 · Cited by 663 · see citing works
The discovery of structural form
Kemp, C., & Tenenbaum, J. B. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0802631105 · Cited by 656 · see citing works
Astafiev, S. V., Shulman, G. L., Stanley, C. M., Snyder, A. Z., Van Essen, D. C., & Corbetta, M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-11-04689.2003 · Cited by 640 · see citing works
Structure and strength in causal induction☆
Griffiths, T. L., & Tenenbaum, J. B. (2005) · Cognitive Psychology · doi:10.1016/j.cogpsych.2005.05.004 · Cited by 618 · see citing works
Roesch, M. R., Calu, D. J., & Schoenbaum, G. (2007) · Nature Neuroscience · doi:10.1038/nn2013 · Cited by 597 · see citing works
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498 · see citing works
DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS
Lau, B., & Glimcher, P. W. (2005) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.2005.110-04 · Cited by 485 · see citing works
Free choice activates a decision circuit between frontal and parietal cortex
Pesaran, B., Nelson, M., & Andersen, R. (2008) · Nature · doi:10.1038/nature06849 · Cited by 479 · see citing works
Self-citationMultiple model-based reinforcement learning
Doya, K., Samejima, K., Katagiri, K.-I., & Kawato, M. (2002) · Neural Computation · doi:10.1162/089976602753712972 · Cited by 476 · see citing works
The computational neurobiology of learning and reward
Daw, N. D., & Doya, K. (2006) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2006.03.006 · Cited by 462 · see citing works
Self-citationBayesian theories of conditioning in a changing world
Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.004 · Cited by 457 · see citing works
Self-citationActivity in human ventral striatum locked to errors of reward prediction
Pagnoni, G., Zink, C. F., Montague, P. R., & Berns, G. S. (2002) · Nature Neuroscience · doi:10.1038/nn802 · Cited by 451 · see citing works
Value Representations in the Primate Striatum during Matching Behavior
Lau, B., & Glimcher, P. W. (2008) · Neuron · doi:10.1016/j.neuron.2008.02.021 · Cited by 411 · see citing works
FMRI evidence for a 'parietal reach region' in the human brain
Connolly, J. D., Andersen, R. A., & Goodale, M. A. (2003) · Experimental Brain Research · doi:10.1007/s00221-003-1587-1 · Cited by 405 · see citing works
Schönberg, T., Daw, N. D., Joel, D., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2496-07.2007 · Cited by 390 · see citing works
Self-citationGläscher, J., Hampton, A. N., & O'Doherty, J. P. (2009) · Cerebral Cortex · doi:10.1093/cercor/bhn098 · Cited by 387 · see citing works
Pesaran, B., Nelson, M. J., & Andersen, R. A. (2006) · Neuron · doi:10.1016/j.neuron.2006.05.025 · Cited by 371 · see citing works
Self-citationLinking nucleus accumbens dopamine and blood oxygenation
Knutson, B., & Gibbs, S. E. B. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0686-7 · Cited by 337 · see citing works
Human medial intraparietal cortex subserves visuomotor coordinate transformation
Grefkes, C., Ritzl, A., Zilles, K., & Fink, G. R. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.08.031 · Cited by 269 · see citing works
Functional Magnetic Resonance Imaging of Macaque Monkeys Performing Visually Guided Saccade Tasks
Koyama, M., Hasegawa, I., Osada, T., Adachi, Y., Nakahara, K., & Miyashita, Y. (2004) · Neuron · doi:10.1016/s0896-6273(04)00047-9 · Cited by 268 · see citing works
Parietal neurons related to memory-guided hand manipulation
Murata, A., Gallese, V., Kaseda, M., & Sakata, H. (1996) · Journal of Neurophysiology · doi:10.1152/jn.1996.75.5.2180 · Cited by 247 · see citing works
Modular decomposition in visuomotor learning
Zoubin Ghahramani, A., & Wolpert, D. M. (1997) · Nature · doi:10.1038/386392a0 · Cited by 244 · see citing works
Posterior Parietal Cortex Encodes Autonomously Selected Motor Plans
Cui, H., & Andersen, R. A. (2007) · Neuron · doi:10.1016/j.neuron.2007.09.031 · Cited by 237 · see citing works
Coe, B. C., Tomihara, K., Matsuzawa, M., & Hikosaka, O. (2002) · Journal of Neuroscience · doi:10.1523/jneurosci.22-12-05081.2002 · Cited by 215 · see citing works
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · see citing works
Self-citationQ-decomposition for reinforcement learning agents
Russell, S., & Zimdars, A. L. (2003) · International Conference on Machine Learning · Cited by 140 · see citing works
Behavioral and Neural Changes after Gains and Losses of Conditioned Reinforcers
Seo, H., & Lee, D. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.4726-08.2009 · Cited by 137 · see citing works
All Learning is Local: Multi-agent learning in global reward games
Chang, Y.-H., Ho, T., & Kaelbling, L. P. (2003) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 120 · see citing works
Mechanisms of selection and guidance of reaching movements in the parietal lobe.
Kalaska, J. F., Cisek, P., & Gosselin-Kessiby, N. (2003) · PubMed · Cited by 30 · see citing works
Multiple model-based reinforcement learning explains dopamine neuronal activity
Bertin, M., Schweighofer, N., & Doya, K. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.028 · Cited by 20 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779 · see citing works
Self-citationGläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166 · see citing works
Self-citationMechanisms underlying cortical activity during value-guided choice
Hunt, L. T., Kolling, N., Soltani, A., Woolrich, M. W., Rushworth, M. F. S., & Behrens, T. E. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3017 · Cited by 454 · see citing works
Learning latent structure: carving nature at its joints
Gershman, S. J., & Niv, Y. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.02.008 · Cited by 326 · see citing works
Self-citationValuation and decision-making in frontal cortex: one or many serial or parallel systems?
Rushworth, M. F., Kolling, N., Sallet, J., & Mars, R. B. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.04.011 · Cited by 309 · see citing works
Neural mechanisms underlying motivation of mental versus physical effort
Schmidt, L., Lebreton, M., Cléry-Melin, M.-L., Daunizeau, J., & Pessiglione, M. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001266 · Cited by 310 · see citing works
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
A mechanism for value-guided choice based on the excitation-inhibition balance in prefrontal cortex
Jocham, G., Hunt, L. T., Near, J., & Behrens, T. E. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3140 · Cited by 183 · see citing works
Skvortsova, V., Palminteri, S., & Pessiglione, M. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.1350-14.2014 · Cited by 177 · see citing works
Assessing inter individual differences with task related functional neuroimaging
Lebreton, M., Bavard, S., Daunizeau, J., & Palminteri, S. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0681-8 · Cited by 160 · see citing works
Generalization of value in reinforcement learning by humans
Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08017.x · Cited by 111 · see citing works
Self-citationNeural integration of risk and effort costs by the frontal pole: only upon request
Burke, C. J., Brünger, C., Kahnt, T., Park, S. Q., & Tobler, P. N. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.3662-12.2013 · Cited by 87 · see citing works
Dissociable reward and timing signals in human midbrain and ventral striatum
Klein-Flügge, M. C., Hunt, L. T., Bach, D. R., Dolan, R. J., & Behrens, T. E. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.08.024 · Cited by 75 · see citing works
Action-specific value signals in reward-related regions of the human brain
FitzGerald, T. H. B., Friston, K. J., & Dolan, R. J. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.3254-12.2012 · Cited by 68 · see citing works
Learning moral values: Another’s desire to punish enhances one’s own punitive behavior.
FeldmanHall, O., Otto, A. R., & Phelps, E. A. (2018) · Journal of Experimental Psychology: General · doi:10.1037/xge0000405 · Cited by 51 · see citing works
Colas, J. T., O’Doherty, J. P., & Grafton, S. T. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011950 · Cited by 6 · see citing works
- Authors
- Gershman, S. J., Pesaran, B., & Daw, N. D.
- Affiliations
- New York University (US)
- Princeton University (US)
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2009-10-28
- Volume
- 29
- Issue
- 43
- Pages
- 13524-13531
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- PMC
- PMC2796632
- OpenAlex
- W2011214729
- Cited by
- 118
- References
- 63
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 5 · 2023: 4 · 2022: 7 · 2021: 9 · 2020: 6 · 2019: 7 · 2018: 11 · 2017: 5 · 2016: 9 · 2015: 7 · 2014: 10 · 2013: 12 · 2012: 12
- FWCI
- 3.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Neuroscience (publishedVersion, cc-by-nc-sa)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- McKnight Endowment Fund for Neuroscience
- Alfred P. Sloan Foundation
- McKnight Foundation
- Wellcome Trust
- National Institute of Mental Health (R01MH087882)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- human reinforcement learning, high-dimensional action spaces, bimanual movement, prediction error, dopamine, striatum, intraparietal sulcus, laterality, BOLD activation, choice behavior
- MeSH terms
- Algorithms, Analysis of Variance, Brain, Brain Mapping, Cerebrovascular Circulation, Choice Behavior, Hand, Humans, Functional Laterality, Learning, Magnetic Resonance Imaging, Models, Psychological, Motor Activity, Neuropsychological Tests, Oxygen, Psychomotor Performance, Reinforcement, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
