Dynamic Interaction between Reinforcement Learning and Attention in Multidimensional Environments
Open accessAbstract
Little is known about the relationship between attention and learning during decision making. Using eye tracking and multivariate pattern analysis of fMRI data, we measured participants' dimensional attention as they performed a trial-and-error learning task in which only one of three stimulus dimensions was relevant for reward at any given time. Analysis of participants' choices revealed that attention biased both value computation during choice and value update during learning. Value signals in the ventromedial prefrontal cortex and prediction errors in the striatum were similarly biased by attention. In turn, participants' focus of attention was dynamically modulated by ongoing learning. Attentional switches across dimensions correlated with activity in a frontoparietal attention network, which showed enhanced connectivity with the ventromedial prefrontal cortex between switches. Our results suggest a bidirectional interaction between attention and learning: attention constrains learning to relevant dimensions of the environment, while we learn what to attend to via trial and error.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Attention influences the choices we make as well as our learning of reward associations over time.
“and that attention influences the choices we make as well as our learning of reward associations over time54.”
From Paying attention to attention in depression (2019) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033 · see citing works
Brainard, D. H. (1997) · Spatial Vision · doi:10.1163/156856897x00357 · Cited by 19,308 · see citing works
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
Control of goal-directed and stimulus-driven attention in the brain
Corbetta, M., & Shulman, G. L. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn755 · Cited by 12,881 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Neural Mechanisms of Selective Visual Attention
Desimone, R., & Duncan, J. (1995) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.18.030195.001205 · Cited by 8,388 · see citing works
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 7,268 · see citing works
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855 · see citing works
Classical conditioning II: Current research and theory.
Black, A. H., & Prokasy, W. F. (1972) · Appleton-Century-Crofts eBooks · Cited by 4,449 · see citing works
The attention system of the human brain: 20 years after
Petersen, S. E., & Posner, M. I. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150525 · Cited by 3,439 · see citing works
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · see citing works
A theory of attention: Variations in the associability of stimuli with reinforcement
Mackintosh, N. J. (1975) · Psychological Review · doi:10.1037/h0076778 · Cited by 2,733 · see citing works
Beyond mind-reading: multi-voxel pattern analysis of fMRI data
Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.07.005 · Cited by 2,271 · see citing works
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · see citing works
A Common Network of Functional Areas for Attention and Eye Movements
Corbetta, M., Akbudak, E., Conturo, T. E., Snyder, A. Z., Ollinger, J. M., Drury, H. A., Linenweber, M. R., Petersen, S. E., Raichle, M. E., Essen, D. C. V., et al. (1998) · Neuron · doi:10.1016/s0896-6273(00)80593-0 · Cited by 1,590 · see citing works
Dissociation in prefrontal cortex of affective and attentional shifts
Dias, R., Robbins, T. W., & Roberts, A. C. (1996) · Nature · doi:10.1038/380069a0 · Cited by 1,527 · see citing works
Visual fixations and the computation and comparison of value in simple choice
Krajbich, I., Armel, C., & Rangel, A. (2010) · Nature Neuroscience · doi:10.1038/nn.2635 · Cited by 1,408 · see citing works
The role of attention in the programming of saccades
Kowler, E., Anderson, E., Dosher, B., & Blaser, E. (1995) · Vision Research · doi:10.1016/0042-6989(94)00279-u · Cited by 1,302 · see citing works
Orbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885 · see citing works
Self-citationA causal link between prediction errors, dopamine neurons and learning
Steinberg, E. E., Keiflin, R., Boivin, J. R., Witten, I. B., Deisseroth, K. A., & Janak, P. H. (2013) · Nature Neuroscience · doi:10.1038/nn.3413 · Cited by 879 · see citing works
fMRI evidence for objects as the units of attentional selection
O'Craven, K. M., Downing, P. E., & Kanwisher, N. (1999) · Nature · doi:10.1038/44134 · Cited by 846 · see citing works
Reinforcement learning: The Good, The Bad and The Ugly
Dayan, P., & Niv, Y. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.08.003 · Cited by 690 · see citing works
Self-citationControl of eye movements and spatial attention
Moore, T., & Fallah, M. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.98.3.1273 · Cited by 649 · see citing works
Temporal difference models describe higher-order learning in humans
Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S. (2004) · Nature · doi:10.1038/nature02581 · Cited by 605 · see citing works
Hare, T. A., Malmaud, J., & Rangel, A. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.6383-10.2011 · Cited by 556 · see citing works
Neural Basis of Reinforcement Learning and Decision Making
Lee, D., Seo, H., & Jung, M. W. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150512 · Cited by 542 · see citing works
Learning and selective attention
Dayan, P., Kakade, S., & Montague, P. R. (2000) · Nature Neuroscience · doi:10.1038/81504 · Cited by 495 · see citing works
PyMVPA: a Python Toolbox for Multivariate Pattern Analysis of fMRI Data
Hanke, M., Halchenko, Y. O., Sederberg, P. B., Hanson, S. J., Haxby, J. V., & Pollmann, S. (2009) · Neuroinformatics · doi:10.1007/s12021-008-9041-y · Cited by 468 · see citing works
Proceedings of the 32nd Annual Conference of the Cognitive Science Society
Rohde, H., & Ettlinger, M. (2010) · Cited by 462 · see citing works
Reinforcement Learning in Multidimensional Environments Relies on Attention Mechanisms
Niv, Y., Daniel, R., Geana, A., Gershman, S. J., Leong, Y. C., Radulescu, A., & Wilson, R. C. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.2978-14.2015 · Cited by 362 · see citing works
Self-citationLim, S.-L., O'Doherty, J. P., & Rangel, A. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.1246-11.2011 · Cited by 329 · see citing works
Learning latent structure: carving nature at its joints
Gershman, S. J., & Niv, Y. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.02.008 · Cited by 326 · see citing works
Self-citationAttention, learning, and the value of information
Gottlieb, J. (2012) · Neuron · doi:10.1016/j.neuron.2012.09.034 · Cited by 241 · cited 2× · see citing works
“These findings are consistent with the view of attention as an informationseekingmechanism that selects information that best informs behavior (Gottlieb, 2012).”
BackgroundHierarchical reinforcement learning and decision making
Botvinick, M. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.05.008 · Cited by 230 · see citing works
Two Mechanisms for Task Switching in the Prefrontal Cortex
Hyafil, A., Summerfield, C., & Koechlin, É. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.2828-08.2009 · Cited by 180 · see citing works
Inferring Relevance in a Changing World
Wilson, R. C., & Niv, Y. (2012) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2011.00189 · Cited by 150 · see citing works
Self-citationExogenous Orienting of Attention Depends upon the Ability to Execute Eye Movements
Smith, D. T., Rorden, C., & Jackson, S. R. (2004) · Current Biology · doi:10.1016/j.cub.2004.04.035 · Cited by 108 · see citing works
Holland, P. C., & Gallagher, M. (2006) · Journal of Neuroscience · doi:10.1523/jneurosci.0390-06.2006 · Cited by 85 · see citing works
Esber, G. R., Roesch, M. R., Bali, S., Trageser, J. C., Bissonette, G. B., Puché, A. C., Holland, P. C., & Schoenbaum, G. M. (2012) · Biological Psychiatry · doi:10.1016/j.biopsych.2012.05.023 · Cited by 53 · see citing works
Abstraction and Generalization in Reinforcement Learning: A Summary and Framework
Ponsen, M., Taylor, M. E., & Tuyls, K. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-11814-2_1 · Cited by 50 · see citing works
The human prefrontal cortex mediates integration of potential causes behind observed outcomes
Wunderlich, K., Beierholm, U., Bossaerts, P. L., & O’Doherty, J. P. (2011) · Journal of Neurophysiology · doi:10.1152/jn.01051.2010 · Cited by 37 · see citing works
O’Doherty, J. P. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08074.x · Cited by 36 · see citing works
Integrating reinforcement learning with models of representation learning
Jones, M., & Canas, F. (2010) · eScholarship (California Digital Library) · Cited by 27 · see citing works
Taylor, M. E., & Tuyls, K. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-11814-2 · Cited by 19 · see citing works
Modeling the Evolution of Beliefs Using an Attentional Focus Mechanism
Marković, D., Gläscher, J. P., Bossaerts, P. L., O’Doherty, J. P., & Kiebel, S. J. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004558 · Cited by 17 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Pearce JM Mackintosh NJ 2010 Two theories of attention: a review and possible integration Attention and Associative Learning Mitchell CJ LePelley ME Oxford University Press 11 14
Cited by
Sorted by citations
Towards a neuroscience of active sampling and curiosity
Gottlieb, J., & Oudeyer, P.-Y. (2018) · Nature Reviews Neuroscience · doi:10.1038/s41583-018-0078-0 · Cited by 494 · see citing works
Learning task-state representations
Niv, Y. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0470-8 · Cited by 356 · see citing works
Self-citationPaying attention to attention in depression
Keller, A. S., Leikauf, J. E., Holt-Gosselin, B., Staveland, B. R., & Williams, L. M. (2019) · Translational Psychiatry · doi:10.1038/s41398-019-0616-1 · Cited by 329 · see citing works
Medial and orbital frontal cortex in decision-making and flexible behavior
Klein-Flügge, M. C., Bongioanni, A., & Rushworth, M. F. S. (2022) · Neuron · doi:10.1016/j.neuron.2022.05.022 · Cited by 244 · see citing works
Fouragnan, E., Retzler, C., & Philiastides, M. G. (2018) · Human Brain Mapping · doi:10.1002/hbm.24047 · Cited by 176 · see citing works
Model-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161 · see citing works
Self-citationTrudel, N., Scholl, J., Klein-Flügge, M. C., Fouragnan, E., Tankelevitch, L., Wittmann, M. K., & Rushworth, M. F. S. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0929-3 · Cited by 74 · see citing works
Radulescu, A., Shin, Y. S., & Niv, Y. (2021) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-092920-120559 · Cited by 63 · see citing works
Self-citationNarcissism and risky decisions: a neurophysiological approach
Yang, Z., Sedikides, C., Gu, R., Luo, Y. L. L., Wang, Y., & Cai, H. (2018) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsy053 · Cited by 35 · see citing works
Modeling subjective relevance in schizophrenia and its relation to aberrant salience
Katthagen, T., Mathys, C., Deserno, L., Walter, H., Kathmann, N., Heinz, A., & Schlagenhauf, F. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006319 · Cited by 29 · see citing works
- Authors
- Leong, Y. C., Radulescu, A., Daniel, R., DeWoskin, V., & Niv, Y.
- Affiliations
- Stanford University (US)
- Princeton University (US)
- Corresponding author
- Leong, Yuan Chang
- Radulescu, Angela
- Niv, Yael
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2017-01
- Volume
- 93
- Issue
- 2
- Pages
- 451-463
- ISSN
- 0896-6273 (ISSN-L)
- PMC
- PMC5287409
- OpenAlex
- W2577645110
- Cited by
- 294
- References
- 45
- Citations by year
- 2026: 13 · 2025: 28 · 2024: 30 · 2023: 26 · 2022: 39 · 2021: 43 · 2020: 47 · 2019: 27 · 2018: 22 · 2017: 18
- FWCI
- 17.68
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Neuron (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Human Frontier Science Program Organization (R01MH098861)
- National Institute for Mental Health (W911NF-14-1-0101)
- Army Research Office
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- trial-and-error learning, attentional bias, value computation, prediction error, ventromedial prefrontal cortex, striatum, frontoparietal attention network, attentional shift, reward learning, eye tracking, functional MRI
- MeSH terms
- Adolescent, Adult, Attention, Brain, Brain Mapping, Choice Behavior, Decision Making, Environment, Female, Humans, Learning, Magnetic Resonance Imaging, Male, Reinforcement, Psychology, Reward, Task Performance and Analysis, Neostriatum, Prefrontal Cortex, Eye Movement Measurements, Young Adult, Functional Neuroimaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
