Working Memory Load Strengthens Reward Prediction Errors
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Reinforcement learning (RL) in simple instrumental tasks is usually modeled as a monolithic process in which reward prediction errors (RPEs) are used to update expected values of choice options. This modeling ignores the different contributions of different memory and decision-making systems thought to contribute even to simple learning. In an fMRI experiment, we investigated how working memory (WM) and incremental RL processes interact to guide human learning. WM load was manipulated by varying the number of stimuli to be learned across blocks. Behavioral results and computational modeling confirmed that learning was best explained as a mixture of two mechanisms: a fast, capacity-limited, and delay-sensitive WM process together with slower RL. Model-based analysis of fMRI data showed that striatum and lateral prefrontal cortex were sensitive to RPE, as shown previously, but, critically, these signals were reduced when the learning problem was within capacity of WM. The degree of this neural interaction related to individual differences in the use of WM to guide behavioral learning. These results indicate that the two systems do not process information independently, but rather interact during learning. SIGNIFICANCE STATEMENT Reinforcement learning (RL) theory has been remarkably productive at improving our understanding of instrumental learning as well as dopaminergic and striatal network function across many mammalian species. However, this neural network is only one contributor to human learning and other mechanisms such as prefrontal cortex working memory also play a key role. Our results also show that these other players interact with the dopaminergic RL system, interfering with its key computation of reward prediction errors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The organization of the human cerebral cortex estimated by intrinsic functional connectivity
Thomas Yeo, B. T., Krienen, F. M., Sepulcre, J., Sabuncu, M. R., Lashkari, D., Hollinshead, M., Roffman, J. L., Smoller, J. W., Zöllei, L., Polimeni, J. R., et al. (2011) · Journal of Neurophysiology · doi:10.1152/jn.00338.2011 · Cited by 9,182
The organization of the human cerebellum estimated by intrinsic functional connectivity
Buckner, R. L., Krienen, F. M., Castellanos, A., Diaz, J. C., & Yeo, B. T. T. (2011) · Journal of Neurophysiology · doi:10.1152/jn.00339.2011 · Cited by 7,304
Cluster failure: why fMRI inferences for spatial extent have inflated false-positive rates
Eklund, A., Nichols, T. E., & Knutsson, H. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1602413113 · Cited by 3,487 · cited 1×
“Note that these appropriately high-cluster-forming threshold ensures that parametric assumptions are valid and the rate of false positives are appropriate (Eklund et al., 2016; Flandin and Friston, 2016).”
BackgroundGetting formal with dopamine and reward
Schultz, W. (2002) · Neuron · doi:10.1016/s0896-6273(02)00967-4 · Cited by 2,574
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Self-citationDopamine neuron systems in the brain: An update
Björklund, A., & Dunnett, S. B. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.03.006 · Cited by 1,777 · cited 2×
“, Badre and Frank, 2011), possibly reflecting a common dopaminergic input to both regions (Björklund and Dunnett, 2007).”
BackgroundResultComputational psychiatry as a bridge from neuroscience to clinical applications
Huys, Q. J. M., Maia, T. V., & Frank, M. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4238 · Cited by 1,002
Self-citationBroad domain generality in focal regions of frontal and parietal cortex
Fedorenko, E., Duncan, J., & Kanwisher, N. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1315235110 · Cited by 952 · cited 1×
“To confirm the robustness of our findings, we then ran a larger frontoparietal network ROI defined from a functionally neutral group (Yeo et al., 2011), along with a functionally defined ROI of the multiple demands network from Fedorenko et al. (2013).”
MethodReasoning, Learning, and Creativity: Frontal Lobe Function and Human Decision-Making
Collins, A., & Koechlin, E. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001293 · Cited by 558 · cited 2×
“…executive processes (such as those involved in representing sequential or hierarchical task structure) contribute substantially to human learning over and above incremental RL (Botvinick et al., 2009; Badre and Frank, 2011; Daw et al., 2011; Collins and Koechlin, 2012; Collins and Frank, 2013).”
BackgroundSelf-citation
Cited by
Uncovering the structure of self-regulation through data-driven ontology discovery
Eisenberg, I. W., Bissett, P. G., Zeynep Enkavi, A., Li, J., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A. (2019) · Nature Communications · doi:10.1038/s41467-019-10301-1 · Cited by 474
The Tortoise and the Hare: Interactions between Reinforcement Learning and Working Memory
Collins, A. G. E. (2018) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01238 · Cited by 97
Self-citationNeural activations associated with feedback and retrieval success.
Wiklund-Hörnqvist, C., Andersson, M., Jonsson, B., & Nyberg, L. (2017) · npj Science of Learning · doi:10.1038/s41539-017-0013-6 · Cited by 10
“In line with prior research, we expected higher striatal activity for successfully retrieved items compared to failures and reduced frontostriatal activity as a function of consecutive successful retrievals, likely reflecting less need for executive control functions.(24,26,27)”
Action chunking as conditional policy compression
Lai, L., Huang, A. Z., & Gershman, S. J. (2022) · doi:10.31234/osf.io/z8yrv · Cited by 9
“To test whether action chunking is sensitive to memory load, we also manipulated the number of states, following previous work by Collins and colleagues showing “set size” effects in RL tasks (Collins and Frank, 2012; Collins et al., 2017; Collins, 2018; Collins and Frank, 2018).”
Influential(2025) · doi:10.1371/journal.pcbi.1012872 · Cited by 3
Self-citation
Metadata
- Authors
- Collins, A. G. E., Ciullo, B., Frank, M. J., & Badre, D.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2017-03-20
- Volume
- 37
- Issue
- 16
- Pages
- 4332-4342
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W2597822319
- Cited by
- 129
- References
- 13
- FWCI
- 6.16
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Working Memory Load Strengthens Reward Prediction Errors. https://app.cochat.ai/papers/10.1523/jneurosci.2700-16.2017
- The paper
- Collins, A. G. E., Ciullo, B., Frank, M. J., & Badre, D. (2017). Working Memory Load Strengthens Reward Prediction Errors. The Journal of Neuroscience. https://doi.org/10.1523/jneurosci.2700-16.2017
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.