Offline Replay Supports Planning: fMRI Evidence from Reward Revaluation
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Making decisions in sequentially structured tasks requires integrating distally acquired information. The extensive computational cost of such integration challenges planning methods that integrate online, at decision time. Furthermore, it remains unclear whether “offline” integration during replay supports planning, and if so which memories should be replayed. Inspired by machine learning, we propose that (a) offline replay of trajectories facilitates integrating representations that guide decisions, and (b) unsigned prediction errors (uncertainty) trigger such integrative replay. We designed a 2-step revaluation task for fMRI, whereby participants needed to integrate changes in rewards with past knowledge to optimally replan decisions. As predicted, we found that (a) multi-voxel pattern evidence for off-task replay predicts subsequent replanning; (b) neural sensitivity to uncertainty predicts subsequent replay and replanning; (c) off-task hippocampus and anterior cingulate activity increase when revaluation is required. These findings elucidate how the brain leverages offline mechanisms in planning and goal-directed behavior under uncertainty.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 3×
“In the formalism of reinforcement learning in artificial intelligence, the Dyna family of algorithms implements offline planning, which is accomplished by learning from replay of experiences that were initially encoded during online behavior (8) (6).”
BackgroundMethodResultRobust group analysis using outlier inference
Woolrich, M. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.02.042 · Cited by 504
Kalisch, R., Korenfeld, E., Stephan, K. E., Weiskopf, N., Seymour, B., & Dolan, R. J. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.2021-06.2006 · Cited by 493
Post-learning hippocampal dynamics promote preferential retention of rewarding events
Gruber, M. J., Ritchey, M., Wang, S.-F., Doss, M. K., & Ranganath, C. (2016) · Neuron · doi:10.1016/j.neuron.2016.01.017 · Cited by 212
Cited by
What Is a Cognitive Map? Organizing Knowledge for Flexible Behavior
Behrens, T. E. J., Muller, T. H., Whittington, J. C. R., Mark, S., Baram, A. B., Stachenfeld, K. L., & Kurth-Nelson, Z. (2018) · Neuron · doi:10.1016/j.neuron.2018.10.002 · Cited by 1,055
Prioritized memory access explains planning and hippocampal replay
Mattar, M. G., & Daw, N. D. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0232-z · Cited by 393
“We note that the same framework can be readily extended to the function approximation case and to policy gradient methods—that is, to computing the utility of a policy change even when the policies differ in multiple states (by using equation (11)).”
Self-citationReward revaluation biases hippocampal replay content away from the preferred outcome
Carey, A. A., Tanaka, Y., & van der Meer, M. A. A. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0464-6 · Cited by 106
A probabilistic successor representation for context-dependent learning
Geerts, J. P., Gershman, S. J., Burgess, N., & Stachenfeld, K. L. (2024) · Psychological Review · doi:10.1037/rev0000414 · Cited by 14
Metadata
- Authors
- Momennejad, I., Ross Otto, A., Daw, N. D., & Norman, K. A.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2017-10-02
- OpenAlex
- W2766795651
- Cited by
- 8
- References
- 5
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.