Offline replay supports planning in human reinforcement learning
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Making decisions in sequentially structured tasks requires integrating distally acquired information. The extensive computational cost of such integration challenges planning methods that integrate online, at decision time. Furthermore, it remains unclear whether ‘offline’ integration during replay supports planning, and if so which memories should be replayed. Inspired by machine learning, we propose that (a) offline replay of trajectories facilitates integrating representations that guide decisions, and (b) unsigned prediction errors (uncertainty) trigger such integrative replay. We designed a 2-step revaluation task for fMRI, whereby participants needed to integrate changes in rewards with past knowledge to optimally replan decisions. As predicted, we found that (a) multi-voxel pattern evidence for off-task replay predicts subsequent replanning; (b) neural sensitivity to uncertainty predicts subsequent replay and replanning; (c) off-task hippocampus and anterior cingulate activity increase when revaluation is required. These findings elucidate how the brain leverages offline mechanisms in planning and goal-directed behavior under uncertainty.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
The expected value of control: an integrative theory of anterior cingulate cortex function
Shenhav, A., Botvinick, M. M., & Cohen, J. D. (2013) · Neuron · doi:10.1016/j.neuron.2013.07.007 · Cited by 2,321
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Coordinated memory replay in the visual cortex and hippocampus during sleep
Ji, D., & Wilson, M. A. (2007) · Nature Neuroscience · doi:10.1038/nn1825 · Cited by 1,669
Eichenbaum (2004) · doi:10.1016/j.neuron.2004.08.028 · Cited by 1,464
Two cortical systems for memory-guided behaviour
Ranganath, C., & Ritchey, M. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3338 · Cited by 1,377 · cited 1×
“The hippocampus has long been identified as the locus of episodic memory (Ranganath and Ritchey, 2012), posterior medial cortex is shown to be involved in recall and integration of memories over long time-scales (Chen et al.”
BackgroundHippocampal place-cell sequences depict future paths to remembered goals
Pfeiffer, B. E., & Foster, D. J. (2013) · Nature · doi:10.1038/nature12112 · Cited by 1,216
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
Self-citationDyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777
Category-Specific Cortical Activity Precedes Retrieval During Memory Search
Polyn, S. M., Natu, V. S., Cohen, J. D., & Norman, K. A. (2005) · Science · doi:10.1126/science.1117645 · Cited by 690 · cited 4×
“Specifically, it has been shown that imagery, reinstatement, and perception of faces and scenes elicit patterns in the temporal cortical regions associated with visual categories, roughly corresponding to the fusiform gyrus for the face category and the parahippocampal gyrus for the scene category (Polyn et al., 2005;…”
BackgroundMethodInfluentialSelf-citationShared memories reveal shared structure in neural activity across individuals
Chen, J., Leong, Y. C., Honey, C. J., Yong, C. H., Norman, K. A., & Hasson, U. (2017) · Nature Neuroscience · doi:10.1038/nn.4450 · Cited by 645
Self-citationRobust group analysis using outlier inference
Woolrich, M. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.02.042 · Cited by 504 · cited 1×
“Higher-level analysis was carried out using FLAME (FMRIB’s Local Analysis of Mixed Effects) stage 1 (Woolrich et al., 2004; Woolrich, 2008; Kuhl et al., 2011).”
MethodKalisch, R., Korenfeld, E., Stephan, K. E., Weiskopf, N., Seymour, B., & Dolan, R. J. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.2021-06.2006 · Cited by 493 · cited 1×
“Hippocampal-prefrontal interactions have been shown to mediate control of memory in memory inhibition (Kalisch et al., 2006) and updating memory in”
MethodGläscher, J., Hampton, A. N., & O'Doherty, J. P. (2009) · Cerebral Cortex · doi:10.1093/cercor/bhn098 · Cited by 387 · cited 2×
“…ventromedial prefrontal cortex has been established as a region involved in memory retrieval of relevant information (Preston and Eichenbaum, 2013; Gläscher et al., 2009), value representation (Fellows and Farah, 2007), and decision making Figure 4 continued DOI:…”
BackgroundFellows, L. K., & Farah, M. J. (2007) · Cerebral Cortex · doi:10.1093/cercor/bhl176 · Cited by 337
Hippocampal place cells construct reward related sequences through unexplored space
Ólafsdóttir, H. F., Barry, C., Saleem, A. B., Hassabis, D., & Spiers, H. J. (2015) · eLife · doi:10.7554/elife.06063 · Cited by 319 · cited 1×
“Indeed, this hypothesis fits closely with a substantial rodent literature revealing that hippocampal replay of previously experienced spatial trajectories is observed during both awake behavior and sleep (Takahashi, 2015; Ólafsdóttir et al., 2015).”
Backgroundvan Kesteren, M. T. R., Beul, S. F., Takashima, A., Henson, R. N., Ruiter, D. J., & Fernández, G. (2013) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2013.05.027 · Cited by 298
Reverse replay of hippocampal place cells is uniquely modulated by changing reward
Ambrose, R. E., Pfeiffer, B. E., & Foster, D. J. (2016) · Neuron · doi:10.1016/j.neuron.2016.07.047 · Cited by 292 · cited 1×
“For instance, in more elaborate models, overall variability in outcomes can be decomposed into contributions of volatility and stochasticity, and PE can be scaled relative to these (Ambrose et al., 2016b).”
BackgroundShift from hippocampal to neocortical centered retrieval network with consolidation
Takashima, A., Nieuwenhuis, I. L. C., Jensen, O., Talamini, L. M., Rijpkema, M., & Fernández, G. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.0799-09.2009 · Cited by 276 · cited 1×
“This offline reactivation could potentially aid in memory consolidation and integration, evidenced by enhanced subsequent memory performance (Takashima et al., 2009; Staresina et al., 2013; Tambini and Davachi, 2013).”
BackgroundImagery and perception share cortical representations of content and location
Cichy, R. M., Heinzle, J., & Haynes, J.-D. (2012) · Cerebral Cortex · doi:10.1093/cercor/bhr106 · Cited by 242
Persistence of hippocampal multivoxel patterns into postencoding rest is related to memory
Tambini, A., & Davachi, L. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1308499110 · Cited by 230
Awake reactivation predicts memory in humans
Staresina, B. P., Alink, A., Kriegeskorte, N., & Henson, R. N. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1311989110 · Cited by 219 · cited 3×
“Unlike animal studies, but similar to previous human fMRI studies of offline replay (Staresina et al., 2013; Cichy et al., 2012; Gruber et al., 2016), here we leverage a controlled design together with evidence of Stage I reinstatement to investigate the role of offline replay in planning.”
BackgroundResultRetrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217 · cited 7×
“To measure revaluation behavior, we computed a replanning magnitude score for each subject and block (Gershman et al., 2014; Momennejad et al., 2017; Russek et al., 2017), defined as the pro-”
BackgroundMethodInfluentialSelf-citationPost-learning hippocampal dynamics promote preferential retention of rewarding events
Gruber, M. J., Ritchey, M., Wang, S.-F., Doss, M. K., & Ranganath, C. (2016) · Neuron · doi:10.1016/j.neuron.2016.01.017 · Cited by 212 · cited 4×
“In parallel, human fMRI research work reveals how memories are reactivated offline, such as during periods of rest that follow rewarding episodes (Gruber et al., 2016; Murty et al., 2016).”
BackgroundResultInfluentialHippocampal Replay Captures the Unique Topological Structure of a Novel Environment
Wu, X., & Foster, D. J. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.3414-13.2014 · Cited by 201
Cited by
Sequential replay of nonspatial task states in the human hippocampus
Schuck, N. W., & Niv, Y. (2019) · Science · doi:10.1126/science.aaw5181 · Cited by 304
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2020) · arXiv · doi:10.48550/arxiv.2012.13490 · Cited by 227
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224
Learning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159
Self-citationPredictive maps in rats and humans for spatial navigation
de Cothi, W., Nyberg, N., Griesbauer, E.-M., Ghanamé, C., Zisch, F., Lefort, J. M., Fletcher, L., Newton, C., Renaudineau, S., Bendor, D., et al. (2022) · Current Biology · doi:10.1016/j.cub.2022.06.090 · Cited by 109
Metadata
- Authors
- Momennejad, I., Otto, A. R., Daw, N. D., & Norman, K. A.
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2018-12-14
- Volume
- 7
- ISSN
- 2050-084X (ISSN-L)
- OpenAlex
- W2904676870
- Cited by
- 131
- References
- 25
- FWCI
- 7.05
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.