Learning offline: memory replay in biological and artificial reinforcement learning
Open accessAbstract
Learning to act in an environment to maximise rewards is among the brain's key functions. This process has often been conceptualised within the framework of reinforcement learning, which has also gained prominence in machine learning and artificial intelligence (AI) as a way to optimise decision making. A common aspect of both biological and machine reinforcement learning is the reactivation of previously experienced episodes, referred to as replay. Replay is important for memory consolidation in biological neural networks and is key to stabilising learning in deep neural networks. Here, we review recent developments concerning the functional roles of replay in the fields of neuroscience and AI. Complementary progress suggests how replay might support learning processes, including generalisation and continual learning, affording opportunities to transfer knowledge across the two fields to advance the understanding of biological and artificial learning and memory.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Non-parametric episodic memory plays a pivotal role in shaping AI systems' behavioral strategies.
“Furthermore, non-parametric episodic memory plays a pivotal role in shaping AI systems' behavioral strategies [185, 186, 187, 143, 188].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [143]
Cites
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · see citing works
Dynamics of the Hippocampal Ensemble Code for Space
Wilson, M. A., & McNaughton, B. L. (1993) · Science · doi:10.1126/science.8351520 · Cited by 2,350 · see citing works
Sleep-dependent memory consolidation
Stickgold, R. (2005) · Nature · doi:10.1038/nature04286 · Cited by 1,857 · see citing works
Selective suppression of hippocampal ripples impairs spatial memory
Girardeau, G., Benchenane, K., Wiener, S. I., Buzsáki, G., & Zugaro, M. B. (2009) · Nature Neuroscience · doi:10.1038/nn.2384 · Cited by 1,560 · see citing works
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557 · see citing works
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · see citing works
Memory of sequential experience in the hippocampus during slow wave sleep
Lee, A. K., & Wilson, M. A. (2002) · Neuron · doi:10.1016/s0896-6273(02)01096-6 · Cited by 1,296 · see citing works
Schemas and Memory Consolidation.
Tse, D., Langston, R. F., Kakeyama, M., Bethus, I., Spooner, P. A., Wood, E. R., Witter, M. P., & Morris, R. G. M. (2007) · Science · doi:10.1126/science.1135935 · Cited by 1,244 · see citing works
Replay and Time Compression of Recurring Spike Sequences in the Hippocampus
Nádasdy, Z., Hirase, H., Czurkó, A., Csicsvari, J., & Buzsáki, G. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-21-09497.1999 · Cited by 932 · see citing works
Awake replay of remote experiences in the hippocampus
Karlsson, M. P., & Frank, L. M. (2009) · Nature Neuroscience · doi:10.1038/nn.2344 · Cited by 769 · see citing works
Fast-Forward Playback of Recent Memory Sequences in Prefrontal Cortex During Sleep
Euston, D. R., Tatsuno, M., & McNaughton, B. L. (2007) · Science · doi:10.1126/science.1148979 · Cited by 586 · see citing works
Human replay spontaneously reorganizes experience
Liu, Y., Dolan, R. J., Kurth-Nelson, Z., & Behrens, T. E. J. (2019) · Cell · doi:10.1016/j.cell.2019.06.012 · Cited by 497 · see citing works
The hippocampal sharp wave-ripple in memory retrieval for immediate use and consolidation
Joo, H. R., & Frank, L. M. (2018) · Nature Reviews Neuroscience · doi:10.1038/s41583-018-0077-1 · Cited by 430 · see citing works
The Role of Hippocampal Replay in Memory and Planning
Ólafsdóttir, H. F., Bush, D., & Barry, C. (2018) · Current Biology · doi:10.1016/j.cub.2017.10.073 · Cited by 413 · see citing works
A cortical-hippocampal-cortical loop of information processing during memory consolidation
Rothschild, G., Eban, E., & Frank, L. M. (2017) · Nature Neuroscience · doi:10.1038/nn.4457 · Cited by 387 · see citing works
Pennartz, C. M. A., Lee, E., Verheul, J., Lipa, P., Barnes, C. A., & McNaughton, B. L. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.0575-04.2004 · Cited by 282 · see citing works
Post-learning hippocampal dynamics promote preferential retention of rewarding events
Gruber, M. J., Ritchey, M., Wang, S.-F., Doss, M. K., & Ranganath, C. (2016) · Neuron · doi:10.1016/j.neuron.2016.01.017 · Cited by 212 · see citing works
Shin, J. D., Tang, W., & Jadhav, S. P. (2019) · Neuron · doi:10.1016/j.neuron.2019.09.012 · Cited by 199 · see citing works
Replay of behavioral sequences in the medial prefrontal cortex during rule switching
Kaefer, K., Nardin, M., Blahna, K., & Csicsvari, J. (2020) · Neuron · doi:10.1016/j.neuron.2020.01.015 · Cited by 120 · see citing works
Hippocampal reactivation of random trajectories resembling brownian diffusion
Stella, F., Baracskay, P., O’Neill, J., & Csicsvari, J. (2019) · Neuron · doi:10.1016/j.neuron.2019.01.052 · Cited by 107 · see citing works
Post-learning hippocampal replay selectively reinforces spatial memory for highly rewarded locations
Michon, F., Sun, J.-J., Kim, C. Y., Ciliberti, D., & Kloosterman, F. (2019) · Current Biology · doi:10.1016/j.cub.2019.03.048 · Cited by 86 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Human-inspired Perspectives: A Survey on AI Long-term Memory
He, Z., Lin, W., Zheng, H., Zhang, F., Jones, M. W., Aitchison, L., Xu, X., Liu, M., Kristensson, P. O., & Shen, J. (2024) · arXiv · doi:10.48550/arxiv.2411.00489 · Cited by 3 · see citing works
“Roscow et al. [137] stored the interaction data comprising states, actions, time steps, and rewards in a memory buffer, and select samples from it to prevent catastrophic forgetting, either randomly or based on criteria such as temporal differences or error rates.”
InfluentialSelf-citation
- Authors
- Roscow, E. L., Chua, R., Costa, R. P., Jones, M. W., & Lepora, N.
- Affiliations
- Centre de Recerca Matemàtica (ES)
- Mila - Quebec Artificial Intelligence Institute (CA)
- University of Bristol (GB)
- Corresponding author
- Roscow, Emma L.
- Venue
- Trends in Neurosciences
- Publisher
- Elsevier BV
- Published
- 2021-10
- Volume
- 44
- Issue
- 10
- Pages
- 808-821
- ISSN
- 0166-2236 (ISSN-L)
- OpenAlex
- W3197039934
- Cited by
- 36
- References
- 21
- Citations by year
- 2026: 5 · 2025: 10 · 2024: 3 · 2023: 9 · 2022: 8 · 2021: 1
- FWCI
- 0.38
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Trends in Neurosciences (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Fonds de recherche du Québec
- Leverhulme Trust (RL-2016-39)
- Wellcome Trust (202810/Z/16/Z, 109070/Z/15/A)
- Fonds de recherche du Québec – Nature et technologies
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- memory replay, reinforcement learning, memory consolidation, deep neural network, generalization, continual learning, knowledge transfer, biological learning, artificial intelligence
- MeSH terms
- Machine Learning, Artificial Intelligence, Hippocampus, Humans, Reinforcement, Psychology, Reward
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
