Model-Free Episodic Control
Open accessAbstract
State of the art deep reinforcement learning algorithms take many millions of interactions to attain human-level performance. Humans, on the other hand, can very quickly exploit highly rewarding nuances of an environment upon first discovery. In the brain, such rapid learning is thought to depend on the hippocampus and its capacity for episodic memory. Here we investigate whether a simple model of hippocampal episodic control can learn to solve difficult sequential decision-making tasks. We demonstrate that it not only attains a highly rewarding strategy significantly faster than state-of-the-art deep reinforcement learning algorithms, but also achieves a higher overall reward on some of the more challenging domains.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Remember paradigm leverages a structured replay buffer of the agent's experiences that guides the agent to more promising avenues.
“It leverages a structured replay buffer of the agent's experiences that guides the agent to more promising avenues (Blundell et al., 2016; Schaul et al., 2016; Parisotto and Salakhutdinov, 2018; Savinov et al., 2018).”
From R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory (2025) · cites this paper as Blundell et al., 2016
Cites
McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995) · Psychological Review · doi:10.1037/0033-295x.102.3.419 · Cited by 5,256 · see citing works
Simple memory: a theory for archicortex
Marr, D. (1971) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1971.0078 · Cited by 2,771 · see citing works
Episodic-like memory during cache recovery by scrub jays
Clayton, N. S., & Dickinson, A. (1998) · Nature · doi:10.1038/26216 · Cited by 1,464 · see citing works
Hippocampal synaptic enhancement and information storage within a distributed memory system
McNaughton, B. L., & Morris, R. G. M. (1987) · Trends in Neurosciences · doi:10.1016/0166-2236(87)90011-7 · Cited by 1,304 · see citing works
(2006) · doi:10.1093/acprof:oso/9780195100273.001.0001 · Cited by 1,186 · see citing works
Decay happens: the role of active forgetting in memory
Hardt, O., Nader, K., & Nadel, L. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.01.001 · Cited by 416 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550 · see citing works
Rainbow: Combining Improvements in Deep Reinforcement Learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11796 · Cited by 1,846 · see citing works
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557 · see citing works
Self-citationReinforcement learning, fast and slow
Botvinick, M., Ritter, S., Wang, J. X., Kurth-Nelson, Z., Blundell, C., & Hassabis, D. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.02.006 · Cited by 667 · see citing works
Self-citationThe hippocampal sharp wave-ripple in memory retrieval for immediate use and consolidation
Joo, H. R., & Frank, L. M. (2018) · Nature Reviews Neuroscience · doi:10.1038/s41583-018-0077-1 · Cited by 430 · see citing works
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269 · see citing works
Stabilizing Transformers for Reinforcement Learning
Parisotto, E., Song, H. F., Rae, J. W., Pascanu, R., Gulcehre, C., Jayakumar, S. M., Jaderberg, M., Kaufman, R. L., Clark, A., Noury, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.06764 · Cited by 199 · see citing works
Self-citationToward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119 · see citing works
Replay in Deep Learning: Current Approaches and Missing Biological Elements
Hayes, T. L., Krishnan, G. P., Bazhenov, M., Siegelmann, H. T., Sejnowski, T. J., & Kanan, C. (2021) · Neural Computation · doi:10.1162/neco_a_01433 · Cited by 106 · see citing works
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
“It is common practice for RL agents to store episodic trajectories to update a parametric policy (Blundell et al., 2016; Pritzel et al., 2017) or establish a non-parametric policy (Ecoffet et al., 2019; Tuyls et al., 2022).”
BackgroundMeta-learned models of cognition
Binz, M., Dasgupta, I., Jagadish, A. K., Botvinick, M., Wang, J. X., & Schulz, E. (2024) · Behavioral and Brain Sciences · doi:10.1017/s0140525x23003266 · Cited by 38 · see citing works
From reinforcement learning to agency: Frameworks for understanding basal cognition
Seifert, G., Sealander, A., Marzen, S., & Levin, M. (2024) · BioSystems · doi:10.1016/j.biosystems.2023.105107 · Cited by 16 · see citing works
Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated
DeChant, C. (2025) · 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00024 · Cited by 5 · see citing works
R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory
Huang, T., Basu, K., Abdelaziz, I., Kapanipathi, P., May, J., & Chen, M. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1464 · Cited by 4 · see citing works
“It leverages a structured replay buffer of the agent’s experiences that guides the agent to more promising avenues (Blundell et al., 2016; Schaul et al., 2016; Parisotto and Salakhut-dinov, 2018; Savinov et al., 2018).”
- Authors
- Blundell, C., Uria, B., Pritzel, A., Li, Y., Ruderman, A., Leibo, J. Z., Rae, J., Wierstra, D., & Hassabis, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1606.04460
- OpenAlex
- W2436711315
- Cited by
- 162
- References
- 6
- Citations by year
- 2026: 1 · 2025: 9 · 2024: 8 · 2023: 27 · 2022: 15 · 2021: 27 · 2020: 19 · 2019: 23 · 2018: 18 · 2017: 12 · 2016: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- episodic memory, sequential decision making, rapid learning, hippocampal model, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
