CoChat
Sign Up
Cited by 393Memory and Neural MechanismsMemory Processes and InfluencesNeural dynamics and brain function

Prioritized memory access explains planning and hippocampal replay

Open access
Mattar, Marcelo G., Daw, Nathaniel D.·2018·Nature Neuroscience
DOI 10.1038/s41593-018-0232-zPMC PMC6203620

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

To make decisions, animals must evaluate candidate choices by accessing memories of relevant experiences. Yet little is known about which experiences are considered or ignored during deliberation, which ultimately governs choice. We propose a normative theory predicting which memories should be accessed at each moment to optimize future decisions. Using nonlocal 'replay' of spatial locations in hippocampus as a window into memory access, we simulate a spatial navigation task in which an agent accesses memories of locations sequentially, ordered by utility: how much extra reward would be earned due to better choices. This prioritization balances two desiderata: the need to evaluate imminent choices versus the gain from propagating newly encountered information to preceding locations. Our theory offers a simple explanation for numerous findings about place cells; unifies seemingly disparate proposed functions of replay including planning, learning, and consolidation; and posits a mechanism whose dysfunction may underlie pathologies like rumination and craving.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41593-018-0232-z

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Biological agents must balance learning against external actions in their lifetime, choosing when and what to learn.

    “Biological agents, however, do not have this luxury: they must balance learning against external actions in their lifetime, choosing when and what to learn (Mattar and Daw, 2018).”

    From Cognitive Architectures for Language Agents (2023) · cites this paper as Mattar and Daw, 2018

    The Introduction (page not provided) says humans and animals selectively engage in planning and asks which experiences the brain should consider at each moment; it supports the claim that agents face choices about when and what to learn. The paper does not explicitly establish the claimed lifetime balance between learning and external actions, and its stated focus is which memories to access rather than whether to deliberate. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. [2]Daw ND & Dayan P The algorithmic anatomy of model-based evaluation. Phil. Trans. R. Soc. B 369, 20130478 (2014). PMC4186231 25267820
  2. [4]Huys QJ Interplay of approximate planning strategies. Proceedings of the National Academy of Sciences 112, 3098– 3103 (2015). PMC4364207 25675480
  3. [5]Doll BB, Duncan KD, Simon DA, Shohamy D & Daw ND Model-based choices involve prospective neural activity. Nature neuroscience 18, 767– 772 (2015). 25799041 PMC4414826
  4. [6]Daw ND, Niv Y & Dayan P Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature neuroscience 8, 1704– 1711 (2005). 16286932
  5. [8]Gillan CM, Kosinski M, Whelan R, Phelps EA & Daw ND Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. Elife 5, e11305 (2016). 26928075 PMC4786435
  6. [9]Wimmer GE & Shohamy D Preference by association: how memory mechanisms in the hippocampus bias decisions. Science 338, 270– 273 (2012). 23066083
  7. [10]Gershman SJ, Markman AB & Otto AR Retrospective revaluation in sequential decision making: A tale of two systems. Journal of Experimental Psychology: General 143, 182 (2014). 23230992
  8. [11]Momennejad I, Otto AR, Daw ND & Norman KA Offline replay supports planning: fmri evidence from reward revaluation. bioRxiv 196758 (2017).
  9. [12]O’keefe J & Nadel L The hippocampus as a cognitive map (Oxford: Clarendon Press, 1978).
  10. [14]Diba K & Buzs´aki G Forward and reverse hippocampal place-cell sequences during ripples. Nature neuroscience 10, 1241 (2007). 17828259 PMC2039924
  11. [15]Pfeiffer BE & Foster DJ Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74– 79 (2013). 23594744 PMC3990408
  12. [16]Foster DJ & Wilson MA Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 440, 680– 683 (2006). 16474382
  13. [18]Gupta AS, van der Meer MA, Touretzky DS & Redish AD Hippocampal replay is not a simple function of experience. Neuron 65, 695– 705 (2010). 20223204 PMC4460981
  14. [19]Ambrose RE, Pfeiffer BE & Foster DJ Reverse replay of hippocampal place cells is uniquely modulated by changing reward. Neuron 91, 1124– 1136 (2016). 27568518 PMC6013068
  15. [22]Johnson A & Redish AD Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Networks 18, 1163– 1171 (2005). 16198539
  16. [23]Carr MF, Jadhav SP & Frank LM Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval. Nature neuroscience 14, 147– 153 (2011). 21270783 PMC3215304
  17. [24]Jadhav SP, Kemere C, German PW & Frank LM Awake hippocampal sharp-wave ripples support spatial memory. Science 336, 1454– 1458 (2012). 22555434 PMC4441285
  18. [26]Sutton RS Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the seventh international conference on machine learning, 216– 224 (1990).
  19. [29]Schaul T, Quan J, Antonoglou I & Silver D Prioritized experience replay. arXiv preprint arXiv:1511.05952 (2015). · arXiv
  20. [30]Dupret D, O’neill J, Pleydell-Bouverie B & Csicsvari J The reorganization and reactivation of hippocampal maps predict spatial memory performance. Nature neuroscience 13, 995– 1002 (2010). 20639874 PMC2923061
  21. [31]Ólafsdóttir HF, Barry C, Saleem AB, Hassabis D & Spiers HJ Hippocampal place cells construct´ reward related sequences through unexplored space. Elife 4, e06063 (2015). 26112828 PMC4479790
  22. [32]Jackson JC, Johnson A & Redish AD Hippocampal sharp waves and reactivation during awake states depend on repeated sequential experience. Journal of Neuroscience 26, 12415– 12426 (2006). 17135403 PMC6674885
  23. [33]Singer AC & Frank LM Rewarded outcomes enhance reactivation of experience in the hippocampus. Neuron 64, 910– 921 (2009). 20064396 PMC2807414
  24. [34]Wu C-T, Haggerty D, Kemere C & Ji D Hippocampal awake replay in fear memory retrieval. Nature neuroscience 20, 571 (2017). 28218916 PMC5373994
  25. [35]O’Neill J, Senior TJ, Allen K, Huxter JR & Csicsvari J Reactivation of experience-dependent cell assembly patterns in the hippocampus. Nature neuroscience 11, 209 (2008). 18193040
  26. [37]Buhry L, Azizi AH & Cheng S Reactivation, replay, and preplay: how it might all fit together. Neural plasticity 2011 (2011). PMC3171894 21918724
  27. [38]Singer AC, Carr MF, Karlsson MP & Frank LM Hippocampal swr activity predicts correct decisions during the initial learning of an alternation task. Neuron 77, 1163– 1173 (2013). 23522050 PMC3751175
  28. [39]Lansink CS, Goltstein PM, Lankelma JV, McNaughton BL & Pennartz CM Hippocampus leads ventral striatum in replay of place-reward information. PLoS biology 7, e1000173 (2009). 19688032 PMC2717326
  29. [40]Gomperts SN, Kloosterman F & Wilson MA Vta neurons coordinate with the hippocampal reactivation of spatial experience. Elife 4, e05360 (2015). 26465113 PMC4695386
  30. [42]Stachenfeld KL, Botvinick MM & Gershman SJ The hippocampus as a predictive map. bioRxiv 097170 (2017). 28967910
  31. [43]Russek EM, Momennejad I, Botvinick MM, Gershman SJ & Daw ND Predictive representations can link model-based reinforcement learning to model-free mechanisms. bioRxiv 083857 (2017). PMC5628940 28945743
  32. [44]Cushman F & Morris A Habitual control of goal selection in humans. Proceedings of the National Academy of Sciences 112, 13817– 13822 (2015). PMC4653221 26460050
  33. [45]Daw ND, Gershman SJ, Seymour B, Dayan P & Dolan RJ Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204– 1215 (2011). 21435563 PMC3077926
  34. [46]Sadacca BF, Jones JL & Schoenbaum G Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework. Elife 5, e13665 (2016). 26949249 PMC4805544
  35. [47]Doll BB, Bath KG, Daw ND & Frank MJ Variability in dopamine genes dissociates model-based and model-free reinforcement learning. Journal of Neuroscience 36, 1211– 1222 (2016). 26818509 PMC4728725
  36. [48]Sharpe MJ Dopamine transients are sufficient and necessary for acquisition of model-based associations. Nature Neuroscience (2017). PMC5413864 28368385
  37. [49]Momennejad I The successor representation in human reinforcement learning. Nature Human Behaviour 1, 680 (2017). PMC6941356 31024137
  38. [50]Botvinick MM, Niv Y & Barto AC Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113, 262– 280 (2009). 18926527 PMC2783353
  39. [51]Schultz W, Dayan P & Montague PR A neural substrate of prediction and reward. Science 275, 1593– 1599 (1997). 9054347
  40. [52]Coulom R Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, 72– 83 (Springer, 2006).
  41. [54]Anderson JR & Milson R Human memory: An adaptive perspective. Psychological Review 96, 703 (1989).
  42. [55]Sutton RS, McAllester DA, Singh SP & Mansour Y Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, 1057– 1063 (2000).

Cited by

Sorted by citations

Authors
Mattar, M. G., & Daw, N. D.
Affiliations
Princeton University (US)
Corresponding author
Mattar, Marcelo G.
ORCID
Venue
Nature Neuroscience
Publisher
Springer Science and Business Media LLC
Published
2018-10-22
Volume
21
Issue
11
Pages
1609-1617
ISSN
  • 1097-6256 (ISSN-L)
  • 1546-1726
DOI
10.1038/s41593-018-0232-z
PMC
PMC6203620
OpenAlex
W2952504784
Cited by
393
References
11
Citations by year
2026: 40 · 2025: 51 · 2024: 59 · 2023: 48 · 2022: 65 · 2021: 48 · 2020: 45 · 2019: 30 · 2018: 7
FWCI
19.36
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Neuroscience (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
  • Cambridge University Engineering Department Publications Database (submittedVersion)
  • Apollo (University of Cambridge) (submittedVersion)
  • Apollo
Funders
  • DeepMind
  • National Institute on Drug Abuse (R01DA038891)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
Keywords
hippocampal replay, memory access, decision-making, planning, normative theory, place cells, spatial navigation, memory prioritization, memory consolidation, rumination, craving
MeSH terms
Animals, Behavior, Animal, Decision Making, Hippocampus, Memory, Neurons, Reward, Rats, Spatial Navigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review