Hippocampal replays under the scrutiny of reinforcement learning models
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Multiple in vivo studies have shown that place cells from the hippocampus replay previously experienced trajectories. These replays are commonly considered to mainly reflect memory consolidation processes. Some data, however, have highlighted a functional link between replays and reinforcement learning (RL). This theory, extensively used in machine learning, has introduced efficient algorithms and can explain various behavioral and physiological measures from different brain regions. RL algorithms could constitute a mechanistic description of replays and explain how replays can reduce the number of iterations required to explore the environment during learning. We review the main findings concerning the different hippocampal replay types and the possible associated RL models (either model-based, model-free, or hybrid model types). We conclude by tying these frameworks together. We illustrate the link between data and RL through a series of model simulations. This review, at the frontier between informatics and biology, paves the way for future work on replays.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A logical calculus of the ideas immanent in nervous activity
McCulloch, W. S., & Pitts, W. (1943) · The Bulletin of Mathematical Biophysics · doi:10.1007/bf02478259 · Cited by 18,566
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
The role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 2×
“This is because in most decision-making tasks in neuroscience, the inference phase of MF-RL simply consists of reading from a table the Q values of a small finite set of actions, to compare them and make a decision (Bavard et al. 2018; O’Doherty et al. 2004; Palminteri et al. 2015; Pessiglione et al. 2006).”
BackgroundMethodTaube, J., Muller, R., & Ranck, J. (1990) · The Journal of Neuroscience · doi:10.1523/jneurosci.10-02-00420.1990 · Cited by 1,809
Hippocampal sharp wave-ripple: A cognitive biomarker for episodic memory and planning
Buzsáki, G. (2015) · Hippocampus · doi:10.1002/hipo.22488 · Cited by 1,759
Reverse replay of behavioural sequences in hippocampal place cells during the awake state
Foster, D. J., & Wilson, M. A. (2006) · Nature · doi:10.1038/nature04587 · Cited by 1,727
Two-stage model of memory trace formation: A role for “noisy” brain states
Buzsáki, G. (1989) · Neuroscience · doi:10.1016/0306-4522(89)90423-5 · Cited by 1,709 · cited 1×
“What could be the role of these reactivations? Following the proposed role of SWRs in the two-stage memory model (Buzsáki, 1989), researchers have mostly interpreted replays as a sign of a consolidation of memory (Chen and Wilson, 2017; Diekelmann and Born, 2010; Ólafsdóttir et al.”
BackgroundCoordinated memory replay in the visual cortex and hippocampus during sleep
Ji, D., & Wilson, M. A. (2007) · Nature Neuroscience · doi:10.1038/nn1825 · Cited by 1,669 · cited 2×
“, 2018; Walker and Stickgold, 2006): they would copy volatile memories for long-term storage into the cortex (Ji and Wilson, 2007; McClelland et al., 1995; Peyrache et al., 2009).”
BackgroundTemporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618
Neurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598
Self-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564
Selective suppression of hippocampal ripples impairs spatial memory
Girardeau, G., Benchenane, K., Wiener, S. I., Buzsáki, G., & Zugaro, M. B. (2009) · Nature Neuroscience · doi:10.1038/nn.2384 · Cited by 1,560
Memory of sequential experience in the hippocampus during slow wave sleep
Lee, A. K., & Wilson, M. A. (2002) · Neuron · doi:10.1016/s0896-6273(02)01096-6 · Cited by 1,296
Hippocampal place-cell sequences depict future paths to remembered goals
Pfeiffer, B. E., & Foster, D. J. (2013) · Nature · doi:10.1038/nature12112 · Cited by 1,216
Forward and reverse hippocampal place-cell sequences during ripples
Diba, K., & Buzsáki, G. (2007) · Nature Neuroscience · doi:10.1038/nn1961 · Cited by 1,205
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
Putting a spin on the dorsal-ventral divide of the striatum
Voorn, P., Vanderschuren, L. J. M. J., Groenewegen, H. J., Robbins, T. W., & Pennartz, C. M. A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.06.006 · Cited by 1,188
Walker, M. P., & Stickgold, R. (2006) · Annual Review of Psychology · doi:10.1146/annurev.psych.56.091103.070307 · Cited by 1,075 · cited 1×
“What could be the role of these reactivations? Following the proposed role of SWRs in the two-stage memory model (Buzsáki, 1989), researchers have mostly interpreted replays as a sign of a consolidation of memory (Chen and Wilson, 2017; Diekelmann and Born, 2010; Ólafsdóttir et al., 2018; Walker and Stickgold, 2006):…”
BackgroundNeural ensembles in CA3 transiently encode paths forward of the animal at a decision point
Johnson, A., & Redish, A. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.3761-07.2007 · Cited by 1,003 · cited 16×
“If from a computational point of view, we assume the same level of exploration during the MB-RL trajectory sampling simulations accounting for awake HPC forward reactivations during VTE (Johnson and Redish, 2007), then simulations do not generate a prevalence of replayed trajectories along the rewarded side of the maz…”
BackgroundInfluentialAwake hippocampal sharp-wave ripples support spatial memory
Jadhav, S. P., Kemere, C., German, P. W., & Frank, L. M. (2012) · Science · doi:10.1126/science.1217230 · Cited by 924 · cited 1×
“For instance, it has been found that disruption of awake HPC reactivations during SWRs in the W-shaped maze only impairs outbound trials, where memory of the previously visited arm is required to know where to go next, but not inbound trials, which just consist of returning to the central arm (Jadhav et al. 2012).”
BackgroundAwake replay of remote experiences in the hippocampus
Karlsson, M. P., & Frank, L. M. (2009) · Nature Neuroscience · doi:10.1038/nn.2344 · Cited by 769 · cited 3×
“finding that has also been replicated since (Diba and Buzsáki, 2007; Gupta et al., 2010; Karlsson and Frank, 2009).”
BackgroundInfluentialReplay of rule-learning related neural patterns in the prefrontal cortex during sleep
Peyrache, A., Khamassi, M., Benchenane, K., Wiener, S. I., & Battaglia, F. P. (2009) · Nature Neuroscience · doi:10.1038/nn.2337 · Cited by 767
Self-citationProjection-specific modulation of dopamine neuron synapses by aversive and rewarding stimuli
Lammel, S., Ion, D. I., Roeper, J., & Malenka, R. C. (2011) · Neuron · doi:10.1016/j.neuron.2011.03.025 · Cited by 766 · cited 7×
“Such an influence may not necessarily be directly from VTA to HPC, because less coordinated VTA activity with HPC has been reported during sleep than during awake rest (Gomperts et al. 2015), but it could well be through the ventral striatum (VS), which receives both reward signals from the VTA (Lammel et al. 2011) an…”
BackgroundInfluentialHippocampal replay is not a simple function of experience
Gupta, A. S., van der Meer, M. A. A., Touretzky, D. S., & Redish, A. D. (2010) · Neuron · doi:10.1016/j.neuron.2010.01.034 · Cited by 733
Hippocampo-cortical coupling mediates memory consolidation during sleep
Maingret, N., Girardeau, G., Todorova, R., Goutierre, M., & Zugaro, M. (2016) · Nature Neuroscience · doi:10.1038/nn.4304 · Cited by 667 · cited 3×
“This enhanced the performance of animals in a difficult recognition task (Maingret et al. 2016).”
BackgroundRoesch, M. R., Calu, D. J., & Schoenbaum, G. (2007) · Nature Neuroscience · doi:10.1038/nn2013 · Cited by 597
Fast-Forward Playback of Recent Memory Sequences in Prefrontal Cortex During Sleep
Euston, D. R., Tatsuno, M., & McNaughton, B. L. (2007) · Science · doi:10.1126/science.1148979 · Cited by 586
Midbrain dopamine neurons encode decisions for future action
Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006) · Nature Neuroscience · doi:10.1038/nn1743 · Cited by 503 · cited 1×
“MF-RL models were later successfully applied to an extended corpus of experimental data, including instrumental conditioning in rodents (Morris et al. 2006; Roesch et al. 2007) but also instrumental learning tasks in humans, through the use of model-based functional MRI data analysis approaches (O’Doherty et al.”
MethodSong replay during sleep and computational rules for sensorimotor vocal learning
Dave, A. S., & Margoliash, D. (2000) · Science · doi:10.1126/science.290.5492.812 · Cited by 458 · cited 1×
“We know, however, that during the night, many species reactivate various brain regions with patterns sometimes mimicking daytime experience (Dave and Margoliash 2000; Euston et al. 2007; Ji and Wilson 2007; Lansink et al. 2008; Lee and Wilson 2002; Ólafsdóttir et al. 2016; Pavlides and Winson 1989; Wilson and McNaught…”
BackgroundAn automatic valuation system in the human brain: Evidence from functional neuroimaging
Lebreton, M., Jorge, S., Michel, V., Thirion, B., & Pessiglione, M. (2009) · Neuron · doi:10.1016/j.neuron.2009.09.040 · Cited by 455
The Role of Hippocampal Replay in Memory and Planning
Ólafsdóttir, H. F., Bush, D., & Barry, C. (2018) · Current Biology · doi:10.1016/j.cub.2017.10.073 · Cited by 413 · cited 1×
“This is in line with recent experimental results (Ambrose et al. 2016; Ólafsdóttir et al. 2017).”
ResultHippocampus leads ventral striatum in replay of place-reward information
Lansink, C. S., Goltstein, P. M., Lankelma, J. V., McNaughton, B. L., & Pennartz, C. M. A. (2009) · PLoS Biology · doi:10.1371/journal.pbio.1000173 · Cited by 400
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · cited 1×
“In contrast, in MB-RL, retrieving the Q values requires some iterative tree search process where one looks into the future through the model to estimate the possible long-term consequences of immediate actions (Daw et al. 2005; Keramati et al. 2011; Lesaint et al. 2014).”
MethodReverse replay of hippocampal place cells is uniquely modulated by changing reward
Ambrose, R. E., Pfeiffer, B. E., & Foster, D. J. (2016) · Neuron · doi:10.1016/j.neuron.2016.07.047 · Cited by 292 · cited 3×
“Nevertheless, as noted by Lin (1992), replaying trajectories backward, starting from the rewarding location, can sometimes speed up learning; this is in line with the initial interpretation of backward replays (Foster and Wilson 2006) and with the recent observations of Ambrose et al. (2016).”
BackgroundResultExpectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex
Takahashi, Y. K., Roesch, M. R., Wilson, R. C., Toreson, K., O'Donnell, P., Niv, Y., & Schoenbaum, G. (2011) · Nature Neuroscience · doi:10.1038/nn.2957 · Cited by 280 · cited 1×
“In the MB-RL context, the possible role of dopamine, and the precise processes that may underlie value inference, are still unclear and debated (Daw et al. 2011; Khamassi and Humphries 2012; Takahashi et al. 2011).”
BackgroundHippocampal awake replay in fear memory retrieval
Wu, C.-T., Haggerty, D., Kemere, C., & Ji, D. (2017) · Nature Neuroscience · doi:10.1038/nn.4507 · Cited by 210
Hippocampal Replay Captures the Unique Topological Structure of a Novel Environment
Wu, X., & Foster, D. J. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.3414-13.2014 · Cited by 201
VTA neurons coordinate with the hippocampal reactivation of spatial experience
Gomperts, S. N., Kloosterman, F., & Wilson, M. A. (2015) · eLife · doi:10.7554/elife.05360 · Cited by 191
Coordinated grid and place cell replay during rest
Ólafsdóttir, H. F., Carpenter, F., & Barry, C. (2016) · Nature Neuroscience · doi:10.1038/nn.4291 · Cited by 175
Preferential Reactivation of Motivationally Relevant Information in the Ventral Striatum
Lansink, C. S., Goltstein, P. M., Lankelma, J. V., Joosten, R. N. J. M. A., McNaughton, B. L., & Pennartz, C. M. A. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1054-08.2008 · Cited by 168
Pezzulo, G., Rigoli, F., & Chersi, F. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00092 · Cited by 163
Hippocampal-prefrontal reactivation during learning is stronger in awake compared with sleep states
Tang, W., Shin, J. D., Frank, L. M., & Jadhav, S. P. (2017) · The Journal of Neuroscience · doi:10.1523/jneurosci.2291-17.2017 · Cited by 160
Hippocampal sharp-wave ripples in waking and sleeping states
Roumis, D. K., & Frank, L. M. (2015) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2015.05.001 · Cited by 155
Cited by
Shin, J. D., Tang, W., & Jadhav, S. P. (2019) · Neuron · doi:10.1016/j.neuron.2019.09.012 · Cited by 199
“The results of this study also connect well to machine learning literature based on hippocampal replay (Caze et al., 2018; Mattar and Daw, 2018), and thus may inspire improved algorithms.”
ResultLearning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159
Replay in Deep Learning: Current Approaches and Missing Biological Elements
Hayes, T. L., Krishnan, G. P., Bazhenov, M., Siegelmann, H. T., Sejnowski, T. J., & Kanan, C. (2021) · Neural Computation · doi:10.1162/neco_a_01433 · Cited by 106
“…further suggests that selective and partial replay in tasks involving reward allow humans to perform generalization and model-based reinforcement learning (Momennejad, 2020), which has inspired several algorithms in machine learning (see Cazé, Khamassi, Aubin, & Girard, 2018, for a review).”
Metadata
- Authors
- Cazé, R., Khamassi, M., Aubin, L., & Girard, B.
- Venue
- Journal of Neurophysiology
- Publisher
- American Physiological Society
- Published
- 2018-12-01
- Volume
- 120
- Issue
- 6
- Pages
- 2877-2896
- ISSN
- 0022-3077 (ISSN-L)
- 1522-1598
- OpenAlex
- W2895961990
- Cited by
- 38
- References
- 45
- FWCI
- 3.18
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.