Reinforcement learning and episodic memory in humans and animals: an integrative framework
Open accessAbstract
We review the psychology and neuroscience of reinforcement learning (RL), which has experienced significant progress in the past two decades, enabled by the comprehensive experimental study of simple learning and decision-making tasks. However, one challenge in the study of RL is computational: The simplicity of these tasks ignores important aspects of reinforcement learning in the real world: (a) State spaces are high-dimensional, continuous, and partially observable; this implies that (b) data are relatively sparse and, indeed, precisely the same situation may never be encountered twice; furthermore, (c) rewards depend on the long-term consequences of actions in ways that violate the classical assumptions that make RL tractable. A seemingly distinct challenge is that, cognitively, theories of RL have largely involved procedural and semantic memory, the way in which knowledge about action values or world models extracted gradually from many experiences can drive choice. This focus on semantic memory leaves out many aspects of memory, such as episodic memory, related to the traces of individual events. We suggest that these two challenges are related. The computational challenge can be dealt with, in part, by endowing RL systems with episodic memory, allowing them to (a) efficiently approximate value functions over complex state spaces, (b) learn with very little data, and (c) bridge long-term dependencies between actions and rewards. We review the computational theory underlying this proposal and the empirical evidence to support it. Our proposal suggests that the ubiquitous and diverse roles of memory in RL may function as part of an integrated learning system.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This approach was extended to show how similar memories could be sampled in order to estimate the value of possible actions.
“This approach was extended to show how similar memories could be sampled in order to estimate the value of possible actions [87].”
From Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated (2025) · cites this paper as [87]
The value of an observation may depend on the long-term past.
“One way of understanding this is to note that the value of an observation may depend on the longterm past [22].”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
A Global Geometric Framework for Nonlinear Dimensionality Reduction
Tenenbaum, J. B., Silva, V. D., & Langford, J. C. (2000) · Science · doi:10.1126/science.290.5500.2319 · Cited by 13,585 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641 · see citing works
Memory and the hippocampus: A synthesis from findings with rats, monkeys, and humans.
Squire, L. R. (1992) · Psychological Review · doi:10.1037/0033-295x.99.2.195 · Cited by 5,376 · see citing works
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · see citing works
The hippocampus as a cognitive map
Passingham, R. E. (1979) · Neuroscience · doi:10.1016/0306-4522(79)90015-0 · Cited by 5,128 · see citing works
Learning With Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Williams, C. K. I. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s269 · Cited by 4,390 · see citing works
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · see citing works
Functional architecture of basal ganglia circuits: neural substrates of parallel processing
Alexander, G. E., & Crutcher, M. D. (1990) · Trends in Neurosciences · doi:10.1016/0166-2236(90)90107-l · Cited by 4,131 · see citing works
Neural systems of reinforcement for drug addiction: from actions to habits to compulsion
Everitt, B. J., & Robbins, T. W. (2005) · Nature Neuroscience · doi:10.1038/nn1579 · Cited by 3,785 · see citing works
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578 · see citing works
Norm theory: Comparing reality to its alternatives.
Kahneman, D. J., & Miller, D. T. (1986) · Psychological Review · doi:10.1037/0033-295x.93.2.136 · Cited by 3,174 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
Self-citationAn Information Processing Theory of Consumer Choice
Ross, I., & Bettman, J. R. (1979) · Journal of Marketing · doi:10.2307/1250155 · Cited by 2,267 · see citing works
Attention, similarity, and the identification-categorization relationship.
Nosofsky, R. M. (1986) · Journal of Experimental Psychology General · doi:10.1037//0096-3445.115.1.39 · Cited by 2,207 · see citing works
Attention, similarity, and the identification–categorization relationship.
Nosofsky, R. M. (1986) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.115.1.39 · Cited by 2,136 · see citing works
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · see citing works
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964 · see citing works
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779 · see citing works
Self-citationA neostriatal habit learning system in humans
Knowlton, B. J., Mangels, J. A., & Squire, L. R. (1996) · Science · doi:10.1126/science.273.5280.1399 · Cited by 1,777 · see citing works
Choice in Context: Tradeoff Contrast and Extremeness Aversion
Simonson, I., & Tversky, A. (1992) · Journal of Marketing Research · doi:10.2307/3172740 · Cited by 1,705 · see citing works
Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans
Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006) · Nature · doi:10.1038/nature05051 · Cited by 1,584 · see citing works
ALCOVE: An exemplar-based connectionist model of category learning.
Kruschke, J. K. (1992) · Psychological Review · doi:10.1037/0033-295x.99.1.22 · Cited by 1,582 · see citing works
Packard, M. G., & McGaugh, J. L. (1996) · Neurobiology of Learning and Memory · doi:10.1006/nlme.1996.0007 · Cited by 1,505 · see citing works
Control of Cognitive Processes: Attention and Performance XVIII
Monsell, S., & Driver, J. (2000) · UCL Discovery (University College London) · Cited by 1,451 · see citing works
Riesbeck, C. K., & Schank, R. C. (2013) · Psychology Press eBooks · doi:10.4324/9780203781821 · Cited by 1,448 · see citing works
The future of memory: remembering, imagining, and the brain
Schacter, D. L., Addis, D. R., Hassabis, D., Martin, V. C., Spreng, R. N., & Szpunar, K. K. (2012) · Neuron · doi:10.1016/j.neuron.2012.11.001 · Cited by 1,398 · see citing works
Neuron-type-specific signals for reward and punishment in the ventral tegmental area
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., & Uchida, N. (2012) · Nature · doi:10.1038/nature10754 · Cited by 1,318 · see citing works
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · see citing works
Hippocampal place-cell sequences depict future paths to remembered goals
Pfeiffer, B. E., & Foster, D. J. (2013) · Nature · doi:10.1038/nature12112 · Cited by 1,216 · see citing works
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190 · see citing works
Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience
Skaggs, W. E., & McNaughton, B. L. (1996) · Science · doi:10.1126/science.271.5257.1870 · Cited by 1,173 · see citing works
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166 · see citing works
Self-citationAll of Nonparametric Statistics
Fotopoulos, S. (2007) · Technometrics · doi:10.1198/tech.2007.s454 · Cited by 1,141 · see citing works
Interactive memory systems in the human brain
Poldrack, R. A., Clark, J., Paré-Blagoev, E. J., Shohamy, D., Creso Moyano, J., Myers, C., & Gluck, M. A. (2001) · Nature · doi:10.1038/35107080 · Cited by 1,132 · see citing works
O’Reilly, R. C., & Frank, M. J. (2005) · Neural Computation · doi:10.1162/089976606775093909 · Cited by 1,066 · see citing works
From Conditioning to Conscious Recollection
Eichenbaum, H., & Cohen, N. J. (2004) · doi:10.1093/acprof:oso/9780195178043.001.0001 · Cited by 1,026 · see citing works
Neuroeconomics: decision making and the brain
(2014) · Choice Reviews Online · doi:10.5860/choice.51-5702 · Cited by 1,020 · see citing works
Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point
Johnson, A., & Redish, A. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.3761-07.2007 · Cited by 1,003 · see citing works
(2010) · Springer eBooks · doi:10.1007/978-0-387-30164-8_588 · Cited by 978 · see citing works
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977 · see citing works
Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval
Carr, M. F., Jadhav, S. P., & Frank, L. M. (2011) · Nature Neuroscience · doi:10.1038/nn.2732 · Cited by 958 · see citing works
Recall and Consumer Consideration Sets: Influencing Choice without Altering Brand Evaluations
Nedungadi, P. (1990) · Journal of Consumer Research · doi:10.1086/208556 · Cited by 927 · see citing works
Mandler, G. (1967) · The Psychology of learning and motivation/The psychology of learning and motivation · doi:10.1016/s0079-7421(08)60516-2 · Cited by 912 · see citing works
The description–experience gap in risky choice
Hertwig, R., & Erev, I. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.09.004 · Cited by 897 · see citing works
A causal link between prediction errors, dopamine neurons and learning
Steinberg, E. E., Keiflin, R., Boivin, J. R., Witten, I. B., Deisseroth, K. A., & Janak, P. H. (2013) · Nature Neuroscience · doi:10.1038/nn.3413 · Cited by 879 · see citing works
COGNITIVE NEUROSCIENCE OF HUMAN MEMORY
Gabrieli, J. D. E. (1998) · Annual Review of Psychology · doi:10.1146/annurev.psych.49.1.87 · Cited by 808 · see citing works
Hare, T. A., O'Doherty, J., Camerer, C. F., Schultz, W., & Rangel, A. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1309-08.2008 · Cited by 792 · see citing works
SUSTAIN: A Network Model of Category Learning.
Love, B. C., Medin, D. L., & Gureckis, T. M. (2004) · Psychological Review · doi:10.1037/0033-295x.111.2.309 · Cited by 778 · see citing works
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 1× · see citing works
“This hybrid architecture was originally proposed in the machine learning literature by Sutton (1991), who referred to it as Dyna.”
MethodStewart, N., Chater, N., & Brown, G. D. A. (2006) · Cognitive Psychology · doi:10.1016/j.cogpsych.2005.10.003 · Cited by 737 · see citing works
Small feedback‐based decisions and their limited correspondence to description‐based decisions
Barron, G., & Erev, I. (2003) · Journal of Behavioral Decision Making · doi:10.1002/bdm.443 · Cited by 725 · see citing works
Reinforcement learning in the brain
Niv, Y. (2009) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2008.12.005 · Cited by 724 · see citing works
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693 · see citing works
Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective
Botvinick, M. M., Niv, Y., & Barto, A. G. (2009) · Cognition · doi:10.1016/j.cognition.2008.08.011 · Cited by 646 · see citing works
Neural computations underlying arbitration between model-based and model-free learning
Lee, S. W., Shimojo, S., & O’Doherty, J. P. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.028 · Cited by 642 · see citing works
Variations in the Sensitivity of Instrumental Responding to Reinforcer Devaluation
Adams, C. D. (1982) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748208400878 · Cited by 635 · see citing works
Characterizing a psychiatric symptom dimension related to deficits in goal-directed control
Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A., & Daw, N. D. (2016) · eLife · doi:10.7554/elife.11305 · Cited by 586 · see citing works
Self-citationMemory and Attentional Factors in Consumer Choice: Concepts and Research Methods
Lynch, J. G., & Srull, T. K. (1982) · Journal of Consumer Research · doi:10.1086/208893 · Cited by 581 · see citing works
Probabilistic models of cognition: exploring representations and inductive biases
Griffiths, T. L., Chater, N., Kemp, C., Perfors, A., & Tenenbaum, J. B. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.05.004 · Cited by 568 · see citing works
Instance‐based learning in dynamic decision making
González, C., Lerch, J. F., & Lebiere, C. J. (2003) · Cognitive Science · doi:10.1207/s15516709cog2704_2 · Cited by 538 · see citing works
Addiction as a Computational Process Gone Awry
Redish, A. D. (2004) · Science · doi:10.1126/science.1102384 · Cited by 534 · see citing works
Integrating Memories in the Human Brain: Hippocampal-Midbrain Encoding of Overlapping Events
Daphna Shohamy, A., & Wagner, A. D. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.023 · Cited by 525 · see citing works
Preference by Association: How Memory Mechanisms in the Hippocampus Bias Decisions
Wimmer, G. E., & Daphna Shohamy, A. (2012) · Science · doi:10.1126/science.1223252 · Cited by 497 · see citing works
DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS
Lau, B., & Glimcher, P. W. (2005) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.2005.110-04 · Cited by 485 · see citing works
The construction system of the brain
Hassabis, D., & Maguire, E. A. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0296 · Cited by 483 · see citing works
Working-memory capacity protects model-based learning from stress
Otto, A. R., Raio, C. M., Chiang, A., Phelps, E. A., & Daw, N. D. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1312011110 · Cited by 472 · see citing works
Self-citationBrogden, W. J. (1939) · Journal of Experimental Psychology · doi:10.1037/h0058944 · Cited by 424 · see citing works
Collins, A. G. E., & Frank, M. J. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07980.x · Cited by 419 · see citing works
Hippocampus leads ventral striatum in replay of place-reward information
Lansink, C. S., Goltstein, P. M., Lankelma, J. V., McNaughton, B. L., & Pennartz, C. M. A. (2009) · PLoS Biology · doi:10.1371/journal.pbio.1000173 · Cited by 400 · see citing works
The Role of Learning in the Operation of Motivational Systems
Dickinson, A., & Balleine, B. W. (2002) · Wiley eBooks · doi:10.1002/0471214426.pas0312 · Cited by 387 · see citing works
Computational perspectives on dopamine function in prefrontal cortex
Cohen, J. D., Braver, T. S., & Brown, J. W. (2002) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(02)00314-8 · Cited by 386 · see citing works
Context, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386 · cited 2× · see citing works
“Related ideas have begun to be explored in RL tasks (Gershman et al. 2010, Niv et al. 2015, Vaidya & Fellows 2015).”
BackgroundSelf-citationA Theory of Case-Based Decisions
Gilboa, I., & Schmeidler, D. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9780511493539 · Cited by 375 · see citing works
Reinforcement Learning in Multidimensional Environments Relies on Attention Mechanisms
Niv, Y., Daniel, R., Geana, A., Gershman, S. J., Leong, Y. C., Radulescu, A., & Wilson, R. C. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.2978-14.2015 · Cited by 362 · see citing works
Self-citationOtto (2013) · doi:10.1177/0956797612463080 · Cited by 352 · see citing works
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · see citing works
Reinforcement learning with Gaussian processes
Engel, Y., Mannor, S., & Meir, R. (2005) · International Conference on Machine Learning · doi:10.1145/1102351.1102377 · Cited by 342 · see citing works
Reward and choice encoding in terminals of midbrain dopamine neurons depends on striatal target
Parker, N. F., Cameron, C. M., Taliaferro, J. P., Lee, J., Choi, J. Y., Davidson, T. J., Daw, N. D., & Witten, I. B. (2016) · Nature Neuroscience · doi:10.1038/nn.4287 · Cited by 340 · see citing works
Self-citationAbsolute Identification by Relative Judgment.
Stewart, N., Brown, G. D. A., & Nick Chater, A. (2005) · Psychological Review · doi:10.1037/0033-295x.112.4.881 · Cited by 339 · see citing works
What constitutes an episode in episodic memory?
Ezzyat, Y., & Davachi, L. (2011) · Psychological Science · doi:10.1177/0956797610393742 · Cited by 298 · see citing works
Hart, A. S., Rutledge, R. B., Glimcher, P. W., & Phillips, P. E. M. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.2489-13.2014 · Cited by 294 · see citing works
The hippocampal-striatal axis in learning, prediction and goal-directed behavior
Pennartz, C. M. A., Ito, R., Verschure, P. F. M. J., Battaglia, F. P., & Robbins, T. W. (2011) · Trends in Neurosciences · doi:10.1016/j.tins.2011.08.001 · Cited by 287 · see citing works
Mahadevan, S., & Maggioni, M. (2007) · doi:10.5555/1314498.1314570 · Cited by 287 · see citing works
Model-based choices involve prospective neural activity
Doll, B. B., Duncan, K., Simon, D. A., Shohamy, D., & Daw, N. (2015) · Nature Neuroscience · doi:10.1038/nn.3981 · Cited by 284 · see citing works
Self-citationInstance-based learning: Integrating sampling and repeated decisions from experience.
Gonzalez, C., & Dutt, V. (2011) · Psychological Review · doi:10.1037/a0024558 · Cited by 255 · see citing works
Kernel-Based Reinforcement Learning
Ormoneit, D., & Sen, Ś. (2002) · Machine Learning · doi:10.1023/a:1017928328829 · Cited by 254 · see citing works
Serotonin Selectively Modulates Reward Value in Human Decision-Making
Seymour, B., Daw, N. D., Roiser, J. P., Dayan, P., & Dolan, R. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.0053-12.2012 · Cited by 252 · see citing works
Self-citationHau, R. C., Pleskac, T. J., Kiefer, J., & Hertwig, R. (2008) · Journal of Behavioral Decision Making · doi:10.1002/bdm.598 · Cited by 240 · see citing works
Theta Phase Precession in Rat Ventral Striatum Links Place and Reward Information
van der Meer, M. A. A., & Redish, A. D. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.4869-10.2011 · Cited by 222 · see citing works
Reminders of past choices bias decisions for reward in humans
Bornstein, A. M., Khaw, M. W., Daphna Shohamy, A., & Daw, N. D. (2017) · Nature Communications · doi:10.1038/ncomms15958 · Cited by 222 · see citing works
Self-citationDezfouli, A., & Balleine, B. W. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003364 · Cited by 221 · see citing works
Retrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217 · see citing works
Self-citationWorking Memory Contributions to Reinforcement Learning Impairments in Schizophrenia
Collins, A. G. E., Brown, J. K., Gold, J. M., Waltz, J. A., & Frank, M. J. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.0989-14.2014 · Cited by 214 · see citing works
Interplay of approximate planning strategies
Huys, Q. J. M., Lally, N., Faulkner, P., Eshel, N., Seifritz, E., Gershman, S. J., Dayan, P., & Roiser, J. P. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1414219112 · Cited by 207 · see citing works
Self-citationLoss aversion, diminishing sensitivity, and the effect of experience on repeated decisions
Erev, I., Ert, E., & Yechiam, E. (2008) · Journal of Behavioral Decision Making · doi:10.1002/bdm.602 · Cited by 198 · see citing works
The algorithmic anatomy of model-based evaluation
Daw, N. D., & Dayan, P. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0478 · Cited by 195 · see citing works
Self-citationRao, R. P. N. (2010) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2010.00146 · Cited by 190 · see citing works
Mistake: The Effect of Previously Encountered Prices on Current Housing Demand
Simonsohn, U., & Loewenstein, G. F. (2006) · The Economic Journal · doi:10.1111/j.1468-0297.2006.01052.x · Cited by 174 · see citing works
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · see citing works
Self-citationHabitual control of goal selection in humans
Cushman, F. A., & Morris, A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1506367112 · Cited by 165 · see citing works
Kernels and Distances for Structured Data
Gärtner, T., Lloyd, J. W., & Flach, P. A. (2004) · Machine Learning · doi:10.1023/b:mach.0000039777.23772.30 · Cited by 159 · see citing works
Discovering latent causes in reinforcement learning
Gershman, S. J., Norman, K. A., & Niv, Y. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.07.007 · Cited by 151 · see citing works
Self-citationStimulus representation and the timing of reward-prediction errors in models of the dopamine system
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008) · Neural Computation · doi:10.1162/neco.2008.11-07-654 · Cited by 148 · see citing works
Integrating memories to guide decisions
Daphna Shohamy, A., & Daw, N. D. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.08.010 · Cited by 141 · see citing works
Self-citationEpisodic memories predict adaptive value-based decision-making.
Murty, V. P., FeldmanHall, O., Hunter, L. E., Phelps, E. A., & Davachi, L. (2016) · Journal of Experimental Psychology General · doi:10.1037/xge0000158 · Cited by 140 · see citing works
Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. M. (2016) · eLife · doi:10.7554/elife.13665 · Cited by 137 · see citing works
Remembering the best and worst of times: Memories for extreme outcomes bias risky decisions
Madan, C. R., Ludvig, E. A., & Spetch, M. L. (2013) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0542-9 · Cited by 134 · see citing works
Episodic Memory Encoding Interferes with Reward Learning and Decreases Striatal Prediction Errors
Wimmer, G. E., Braun, E. K., Daw, N. D., & Daphna Shohamy, A. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.0204-14.2014 · Cited by 126 · see citing works
Self-citationSensory preconditioning of human subjects.
Brogden, W. J. (1947) · Journal of Experimental Psychology · doi:10.1037/h0058465 · Cited by 124 · see citing works
Learning, risk attitude and hot stoves in restless bandit problems
Biele, G., Erev, I., & Ert, E. (2008) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2008.05.006 · Cited by 119 · see citing works
The role of dopamine in cognitive sequence learning: evidence from Parkinson’s disease
Daphna Shohamy, A., Myers, C. E., Grossman, S., Sage, J., & Gluck, M. A. (2004) · Behavioural Brain Research · doi:10.1016/j.bbr.2004.05.023 · Cited by 114 · see citing works
Novelty and Inductive Generalization in Human Reinforcement Learning
Gershman, S., & Niv, Y. (2015) · Topics in Cognitive Science · doi:10.1111/tops.12138 · Cited by 102 · see citing works
Self-citationMultiple Systems for Value Learning
Daw, N. D., & O’Doherty, J. P. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00021-8 · Cited by 98 · see citing works
Self-citationValue Learning through Reinforcement
Daw, N. D., & Tobler, P. N. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00015-2 · Cited by 96 · see citing works
Self-citationCooperative interactions between hippocampal and striatal systems support flexible navigation
Brown, T. I., Ross, R. S., Tobyne, S. M., & Stern, C. E. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.046 · Cited by 86 · see citing works
Schönberg, T., O’Doherty, J. P., Joel, D., Inzelberg, R., Segev, Y., & Daw, N. D. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.08.011 · Cited by 81 · see citing works
Self-citationDiuk, C., Tsai, K., Wallis, J. D., Botvinick, M., & Niv, Y. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.5445-12.2013 · Cited by 79 · see citing works
Modeling the role of working memory and episodic memory in behavioral tasks
Zilli, E. A., & Hasselmo, M. E. (2007) · Hippocampus · doi:10.1002/hipo.20382 · Cited by 76 · see citing works
Temporal structure in associative retrieval
Kurth-Nelson, Z. L., Barnes, G. R., Sejdinović, D., Dolan, R. J., & Dayan, P. (2015) · eLife · doi:10.7554/elife.04919 · Cited by 73 · see citing works
Priming memories of past wins induces risk seeking.
Ludvig, E. A., Madan, C. R., & Spetch, M. L. (2014) · Journal of Experimental Psychology General · doi:10.1037/xge0000046 · Cited by 67 · see citing works
Grid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement Learning
Gustafson, N. J., & Daw, N. D. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002235 · Cited by 65 · see citing works
Self-citationDoes Cognitive Science Need Kernels?
Jäkel, F., Schölkopf, B., & Wichmann, F. A. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.06.002 · Cited by 64 · see citing works
Evidence integration in model-based tree search
Solway, A., & Botvinick, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1505483112 · Cited by 56 · see citing works
Fu, W., & Anderson, J. R. (2007) · Psychological Research · doi:10.1007/s00426-007-0113-7 · Cited by 52 · see citing works
Advanced Reinforcement Learning
Daw, N. D. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00016-4 · Cited by 44 · see citing works
Self-citationRoss, R. S., Sherrill, K. R., & Stern, C. E. (2011) · Brain Research · doi:10.1016/j.brainres.2011.09.038 · Cited by 39 · see citing works
Vaidya, A. R., & Fellows, L. K. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.1607-15.2015 · Cited by 37 · see citing works
Reminders of past choices bias decisions for reward in humansPreprint
Bornstein, A. M., Khaw, M. W., Daphna Shohamy, A., & Daw, N. D. (2015) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/033910 · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 6
- Bellman R 1957 Dynamic Programming Princeton University Press
- Kearns MJ Singh SP 2000 Bias-Variance Error Bounds for Temporal Difference Updates COLT 142 147
- Lengyel M Dayan P 2007 Hippocampal Contributions to Control: The Third Way NIPS 20 889 896
- Lieder F Hsu M Griffiths TL 2014 The high availability of extreme events serves resource-rational decision-making Proceedings of the 36th Annual Conference of the Cognitive Science Society 2567 2572
- Stachenfeld KL Botvinick M Gershman SJ 2014 Design principles of the hippocampal cognitive map Advances in Neural Information Processing Systems 27 2528 2536
- Todd MT Niv Y Cohen JD 2009 Learning to use working memory in partially observable environments through dopaminergic reinforcement Advances in neural information processing systems 1689 1696
Cited by
Sorted by citations
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557 · see citing works
Reinforcement learning, fast and slow
Botvinick, M., Ritter, S., Wang, J. X., Kurth-Nelson, Z., Blundell, C., & Hassabis, D. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.02.006 · Cited by 667 · see citing works
Cognitive computational neuroscience
Kriegeskorte, N., & Douglas, P. K. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0210-5 · Cited by 535 · see citing works
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269 · see citing works
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236 · see citing works
Self-citationPlanning and navigation as active inference
Kaplan, R., & Friston, K. J. (2018) · Biological Cybernetics · doi:10.1007/s00422-018-0753-2 · Cited by 203 · see citing works
A Shared Vision for Machine Learning in Neuroscience
Vu, M.-A. T., Adalı, T., Ba, D., Buzsáki, G., Carlson, D., Heller, K., Liston, C., Rudin, C., Sohal, V. S., Widge, A. S., et al. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0508-17.2018 · Cited by 191 · see citing works
The successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191 · see citing works
Self-citationThe Tortoise and the Hare: Interactions between Reinforcement Learning and Working Memory
Collins, A. G. E. (2018) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01238 · Cited by 97 · see citing works
Goal-directed actions transiently depend on dorsal hippocampus
Bradfield, L. A., Leung, B. K., Boldt, S., Liang, S., & Balleine, B. W. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-0693-8 · Cited by 65 · see citing works
Degeneracy and Redundancy in Active Inference
Sajid, N., Parr, T., Hope, T. M., Price, C. J., & Friston, K. J. (2020) · Cerebral Cortex · doi:10.1093/cercor/bhaa148 · Cited by 49 · see citing works
Emergence of belief-like representations through reinforcement learning
Hennig, J. A., Romero Pinto, S. A., Yamaguchi, T., Linderman, S. W., Uchida, N., & Gershman, S. J. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011067 · Cited by 41 · see citing works
Self-citationMeta-learned models of cognition
Binz, M., Dasgupta, I., Jagadish, A. K., Botvinick, M., Wang, J. X., & Schulz, E. (2024) · Behavioral and Brain Sciences · doi:10.1017/s0140525x23003266 · Cited by 38 · see citing works
Reinforcement learning and artificial agency
Butlin, P. (2024) · Mind & Language · doi:10.1111/mila.12458 · Cited by 40 · see citing works
When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment
Ni, T., Ma, M., Eysenbach, B., & Bacon, P.-L. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2194 · Cited by 7 · see citing works
Elements of episodic memory: insights from artificial agents
Boyle, A., & Blomkvist, A. (2024) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0416 · Cited by 6 · see citing works
Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated
DeChant, C. (2025) · 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00024 · Cited by 5 · see citing works
“This approach was extended to show how similar memories could be sampled in order to estimate the value of possible actions [87].”
Flexible Prefrontal Control over Hippocampal Episodic Memory for Goal-Directed Generalization
Zheng, Y., Wolf, N., Ranganath, C., O'Reilly, R. C., & McKee, K. L. (2025) · Proceedings of Cognitive Computational Neuroscience 2025 · doi:10.32470/jnqbpro
- Authors
- Gershman, S. J., & Daw, N. D.
- Affiliations
- Harvard University (US)
- Princeton University (US)
- Venue
- Annual Review of Psychology
- Publisher
- Annual Reviews
- Published
- 2017-01-03
- Volume
- 68
- Issue
- 1
- Pages
- 101-128
- ISSN
- 0066-4308 (ISSN-L)
- 1545-2085
- PMC
- PMC5953519
- OpenAlex
- W2346736747
- Cited by
- 446
- References
- 133
- Citations by year
- 2026: 18 · 2025: 63 · 2024: 53 · 2023: 53 · 2022: 58 · 2021: 51 · 2020: 45 · 2019: 35 · 2018: 38 · 2017: 30 · 2016: 1 · 2015: 1
- FWCI
- 5.51
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Annual Review of Psychology (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute on Drug Abuse (R01DA038891)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- reinforcement learning, episodic memory, semantic memory, procedural memory, value function approximation, instance-based learning, decision-making
- MeSH terms
- Animals, Brain, Decision Making, Humans, Learning, Reinforcement, Psychology, Reward, Memory, Episodic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
