Learning to predict by the methods of temporal differences
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Brain discoveries made in the twentieth century inspired new machine learning algorithms: the hierarchy of areas in visual cortex inspired convolutional neural networks, and operant conditioning inspired the temporal difference learning algorithm for reinforcement learning.
“Brain discoveries made in the twentieth century inspired new machine learning algorithms: the hierarchy of areas in visual cortex inspired convolutional neural networks (LeCun et al., 1998, 2015), and operant conditioning inspired the temporal difference learning algorithm for reinforcement learning (Sutton, 1988).”
From Large Language Models and the Reverse Turing Test (2023) · cites this paper as Sutton, 1988
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredTD-Gammon was based on the temporal difference learning algorithm coupled with a neural network that learned to predict the value function for board positions.
“TD-Gammon was based on the temporal difference learning algorithm (Sutton, 1988) coupled with a neural network that learned to predict the value function for board positions and was a harbinger of the future.”
From Large Language Models and the Reverse Turing Test (2023) · cites this paper as Sutton, 1988
Model-free control involves a particular sort of prediction error, the best known example of which is the temporal difference (TD) prediction error.
“By contrast, model-free control involves a particular sort of prediction error, the best known example of which is the temporal difference (TD) prediction error (Sutton, 1988).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1988
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,501 · see citing works
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
Varga, R. S. (2000) · Springer series in computational mathematics · doi:10.1007/978-3-642-05156-2 · Cited by 5,835 · see citing works
(1991) · Springer eBooks · doi:10.1007/978-3-7091-2840-4 · Cited by 5,095 · see citing works
A Learning Algorithm for Boltzmann Machines
Ackley, D., Hinton, G. E., & Sejnowski, T. (1985) · Cognitive Sciences · doi:10.1016/b978-0-08-051581-6.50053-2 · Cited by 4,462 · see citing works
Some Studies in Machine Learning Using the Game of Checkers
Samuel, A. L. (1959) · IBM Journal of Research and Development · doi:10.1147/rd.33.0210 · Cited by 4,443 · see citing works
Bramble, J. H., & Varga, R. S. (1963) · Mathematics of Computation · doi:10.2307/2003862 · Cited by 4,154 · see citing works
WIDROW, B., & HOFF, M. E. (1960) · doi:10.21236/ad0241531 · Cited by 4,009 · see citing works
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402 · see citing works
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,238 · see citing works
Self-citationKemeny, J. G., & Snell, J. L. (1976) · Medical Entomology and Zoology · Cited by 2,813 · see citing works
Alexander, S. T. (1986) · Springer eBooks · doi:10.1007/978-1-4612-4978-8 · Cited by 2,646 · see citing works
(2013) · Springer eBooks · Cited by 2,625 · see citing works
Toward a modern theory of adaptive networks: Expectation and prediction.
Sutton, R. S., & Barto, A. G. (1981) · Psychological Review · doi:10.1037/0033-295x.88.2.135 · Cited by 1,450 · see citing works
Self-citationSome studies in machine learning using the game of checkers
Samuel, A. L. (2000) · IBM Journal of Research and Development · doi:10.1147/rd.441.0206 · Cited by 1,439 · see citing works
Machine learning: An artificial intelligence approach
Stefik, M. (1985) · Artificial Intelligence · doi:10.1016/0004-3702(85)90005-0 · Cited by 982 · see citing works
Feigenbaum, E. A., & Feldman, J. (1963) · Cited by 919 · see citing works
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 769 · see citing works
Self-citationStuck, B. W., & Arthurs, E. (1987) · Int. CMG Conference · Cited by 563 · see citing works
Widrow, B., & Stearns, S. D. (1985) · International Linear Collider · Cited by 562 · see citing works
Dynamic programming: Models and applications
Denardo, E. V. (1982) · CERN Document Server (European Organization for Nuclear Research) · Cited by 409 · see citing works
Model Neural Networks and Behavior
Selverston, A. I. (1985) · Springer eBooks · doi:10.1007/978-1-4757-5858-0 · Cited by 397 · see citing works
Holland, J. H. (1986) · American Association for Artificial Intelligence eBooks · Cited by 328 · see citing works
A neuronal model of classical conditioning
Klopf, A. H. (1988) · Psychobiology · doi:10.3758/bf03333113 · Cited by 318 · see citing works
Strategy Learning with Multilayer Connectionist Representations
Anderson, C. W. (1987) · Elsevier eBooks · doi:10.1016/b978-0-934613-41-5.50014-3 · Cited by 197 · see citing works
Intelligent behavior as an adaptation to the task environment
Booker, L. B. (1982) · Deep Blue (University of Michigan) · Cited by 187 · see citing works
Learning by statistical cooperation of self-interested neuron-like computing elements.
Barto, A. (1985) · PubMed · Cited by 183 · see citing works
An adaptive optimal controller for discrete-time Markov environments
Witten, I. H. (1977) · Information and Control · doi:10.1016/s0019-9958(77)90354-0 · Cited by 180 · see citing works
Machine learning: a guide to current research
Mitchell, T. M., Carbonell, J., & Michalski, R. S. (1986) · CERN Document Server (European Organization for Nuclear Research) · Cited by 135 · see citing works
Learning and Problem Solving with Multilayer Connectionist Systems
Anderson, C. W. (1986) · Cited by 110 · see citing works
A Temporal-Difference Model of Classical Conditioning
Sutton, R. S., & Barto, A. G. (1987) · eScholarship (California Digital Library) · Cited by 89 · see citing works
Self-citationMoore, J. W., Desmond, J. E., Berthier, N. E., Blazis, D. E. J., Sutton, R. S., & Barto, A. G. (1986) · Behavioural Brain Research · doi:10.1016/0166-4328(86)90092-6 · Cited by 86 · see citing works
Self-citationDietterich, T., & Michalski, R. S. (1985) · doi:10.13021/mars/3396 · Cited by 71 · see citing works
Kehoe, E. J., Schreurs, B. G., & Graham, P. (1987) · Animal Learning & Behavior · doi:10.3758/bf03205056 · Cited by 66 · see citing works
Gelperin, A., Hopfield, J. J., & Tank, D. W. (1985) · Springer eBooks · doi:10.1007/978-1-4757-5858-0_13 · Cited by 59 · see citing works
A unified theory of heuristic evaluation functions and its application to learning
Christensen, J., & Korf, R. E. (1986) · Cited by 42 · see citing works
Disjunctive models of Boolean category learning
Hampson, S. E., & Volper, D. J. (1987) · Biological Cybernetics · doi:10.1007/bf00317987 · Cited by 28 · see citing works
The Learning of World Models By Connectionist Networks
Sutton, R. S., & Pinette, B. (1985) · eScholarship (California Digital Library) · Cited by 16 · see citing works
Self-citationA neural model of adaptive behavior
Hampson, S. E. (1983) · eScholarship (California Digital Library) · Cited by 15 · see citing works
Learning Static Evaluation Functions by Linear Regression
Christensen, J. (1986) · Kluwer international series in engineering and computer science · doi:10.1007/978-1-4613-2279-5_9 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Anderson, C. W. (1986). Learning and problem solving with multilayer con- nectionist sy,stcrn,s. Doctoral dissertation, Department of Computer and Information Science, University of Massaclmsetts, Amherst.
- Anderson, C. W. (1987). Strategy learning with multilayer conneetionist repre- sentations. Proceedings of the Fourth International Work,shop on Machine Learning (pp. 103 114). Irvine. CA: Morgan Kauflnann.
- Booker. L. B. (19~2). Intelligettt behavior a.~ an adaptation to the task et~t~'- ronrnent. Doctoral dissertation. Department of Computer and Colmmlni- cation Sciences, University of Michigan. Aim Arbor.
- Christensen. ,l., & Korf, R. E. (1986). A unified theory of hem'istic evaluathm flmetions and its application to learning. Proceedings of the Fifth National Conference on Artificial Intelligence (Pt)-1.18 152). Philadelphia. PA: Morgan Kaufmann.
- Kemeny, J. G., & Snell, J, L. (1976). Finite Markov chain,s. New York: Springer-Verlag.
Cited by
Sorted by citations
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084 · see citing works
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · see citing works
Predictive reward signal of dopamine neurons
Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.1.1 · Cited by 4,564 · see citing works
Holroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027 · see citing works
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · see citing works
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · see citing works
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · see citing works
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
Elfwing, S., Uchibe, E., & Doya, K. (2018) · Neural Networks · doi:10.1016/j.neunet.2017.12.012 · Cited by 2,119 · see citing works
Reverse replay of behavioural sequences in hippocampal place cells during the awake state
Foster, D. J., & Wilson, M. A. (2006) · Nature · doi:10.1038/nature04587 · Cited by 1,727 · see citing works
Rainbow: Combining Improvements in Deep Reinforcement Learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11796 · Cited by 1,846 · see citing works
Temporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618 · see citing works
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · see citing works
Cooperative Multi-Agent Learning: The State of the Art
Panait, L., & Luke, S. (2005) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-005-2631-2 · Cited by 1,237 · see citing works
Boutaba, R., Salahuddin, M. A., Limam, N., Ayoubi, S., Shahriar, N., Estrada-Solano, F., & Caicedo, O. M. (2018) · Journal of Internet Services and Applications · doi:10.1186/s13174-018-0087-2 · Cited by 1,036 · see citing works
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977 · see citing works
Computational roles for dopamine in behavioural control
Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004) · Nature · doi:10.1038/nature03015 · Cited by 966 · see citing works
Opponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844 · see citing works
Human insula activation reflects risk prediction errors as well as risk
Preuschoff, K., Quartz, S. R., & Bossaerts, P. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.4286-07.2008 · Cited by 784 · see citing works
Intact-brain analyses reveal distinct information carried by SNc dopamine subcircuits
Lerner, T. N., Shilyansky, C., Davidson, T. J., Evans, K. E., Beier, K. T., Zalocusky, K. A., Crow, A. K., Malenka, R. C., Luo, L., Tomer, R., et al. (2015) · Cell · doi:10.1016/j.cell.2015.07.014 · Cited by 794 · see citing works
“…neurons suggested that a key function of DA is to transmit reward prediction error signals (Mirenowicz and Schultz, 1996; Schultz et al., 1997; Waelti et al., 2001), a hypothesis concordant with temporal difference learning models (Montague et al., 1996, 2004; Steinberg et al., 2013; Sutton, 1988).”
BackgroundDyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · see citing works
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762 · see citing works
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693 · see citing works
Neuroevolution: from architectures to learning
Floreano, D., Dürr, P., & Mattiussi, C. (2008) · Evolutionary Intelligence · doi:10.1007/s12065-007-0002-4 · Cited by 734 · see citing works
Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making
Frank, M. J. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.03.006 · Cited by 706 · see citing works
Predictability modulates human brain response to reward
Berns, G. S., McClure, S. M., Pagnoni, G., & Montague, P. R. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-08-02793.2001 · Cited by 711 · see citing works
Neural differentiation of expected reward and risk in human subcortical structures
Preuschoff, K., Bossaerts, P., & Quartz, S. R. (2006) · Neuron · doi:10.1016/j.neuron.2006.06.024 · Cited by 692 · see citing works
Karbach, J., & Verhaeghen, P. (2014) · Psychological Science · doi:10.1177/0956797614548725 · Cited by 589 · see citing works
Decision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580 · see citing works
What the orbitofrontal cortex does not do
Stalnaker, T. A., Cooch, N. K., & Schoenbaum, G. (2015) · Nature Neuroscience · doi:10.1038/nn.3982 · Cited by 531 · see citing works
Learning, reward, and decision making
O'Doherty, J. P., Cockburn, J., & Pauli, W. M. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-010416-044216 · Cited by 475 · see citing works
Cognition and control in schizophrenia: a computational model of dopamine and prefrontal function
Braver, T. S., Barch, D. M., & Cohen, J. D. (1999) · Biological Psychiatry · doi:10.1016/s0006-3223(99)00116-x · Cited by 507 · see citing works
Actor–critic models of the basal ganglia: new anatomical and computational perspectives
Joel, D., Niv, Y., & Ruppin, E. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00047-3 · Cited by 500 · see citing works
Learning and selective attention
Dayan, P., Kakade, S., & Montague, P. R. (2000) · Nature Neuroscience · doi:10.1038/81504 · Cited by 495 · see citing works
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446 · see citing works
Multiple model-based reinforcement learning
Doya, K., Samejima, K., Katagiri, K.-I., & Kawato, M. (2002) · Neural Computation · doi:10.1162/089976602753712972 · Cited by 476 · see citing works
Opponency revisited: competition and cooperation between dopamine and serotonin
Boureau, Y.-L., & Dayan, P. (2011) · Neuropsychopharmacology · doi:10.1038/npp.2010.151 · Cited by 456 · see citing works
Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103535 · Cited by 445 · see citing works
Self-citationPan, W.-X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.1478-05.2005 · Cited by 429 · see citing works
Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain
Niv, Y., Edlund, J. A., Dayan, P., & O'Doherty, J. P. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.5498-10.2012 · Cited by 356 · see citing works
Barto, A., Mirolli, M., & Baldassarre, G. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00907 · Cited by 397 · see citing works
“Error correction is also the basis of Temporal Difference (TD) learning (Sutton, 1988), where the error incorporates information about the long-term expectation of reward and not just the immediate reward.”
BackgroundBanishing the homunculus: Making working memory work
Hazy, T. E., Frank, M. J., & O’Reilly, R. C. (2006) · Neuroscience · doi:10.1016/j.neuroscience.2005.04.067 · Cited by 385 · see citing works
Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation
Dayan, P., & Berridge, K. C. (2014) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0277-8 · Cited by 357 · see citing works
Value iteration and adaptive dynamic programming for data-driven adaptive optimal control design
Bian, T., & Jiang, Z.-P. (2016) · Automatica · doi:10.1016/j.automatica.2016.05.003 · Cited by 348 · see citing works
O'Doherty, J. P., Buchanan, T. W., Seymour, B., & Dolan, R. J. (2006) · Neuron · doi:10.1016/j.neuron.2005.11.014 · Cited by 322 · see citing works
Reinforcement learning for intelligent healthcare applications: A survey
Coronato, A., Naeem, M., De Pietro, G., & Paragliola, G. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101964 · Cited by 320 · see citing works
Learning to perceive and act by trial and error
Whitehead, S. D., & Ballard, D. H. (1991) · Machine Learning · doi:10.1007/bf00058926 · Cited by 300 · see citing works
Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons
Satoh, T., Nakai, S., Sato, T., & Kimura, M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-30-09913.2003 · Cited by 298 · see citing works
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236 · see citing works
Probabilistic Incremental Program Evolution
Salustowicz, R., & Schmidhuber, J. (1997) · Evolutionary Computation · doi:10.1162/evco.1997.5.2.123 · Cited by 249 · see citing works
Dopamine reward prediction errors reflect hidden-state inference across time
Starkweather, C. K., Babayan, B. M., Uchida, N., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4520 · Cited by 206 · see citing works
Climbing fibers encode a temporal-difference prediction error during cerebellar learning in mice
Ohmae, S., & Medina, J. F. (2015) · Nature Neuroscience · doi:10.1038/nn.4167 · Cited by 243 · see citing works
O'Reilly, R. C. (2002) · Cerebral Cortex · doi:10.1093/cercor/12.3.246 · Cited by 241 · see citing works
Model-Free reinforcement learning with continuous action in practice
Degris, T., Pilarski, P. M., & Sutton, R. S. (2012) · 2012 American Control Conference (ACC) · doi:10.1109/acc.2012.6315022 · Cited by 239 · see citing works
Self-citationDepression: a decision-theoretic analysis
Huys, Q. J. M., Daw, N. D., & Dayan, P. (2015) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071714-033928 · Cited by 221 · see citing works
Dopamine neurons respond to errors in the prediction of sensory features of expected rewards
Takahashi, Y. K., Batchelor, H. M., Liu, B., Khanna, A., Morales, M., & Schoenbaum, G. (2017) · Neuron · doi:10.1016/j.neuron.2017.08.025 · Cited by 200 · see citing works
Computational Models of Performance Monitoring and Cognitive Control
Alexander, W. H., & Brown, J. W. (2010) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2010.01085.x · Cited by 193 · see citing works
Twenty-Five Lessons from Computational Neuromodulation
Dayan, P. (2012) · Neuron · doi:10.1016/j.neuron.2012.09.027 · Cited by 190 · see citing works
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · see citing works
Wei, C., Zhang, Z., Qiao, W., & Qu, L. (2015) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2015.2420792 · Cited by 178 · see citing works
Cognitive Control: Core Constructs and Current Considerations
Cohen, J. D. (2017) · The Wiley Handbook of Cognitive Control · doi:10.1002/9781118920497.ch1 · Cited by 179 · see citing works
Large Language Models and the Reverse Turing Test
Sejnowski, T. J. (2023) · Neural Computation · doi:10.1162/neco_a_01563 · Cited by 169 · see citing works
Stimulus representation and the timing of reward-prediction errors in models of the dopamine system
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008) · Neural Computation · doi:10.1162/neco.2008.11-07-654 · Cited by 148 · see citing works
Self-citationSpatial goal coding in the hippocampal formation
Nyberg, N., Duvelle, É., Barry, C., & Spiers, H. J. (2022) · Neuron · doi:10.1016/j.neuron.2021.12.012 · Cited by 141 · see citing works
Chapter 19 Dopamine, cognitive control, and schizophrenia: the gating model
Braver, T. S., & Cohen, J. D. (1999) · Progress in Brain Research · doi:10.1016/s0079-6123(08)63082-4 · Cited by 159 · see citing works
Dopamine neurons encoding long-term memory of object value for habitual behavior
Kim, H. F., Ghazizadeh, A., & Hikosaka, O. (2015) · Cell · doi:10.1016/j.cell.2015.10.063 · Cited by 157 · see citing works
The medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty
Starkweather, C. K., Gershman, S. J., & Uchida, N. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.036 · Cited by 136 · see citing works
Hierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129 · see citing works
Iigaya, K., Fonseca, M. S., Murakami, M., Mainen, Z. F., & Dayan, P. (2018) · Nature Communications · doi:10.1038/s41467-018-04840-2 · Cited by 120 · see citing works
Reinforcement learning: Computational theory and biological mechanisms
Doya, K. (2007) · HFSP journal · doi:10.2976/1.2732246 · Cited by 150 · see citing works
Midbrain activation during pavlovian conditioning and delusional symptoms in schizophrenia
Romaniuk, L., Honey, G. D., King, J. R. L., Whalley, H. C., McIntosh, A. M., Levita, L., Hughes, M., Johnstone, E. C., Day, M., Lawrie, S. M., et al. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2010.169 · Cited by 113 · see citing works
Predictive maps in rats and humans for spatial navigation
de Cothi, W., Nyberg, N., Griesbauer, E.-M., Ghanamé, C., Zisch, F., Lefort, J. M., Fletcher, L., Newton, C., Renaudineau, S., Bendor, D., et al. (2022) · Current Biology · doi:10.1016/j.cub.2022.06.090 · Cited by 109 · see citing works
The successor representation and temporal context
Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B. (2012) · Neural Computation · doi:10.1162/neco_a_00282 · Cited by 117 · see citing works
Dynamic shaping of dopamine signals during probabilistic Pavlovian conditioning
Hart, A. S., Clark, J. J., & Phillips, P. E. M. (2015) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2014.07.010 · Cited by 82 · see citing works
Radulescu, A., Shin, Y. S., & Niv, Y. (2021) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-092920-120559 · Cited by 63 · see citing works
Tasks and their role in visual neuroscience
Kay, K., Bonnen, K., Denison, R. N., Arcaro, M. J., & Barack, D. L. (2023) · Neuron · doi:10.1016/j.neuron.2023.03.022 · Cited by 61 · see citing works
LEARNING-BASED MODEL PREDICTIVE CONTROL FOR MARKOV DECISION PROCESSES
Negenborn, R. R., De Schutter, B., Wiering, M. A., & Hellendoorn, H. (2005) · IFAC Proceedings Volumes · doi:10.3182/20050703-6-cz-1902.00280 · Cited by 59 · see citing works
Combining fMRI and behavioral measures to examine the process of human learning
Karuza, E. A., Emberson, L. L., & Aslin, R. N. (2014) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2013.09.012 · Cited by 50 · see citing works
Jhou, T. C. (2021) · Neuropharmacology · doi:10.1016/j.neuropharm.2021.108763 · Cited by 40 · see citing works
Reinforcement Learning with an Ensemble of Binary Action Deep Q-Networks
M. Hafiz, A., Hassaballah, M., Alqahtani, A., Alsubai, S., & Abdel Hameed, M. (2023) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.031720 · Cited by 33 · see citing works
Pauli, W. M., Hazy, T. E., & O'Reilly, R. C. (2012) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00155 · Cited by 32 · see citing works
Sensing flow gradients is necessary for learning autonomous underwater navigation
Jiao, Y., Hang, H., Merel, J., & Kanso, E. (2025) · Nature Communications · doi:10.1038/s41467-025-58125-6 · Cited by 23 · see citing works
Smart Congestion Control for Delay- and Disruption Tolerant Networks
Silva, A. P., Obraczka, K., Burleigh, S., & Hirata, C. M. (2016) · 2016 13th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON) · doi:10.1109/sahcn.2016.7733018 · Cited by 20 · see citing works
Count-based exploration with the successor representation
Machado, M. C., Bellemare, M. G., & Bowling, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5955 · Cited by 19 · see citing works
C-Learning: Learning to Achieve Goals via Recursive Classification
Eysenbach, B., Salakhutdinov, R., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2011.08909 · Cited by 18 · see citing works
Interference and Generalization in Temporal Difference Learning
Bengio, E., Pineau, J., & Precup, D. (2020) · arXiv · doi:10.48550/arxiv.2003.06350 · Cited by 14 · see citing works
Learning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 12 · see citing works
“In addition to learning high-level vision-language policies, we illustrate how the simulator can enable learning low-level control policies by leveraging model-based reinforcement learning (RL) [13].”
BackgroundTD-MPC2: Scalable, Robust World Models for Continuous Control
Hansen, N., Su, H., & Wang, X. (2023) · arXiv · doi:10.48550/arxiv.2310.16828 · Cited by 8 · see citing works
“Specifically, TD-MPC 2 learns an implicit , control-centric world model from environment interaction using a combination of joint-embedding prediction (Grill et al., 2020), reward prediction, and TD-learning (Sutton, 1998), without decoding observations.”
MethodColas, J. T., O’Doherty, J. P., & Grafton, S. T. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011950 · Cited by 6 · see citing works
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- Machine Learning
- Published
- 1988-08
- Volume
- 3
- Issue
- 1
- Pages
- 9-44
- ISSN
- 0885-6125 (ISSN-L)
- 1573-0565
- OpenAlex
- W2100677568
- Cited by
- 5,148
- References
- 40
- Citations by year
- 2026: 164 · 2025: 210 · 2024: 218 · 2023: 230 · 2022: 212 · 2021: 303 · 2020: 284 · 2019: 251 · 2018: 208 · 2017: 141 · 2016: 135 · 2015: 123 · 2014: 124 · 2013: 126 · 2012: 135
- FWCI
- 44.20
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Machine Learning (publishedVersion)
- Funders
- Air Force Office of Scientific Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Keywords
- temporal difference learning, prediction learning, incremental learning, convergence proof, optimality, supervised learning, bucket brigade, credit assignment, memory efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
