The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Dyna, an integrated architecture for learning, planning, and reacting
Abstract
Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1991
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citationCraik, K. J. W. (1943) · Cited by 1,845
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 916
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 583
Learning and Sequential Decision Making
Barto, A. G., Sutton, R. S., & Watkins, C. (1989) · Cited by 349
Why the Law of Effect will not Go Away*
Dennett, D. C. (1975) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1975.tb00350.x · Cited by 330
Research works printed but not linked: 10
- Dennett D. C., 1978, Brainstorms, by D. C. Dennett, 71--89
- Holland J. H., 1986, R. Michalski, J. Carbonell &
- Lin Ji., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, 297--305
- Riolo R., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, MIT Press.
- Russell S. J., 1989, Proceedings of IJCAI-89
- 10.1023/A:1022633531479 · DOI · Paper page
- 10.5555/101883.102055 · DOI
- Sutton R.S., 1981, An adaptive network that constructs and uses an internal model of its environment, Cognition and Brain Theory Quarterly, 4, 217
- Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department. Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department.
- 10.1023/A:1022619109594 · DOI · Paper page
Cited by
Sorted by citations
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
“The policy is, in a sense, a plan, but one that is completely conditioned by current input.”
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
InfluentialNeural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8463189 · Cited by 898
The Option-Critic Architecture
Bacon, P., Harb, J., & Precup, D. (2017) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v31i1.10916 · Cited by 779
“Classical dynamic programming methods such as value iteration and policy iteration [Puterman, 1994] or Dyna planning [Sutton, 1991; Sorg and Singh, 2010] can be used to provide a solution.”
MethodReinforcement Learning and Markov Decision Processes
van Otterlo, M., & Wiering, M. (2012) · Adaptation, learning, and optimization · doi:10.1007/978-3-642-27645-3_1 · Cited by 672
Survey of Model-Based Reinforcement Learning: Applications on Robotics
Polydoros, A., & Nalpantidis, L. (2017) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-017-0468-y · Cited by 560
“In [14] the authors are using the Dyna-Q algorithm, which combines the Q-learning [54] with the Dyna framework [55] that integrates model learning, value estimation and the agent’s reacting.”
MethodModel-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000086 · Cited by 871
Model learning for robot control: A survey
Nguyen-Tuong, D., & Peters, J. (2011) · Cognitive Processing · doi:10.1007/s10339-011-0404-1 · Cited by 503
“Forward models have also been essential in model based reinforcement learning approaches, which relate to the problem of optimal control [156,9, 103].”
BackgroundMethodReinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446
“This hybrid architecture was originally proposed in the machine learning literature by Sutton (1991), who referred to it as Dyna.”
MethodApplications of reinforcement learning in energy systems
Perera, A. T. D., & Kamalaruban, P. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2020.110618 · Cited by 457
“The Dyna algorithm [42] alternates between learning the model based on the gathered data that executes the current policy on the environment and improving the policy with imaginary data from the learned model.”
BackgroundMethodThe successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 408
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
Prudencio, R. F., Máximo, M. R. O. A., & Colombini, E. L. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3250269 · Cited by 460
“With these modified reward functions in hand, we can then use classic model-based approaches to tackle offline RL problems, such as Dyna-based methods that sample transitions from the model to train a model-free algorithm [19], [34].”
MethodPredictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M., Gershman, S. J., & Daw, N. D. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 335
MOPO: Model-based Offline Policy OptimizationPreprint
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., & Ma, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13239 · Cited by 406
A review of reinforcement learning for autonomous building energy management
Mason, K., & Grijalva, S. (2019) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2019.07.019 · Cited by 313
Ganesh, A. H., & Xu, B. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.111833 · Cited by 295
Dopamine Neurons Can Represent Context-Dependent Prediction Error
Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004) · Neuron · doi:10.1016/s0896-6273(03)00869-9 · Cited by 276
“One may wonder how this contextual TD model behaves in the noncontex- (Sutton, 1991; Sutton and Barto, 1998) suggested that the internal model of the environment and the hidden tual task.”
Model-Ensemble Trust-Region Policy OptimizationPreprint
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10592 · Cited by 405
“The trained model can be used as a simulator in which the policy can be trained, and also provides gradient information (Sutton, 1990; Deisenroth & Rasmussen, 2011; Depeweg et al., 2017; Sutton, 1991).”
MethodBayesian Reinforcement Learning: A Survey
Ghavamzadeh, M., Mannor, S., Pineau, J., & Tamar, A. (2015) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000049 · Cited by 347
“Some examples of these methods are the Dyna architecture [Sutton, 1991] and prioritized sweeping [Moore and Atkeson, 1993].”
Cushman, F. A. (2019) · Behavioral and Brain Sciences · doi:10.1017/s0140525x19001730 · Cited by 282
Reinforcement Learning Based EV Charging Management Systems–A Review
Abdullah, H. M., Gastli, A., & Ben‐Brahim, L. (2021) · IEEE Access · doi:10.1109/access.2021.3064354 · Cited by 249
“The process is repeated until the termination state or time step has reached its limit [37].”
BackgroundMethodInfluentialHu, X., Liu, T., Qi, X., & Barth, M. (2019) · IEEE Industrial Electronics Magazine · doi:10.1109/mie.2019.2913015 · Cited by 240
“The first is “simplex algorithms,” meaning that only a single algorithm is used to derive the energy-management policy, such as the Q-learning, Dyna, and Sarsa algorithms [8].”
MethodDepression: a decision-theoretic analysis
Huys, Q. J. M., Daw, N. D., & Dayan, P. (2015) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071714-033928 · Cited by 221
A comparison of direct and model-based reinforcement learning
Atkeson, C. G., & Santamaría, J. C. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1997.606886 · Cited by 239
Reinforcement Learning-Based NOMA Power Allocation in the Presence of Smart Jamming
Xiao, L., Li, Y., Dai, C., Dai, H., & Poor, H. V. (2017) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2017.2782726 · Cited by 230
“Meanwhile, the Dyna architecture, which formulates a learned world model from real experience, can be used to increase the learning speed to derive the optimal policy [15].”
MethodTowards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224
Reminders of past choices bias decisions for reward in humans
Bornstein, A. M., Khaw, M. W., Shohamy, D., & Daw, N. D. (2017) · Nature Communications · doi:10.1038/ncomms15958 · Cited by 231
“In this way, rather than computing sampled values separately, direct and sampled experiences might be mixed freely, buffered through a single prediction error signal and net value store [37, 38].”
BackgroundRobot learning towards smart robotic manufacturing: A review
Liu, Z., Liu, Q., Xu, W., Wang, L., & Zhou, Z. (2022) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2022.102360 · Cited by 251
Solving Semi-Markov Decision Problems Using Average Reward Reinforcement Learning
Das, T. K., Gosavi, A., Mahadevan, S., & Marchalleck, N. (1999) · Management Science · doi:10.1287/mnsc.45.4.560 · Cited by 223
Deep reinforcement learning based energy management for a hybrid electric vehicle
Du, G., Zou, Y., Zhang, X., Liu, T., Wu, J., & He, D. (2020) · Energy · doi:10.1016/j.energy.2020.117591 · Cited by 214
Liu, T., Hu, X., Hu, W., & Zou, Y. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2903098 · Cited by 209
“95]; CL1 , CL2 , and BK1 {0, TABLE II PSEUDOCODE OF THE DYNA ALGORITHM [37]”
BackgroundMethodInfluentialSelf-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation
Kahn, G., Villaflor, A., Ding, B., Abbeel, P., & Levine, S. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460655 · Cited by 308
Fast Sequences of Non-spatial State Representations in Humans
Kurth-Nelson, Z. L., Economides, M., Dolan, R. J., & Dayan, P. (2016) · Neuron · doi:10.1016/j.neuron.2016.05.028 · Cited by 206
“Another possible intermediate function is offline planning (Shohamy and Daw, 2015; Sutton, 1991), where values of paths or sub-paths might be calculated and stored in anticipation of future use.”
A Deeper Look at Experience ReplayPreprint
Zhang, S., & Sutton, R. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.01275 · Cited by 193
“However, the key difference is that Dyna only samples states , while experience replay samples full transitions , which may be biased and potentially harmful.”
MethodExperience replay is associated with efficient nonlocal learning
Liu, Y., Mattar, M. G., Behrens, T. E. J., Daw, N. D., & Dolan, R. J. (2021) · Science · doi:10.1126/science.abf1357 · Cited by 204
“These covert trajectories can substitute for direct experience and thereby span the gaps between actions and outcomes (11), a process known as experience replay.”
BackgroundLearning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159
Neuronal Computation Underlying Inferential Reasoning in Humans and Mice
Barron, H. C., Reeve, H. M., Koolschijn, R. S., Perestenko, P. V., Shpektor, A., Nili, H., Rothaermel, R., Campo-Urriza, N., O’Reilly, J. X., Bannerman, D. M., et al. (2020) · Cell · doi:10.1016/j.cell.2020.08.035 · Cited by 184
“One candidate mechanism involves using multiple memories to internally simulate and cache statistics of the environment (Sutton, 1991).”
BackgroundAnticipatory Behavior in Adaptive Learning Systems
Butz, M. V., Sigaud, O., Pezzulo, G., & Baldassarre, G. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74262-3 · Cited by 178
Deep learning in computational mechanics: a review
Herrmann, L., & Kollmannsberger, S. (2024) · Computational Mechanics · doi:10.1007/s00466-023-02434-4 · Cited by 179
Spatial goal coding in the hippocampal formation
Nyberg, N., Duvelle, É., Barry, C., & Spiers, H. J. (2022) · Neuron · doi:10.1016/j.neuron.2021.12.012 · Cited by 141
Stochastic latent actor–critic: Deep reinforcement learning with a latent variable model
Lee, A. X., Nagabandi, A., Abbeel, P., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1907.00953 · Cited by 159
A survey on model-based reinforcement learning
Luo, F.-M., Xu, T., Lai, H., Chen, X., Zhang, W., & Yu, Y. (2024) · Science China Information Sciences · doi:10.1007/s11432-022-3696-5 · Cited by 205
“Dynamic programming [Sutton and Barto, 2018], tabular Dyna [Sutton, 1990b, 1991], and prioritized sweeping [Moore and Atkeson, 1993] all belong to background planning methods.”
BackgroundMastering diverse control tasks through world models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2025) · Nature · doi:10.1038/s41586-025-08744-2 · Cited by 189
A Review on Data-Driven Constitutive Laws for Solids
Fuhg, J. N., Padmanabha, G. A., Bouklas, N., Bahmani, B., Sun, W., Vlassis, N. N., Flaschel, M., Carrara, P., & Lorenzis, L. D. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10196-2 · Cited by 170
“On the other hand, model-based RL algorithms such as Dyna [55] and AlphaGo Zero [56] depend on planning based on the learned model.”
A Bayesian formulation of behavioral control
Huys, Q. J. M., & Dayan, P. (2009) · Cognition · doi:10.1016/j.cognition.2009.01.008 · Cited by 150
“In reinforcement learning, it is called an exploration bonus (Dayan & Sejnowski, 1996; Sutton, 1991).”
BackgroundThe theory of social functions: challenges for computational social science and multi-agent learning
Castelfranchi, C. (2001) · Cognitive Systems Research · doi:10.1016/s1389-0417(01)00013-4 · Cited by 144
Integrating memories to guide decisions
Shohamy, D., & Daw, N. D. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.08.010 · Cited by 155
“In machine learning, such replay-driven learning has been proposed in an architecture called DYNA [45] , which demonstrates how actual, remembered, and model-simulated experiences can be blended together for learning.”
BackgroundFreezing revisited: coordinated autonomic and central optimization of threat coping
Roelofs, K., & Dayan, P. (2022) · Nature reviews. Neuroscience · doi:10.1038/s41583-022-00608-2 · Cited by 160
Offline replay supports planning in human reinforcement learning
Momennejad, I., Otto, A. R., Daw, N. D., & Norman, K. A. (2018) · eLife · doi:10.7554/elife.32548 · Cited by 149
Deep Reinforcement Learning for Soft, Flexible Robots: Brief Review with Impending Challenges
Bhagat, S., Banerjee, H., Tse, Z. T. H., & Ren, H. (2019) · Robotics · doi:10.3390/robotics8010004 · Cited by 139
COMBO: Conservative Offline Model-Based Policy OptimizationPreprint
Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., & Finn, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08363 · Cited by 179
“While any RL or planning algorithm can be used to learn the optimal policy for (cid:99) M , we focus specifically on MBPO [20, 57] which was used in MOPO.”
MethodInfluentialAlgorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical GuaranteesPreprint
Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., & Ma, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03858 · Cited by 190
“…et al. (2018) also study the discrepancy bounds under Lipschitz condition of the MDP. Prior work explores a variety of ways of combining model-free and model-based ideas to achieve the best of the two methods (Sutton, 1991; 1990; Racanière et al., 2017; Mordatch et al., 2016; Sun et al., 2018).”
MethodPlanning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129
“World models summarize past experience into a representation of the environment that enables predicting imagined future sequences (Sutton, 1991; Watter et al., 2015; Ha & Schmidhuber, 2018).”
BackgroundThe temporal dynamics of opportunity costs: A normative account of cognitive fatigue and boredom.
Agrawal, M., Mattar, M. G., Cohen, J. D., & Daw, N. D. (2021) · Psychological Review · doi:10.1037/rev0000309 · Cited by 134
Robot Learning From Randomized Simulations: A Review
Muratore, F., Ramos, F., Turk, G., Yu, W., Gienger, M., & Peters, J. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.799893 · Cited by 128
“By describing the integration policy optimization via model-free RL, the authors created an online variant of the original method.”
BackgroundInfluentialA Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials
Chatzilygeroudis, K., Vassiliades, V., Stulp, F., Calinon, S., & Mouret, J.-B. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2958211 · Cited by 160
“This idea leads to model-based policy search algorithms [39, 120], in which the trajectory data are used to learn the dynamics model, then policy search is performed on the model [71, 144].”
MethodRapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
Allen, K. R., Smith, K. A., & Tenenbaum, J. B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912341117 · Cited by 144
“SSUP is inspired by the theory of “problem solving as search” (Newell & Simon, 1972), as well as Dyna and other model-based policy optimization methods (Deisenroth, Neumann, Peters, et al., 2013; Sutton, 1991).”
MethodVenkatasatish, R., & Dhanamjayulu, C. (2022) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2022.06.088 · Cited by 123
Varieties of Helmholtz Machine
Dayan, P., & Hinton, G. E. (1996) · Neural Networks · doi:10.1016/s0893-6080(96)00009-3 · Cited by 120
Mattar, M. G., & Lengyel, M. (2022) · Neuron · doi:10.1016/j.neuron.2021.12.018 · Cited by 120
“This 8 Neuron 110, March 16, 2022 strategy is particularly fruitful in stochastic environments, where algorithms can improve the agent’s policy by expanding a large number of such single-step trees, starting from all possible states (as in dynamic programming algorithms, such as value iteration or policy iteration) or…”
Zhao, X., Xia, L., Tang, J., & Yin, D. (2019) · ACM SIGWEB Newsletter · doi:10.1145/3320496.3320500 · Cited by 121
“These methods include popular algorithms such as the Dyna [Sutton 1991], Prioritized Sweeping [Moore and Atkeson 1993], Q -iteration [Buso-niu et al. 2010], Policy Gradient (PG) [Williams 1992], and the variation of PG [Baxter and Bartlett 2001; Kakade 2001].”
MethodZou, L., Xia, L., Du, P., Zhang, Z., Bai, T., Liu, W., Nie, J., & Yin, D. (2020) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3336191.3371801 · Cited by 107
“To these ends, our main contributions in this work are as follows: • We present Pseudo Dyna-Q (PDQ) for interactive recommendation, which provides a general framework that can be instantiated in different neural architectures, and tailored for specific recommendation tasks.”
MethodInfluentialHossain, M. A., Noor, R. M., Yau, K. A., Azzuhri, S. R., Z’aba, M. R., & Ahmedy, I. (2020) · IEEE Access · doi:10.1109/access.2020.2989870 · Cited by 111
Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey
Majid, A. Y., Majid, A. Y., Saaybi, S., van Rietbergen, T., François-Lavet, V., Prasad, R. R. V., & Verhoeven, C. J. M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3264540 · Cited by 124
“Dyna-Q [14] is a classical model-based RL algorithm.”
BackgroundMethodLee, H., Kim, K.-H., Kim, N., & Won, S. (2022) · Applied Energy · doi:10.1016/j.apenergy.2021.118460 · Cited by 103
Adaptive Experience Engine for Serious Games
Bellotti, F., Berta, R., Gloria, A. D., & Primavera, L. (2009) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2009.2035923 · Cited by 106
“• It allows domain experts and pedagogical authors to focus on content development and learning strategy specification, without asking them to do (or deeply interact with a software development team that does) game scripting, which typically involves hard-coding events, actions, locations, and…”
BackgroundKoenig, S., & Simmons, R. G. (1996) · Machine Learning · doi:10.1007/bf00114729 · Cited by 105
“Sutton's DYNA architecture ( Sutton, 1990 and 1991 ), for example, implements planning in the framework of Q- or Q-learning.”
MethodAn Improved Dyna-Q Algorithm for Mobile Robot Path Planning in Unknown Dynamic Environment
Pei, M., An, H., Liu, B., & Wang, C. (2021) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2021.3096935 · Cited by 104
“Hence, a model-based architecture, Dyna [20], has been extended to the classical 2168-2216 c (cid:2) 2021 IEEE.”
How to set the switches on this thing
Dayan, P. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.05.011 · Cited by 107
“This reason can in fact be criticized [31] on the grounds that curiosity is actually a perfectly normal facet of control in the face of uncertainty — rewards associated with it are forms of bonus [32,33] that allow agents to trade exploration of potential opportunities afforded by a partially known environment against…”
BackgroundDecoding cognition from spontaneous neural activity
Liu, Y., Nour, M. M., Schuck, N. W., Behrens, T. E. J., & Dolan, R. J. (2022) · Nature reviews. Neuroscience · doi:10.1038/s41583-022-00570-z · Cited by 107
Mastering Diverse Domains through World Models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023) · arXiv · doi:10.48550/arxiv.2301.04104 · Cited by 124
“The algorithm is based on the idea of learning a world model that equips the agent with rich perception and the ability to imagine the future 14,15,16 .”
BackgroundMethodReplay and compositional computation
Kurth-Nelson, Z. L., Behrens, T. E. J., Wayne, G., Miller, K. J., Luettgau, L., Dolan, R. J., Liu, Y., & Schwartenbeck, P. (2023) · Neuron · doi:10.1016/j.neuron.2022.12.028 · Cited by 100
IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks
Lee, Y., Hu, E. S., & Lim, J. J. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560986 · Cited by 102
“re assembly [42, 18]. Model-based reinforcement learning could be another direction for tackling the long-horizon problem by giving the model a mechanism to predict the future before taking an action [43]. Dexterous Control: Dexterous manipulation is challenging due to agent and object complexity. Real world objects h…”
BackgroundDiscovery of hierarchical representations for efficient planning
Tomov, M. S., Yagati, S., Kumar, A., Yang, W., & Gershman, S. J. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007594 · Cited by 100
Goal-directed control and its antipodes
Dayan, P. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.03.004 · Cited by 93
“…(‘dreamt’) by the generative model can be used during times (notably sleeping, grooming and eating for rats, Diba & Buzsáki, 2007; Foster & Wilson, 2006; Lee & Wilson, 2002; Louie &Wilson, 2001; Skaggs &Mcnaughton, 1996) that active control is suppressed (Hinton et al., 1995; Sutton, 1991).”
Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
Parker-Holder, J., Rajan, R., Song, X., Biedenkapp, A., Miao, Y., Eimer, T., Zhang, B., Nguyen, V., Calandra, R., Faust, A., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13596 · Cited by 114
“In contrast, Dyna (Sutton, 1991) is a canonical MBRL algorithm that essentially treats a learned dynamics model as an environment for model-free RL. Dyna-style approaches are popular since they facilitate faster inference at deployment time (forward-pass of the policy rather than an optimization loop) and have been sh…”
BackgroundInfluentialHierarchical Action Control: Adaptive Collaboration Between Actions and Habits
Balleine, B. W., & Dezfouli, A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02735 · Cited by 98
“For example, one collaborative view, Dyna (Sutton, 1991; see also Gershman et al., 2014; Momennejad et al., 2018), proposes that model-based replay can train the model-free system; a suggestion that devolves to something like rehearsal or perhaps consolidation.”
BackgroundLearning locomotion over rough terrain using terrain templates
Kalakrishnan, M., Buchli, J., Pástor, P., & Schaal, S. (2009) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2009.5354701 · Cited by 90
Q-Learning with Linear Function Approximation
Melo, F. S., & Ribeiro, I. (2007) · Springer eBooks · doi:10.1007/978-3-540-72927-3_23 · Cited by 96
“Value-based methods such as TD-learning [1], Q-learning [2], SARSA [3] and others [4,5,6] have been exhaustively covered in the reinforcement learning (RL) literature and, under mild assumptions, have been proven to converge to the desired solution [7].”
MethodOccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
Zheng, W., Chen, W., Huang, Y., Zhang, B., Duan, Y., & Lu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72624-8_4 · Cited by 152
Xu, J. L., Hou, Z., Liu, Z., & Qiao, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05240 · Cited by 87
“Additionally, some researchers have focused on exploring how to make full use of the learned dynamic model in addition to the commonly used Dyna architecture [97] and GPS-based methods [26] in order to simulate the entire trajectory every iteration.”
MethodAn object-oriented representation for efficient reinforcement learning
Diuk, C., Cohen, A., & Littman, M. L. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390187 · Cited by 226
Exploration of Multi-State Environments: Local Measures and Back-Propagation of Uncertainty
Meuleau, N., & Bourgine, P. (1999) · Machine Learning · doi:10.1023/a:1007541107674 · Cited by 91
“It has already been used by Sutton (1990) to deal with non-stationarity in an indirect algorithm, and mentioned by Thrun (1992a, 1992b).”
BackgroundMethodInfluentialExploring Model-based Planning with Policy NetworksPreprint
Wang, T., & Ba, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08649 · Cited by 131
“Dyna [38, 39] algorithm alternately performs sampling in the real environments and optimize the controllers on the learned model of the environments.”
BackgroundHuys, Q. J. M., Guitart‐Masip, M., Dolan, R. J., & Dayan, P. (2015) · Clinical Psychological Science · doi:10.1177/2167702614562040 · Cited by 85
“This follows a venerable computational suggestion (Sutton, 1991) for improving the performance of model-free control.”
BackgroundAutonomous robotic nanofabrication with reinforcement learning
Leinen, P., Esders, M., Schütt, K. T., Wagner, C., Müller, K.-R., & Tautz, F. S. (2020) · Science Advances · doi:10.1126/sciadv.abb6987 · Cited by 89
Model Based Reinforcement Learning for Atari
Kaiser, Ł., Babaeizadeh, M., Miłoś, P., Osiński, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. (2020) · International Conference on Learning Representations · Cited by 75
Inventory management of new products in retailers using model-based deep reinforcement learning
Demizu, T., Fukazawa, Y., & Morita, H. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120256 · Cited by 76
Natural communication with robots
Torrance, M. (1994) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 76
World Models for Autonomous Driving: An Initial Survey
Guan, Y., Liao, H., Li, Z., Hu, J., Yuan, R., Zhang, G., & Xu, C. (2025) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3398357 · Cited by 76
AoI-minimal UAV Crowdsensing by Model-based Graph Convolutional Reinforcement Learning
Dai, Z., Liu, C. H., Ye, Y., Han, R., Yuan, Y., Wang, G., & Tang, J. (2022) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom48880.2022.9796732 · Cited by 78
Anxiety, Avoidance, and Sequential Evaluation
Zorowitz, S., Momennejad, I., & Daw, N. D. (2020) · Computational Psychiatry · doi:10.1162/cpsy_a_00026 · Cited by 75
A multifaceted vision of the human-AI collaboration: a comprehensive review
Puerta-Beldarrain, M., Gómez-Carmona, O., Sánchez-Corcuera, R., Casado-Mansilla, D., López-de-Ipiña, D., & Chen, L. (2025) · IEEE Access · doi:10.1109/access.2025.3536095 · Cited by 73
“Popular model-free methods include Monte Carlo [123], Temporal Difference Learning [145], Q-learning [146], SARSA [131], and Dyna-Q [147].”
Temporal structure in associative retrieval
Kurth-Nelson, Z. L., Barnes, G. R., Sejdinović, D., Dolan, R. J., & Dayan, P. (2015) · eLife · doi:10.7554/elife.04919 · Cited by 77
“…autonomously, is suggested as central to a wide variety of processes and functions, including online and offline planning and model-learning (Sutton, 1991; Moore and Atkeson, 1993; Foster and Wilson, 2006; Johnson and Redish, 2007; Hasselmo, 2008; Lisman and Redish, 2009; Gupta et al.,…”
BackgroundThe roles of online and offline replay in planning
Eldar, E., Lièvre, G., Dayan, P., & Dolan, R. J. (2020) · eLife · doi:10.7554/elife.56911 · Cited by 75
Information generation as a functional basis of consciousness
Kanai, R., Chang, A., Yu, Y., Magrans de Abril, I., Biehl, M., & Guttenberg, N. (2019) · Neuroscience of Consciousness · doi:10.1093/nc/niz016 · Cited by 72
Learning model-based planning from scratchPreprint
Pascanu, R., Li, Y., Vinyals, O., Heess, N., Buesing, L., Racanière, S., Reichert, D., Weber, T., Wierstra, D., & Battaglia, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06170 · Cited by 112
“The classic “Dyna” algorithm learns a model, which is then used to train a policy [21].”
MethodWeinberg, D., Wang, Q., Timoudas, T. O., & Fischione, C. (2022) · Sustainable Cities and Society · doi:10.1016/j.scs.2022.104351 · Cited by 70
Metadata
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- ACM SIGART Bulletin
- Published
- 1991-07
- Volume
- 2
- Issue
- 4
- Pages
- 160-163
- ISSN
- 0163-5719 (ISSN-L)
- OpenAlex
- W1980035368
- Cited by
- 777
- References
- 18
- Citations by year
- 2026: 56 · 2025: 71 · 2024: 57 · 2023: 63 · 2022: 57 · 2021: 79 · 2020: 115 · 2019: 62 · 2018: 42 · 2017: 25 · 2016: 9 · 2015: 16 · 2014: 5 · 2013: 13 · 2012: 11
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGART Bulletin (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- learning from examples, planning, model-based learning, incremental planning, learning-based planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
