Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Abstract
Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the Gershman et al. design, subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms.
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1991
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · see citing works
Craik, K. J. W. (1943) · Cited by 1,685 · see citing works
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,348 · see citing works
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348 · see citing works
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 916 · see citing works
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 769 · see citing works
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599 · see citing works
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 591 · see citing works
Learning and Sequential Decision Making
Barto, A. G., Sutton, R. S., & Watkins, C. (1989) · Cited by 349 · see citing works
Why the Law of Effect will not Go Away*
Dennett, D. C. (1975) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1975.tb00350.x · Cited by 330 · see citing works
Learning to perceive and act by trial and error
Whitehead, S. D., & Ballard, D. H. (1991) · Machine Learning · doi:10.1007/bf00058926 · Cited by 300 · see citing works
Werbos, P. J. (1987) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1987.289329 · Cited by 249 · see citing works
Learning Sequential Decision Rules Using Simulation Models and Competition
Grefenstette, J. J., Ramsey, C. L., & Schultz, A. C. (1990) · Machine Learning · doi:10.1023/a:1022677607120 · Cited by 233 · see citing works
Holland, J. H. (1986) · Machine Learning · Cited by 121 · see citing works
Lookahead Planning and Latent Learning in a Classifier System
Riolo, R. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/3115.003.0043 · Cited by 87 · see citing works
Self-improving Reactive Agents: Case Studies of Reinforcement Learning Frameworks
Lin, L.-J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/3115.003.0041 · Cited by 52 · see citing works
Execution architectures and compilation
Russell, S. (1989) · International Joint Conference on Artificial Intelligence · Cited by 27 · see citing works
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557 · see citing works
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,348 · see citing works
“The policy is, in a sense, a plan, but one that is completely conditioned by current input.”
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000 · see citing works
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977 · see citing works
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
InfluentialNeural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8463189 · Cited by 898 · see citing works
The Option-Critic Architecture
Bacon, P., Harb, J., & Precup, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10916 · Cited by 219 · see citing works
“Classical dynamic programming methods such as value iteration and policy iteration [Puterman, 1994] or Dyna planning [Sutton, 1991; Sorg and Singh, 2010] can be used to provide a solution.”
MethodReinforcement Learning and Markov Decision Processes
van Otterlo, M., & Wiering, M. (2012) · Adaptation, learning, and optimization · doi:10.1007/978-3-642-27645-3_1 · Cited by 631 · see citing works
Survey of Model-Based Reinforcement Learning: Applications on Robotics
Polydoros, A., & Nalpantidis, L. (2017) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-017-0468-y · Cited by 536 · see citing works
“In [14] the authors are using the Dyna-Q algorithm, which combines the Q-learning [54] with the Dyna framework [55] that integrates model learning, value estimation and the agent’s reacting.”
MethodModel-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000086 · Cited by 668 · see citing works
Model learning for robot control: A survey
Nguyen-Tuong, D., & Peters, J. (2011) · Cognitive Processing · doi:10.1007/s10339-011-0404-1 · Cited by 503 · see citing works
“Forward models have also been essential in model based reinforcement learning approaches, which relate to the problem of optimal control [156,9, 103].”
BackgroundMethodReinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446 · see citing works
“This hybrid architecture was originally proposed in the machine learning literature by Sutton (1991), who referred to it as Dyna.”
MethodApplications of reinforcement learning in energy systems
Perera, A. T. D., & Kamalaruban, P. (2021) · Renewable & Sustainable Energy Reviews · doi:10.1016/j.rser.2020.110618 · Cited by 388 · see citing works
“The Dyna algorithm [42] alternates between learning the model based on the gathered data that executes the current policy on the environment and improving the policy with imaginary data from the learned model.”
BackgroundMethodThe successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 384 · see citing works
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
Prudencio, R. F., Máximo, M. R. O. A., & Colombini, E. L. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3250269 · Cited by 369 · see citing works
“With these modified reward functions in hand, we can then use classic model-based approaches to tackle offline RL problems, such as Dyna-based methods that sample transitions from the model to train a model-free algorithm [19], [34].”
MethodPredictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M., Gershman, S. J., & Daw, N. D. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 321 · see citing works
MOPO: Model-based Offline Policy OptimizationPreprint
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., & Ma, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13239 · Cited by 324 · see citing works
A review of reinforcement learning for autonomous building energy management
Mason, K., & Grijalva, S. (2019) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2019.07.019 · Cited by 306 · see citing works
Ganesh, A. H., & Xu, B. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.111833 · Cited by 290 · see citing works
Dopamine Neurons Can Represent Context-Dependent Prediction Error
Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004) · Neuron · doi:10.1016/s0896-6273(03)00869-9 · Cited by 276 · see citing works
“One may wonder how this contextual TD model behaves in the noncontex- (Sutton, 1991; Sutton and Barto, 1998) suggested that the internal model of the environment and the hidden tual task.”
Model-Ensemble Trust-Region Policy OptimizationPreprint
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10592 · Cited by 276 · see citing works
“The trained model can be used as a simulator in which the policy can be trained, and also provides gradient information (Sutton, 1990; Deisenroth & Rasmussen, 2011; Depeweg et al., 2017; Sutton, 1991).”
MethodBayesian Reinforcement Learning: A Survey
Ghavamzadeh, M., Mannor, S., Pineau, J., & Tamar, A. (2015) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000049 · Cited by 289 · see citing works
“Some examples of these methods are the Dyna architecture [Sutton, 1991] and prioritized sweeping [Moore and Atkeson, 1993].”
Cushman, F. A. (2019) · Behavioral and Brain Sciences · doi:10.1017/s0140525x19001730 · Cited by 268 · see citing works
Reinforcement Learning Based EV Charging Management Systems–A Review
Abdullah, H. M., Gastli, A., & Ben‐Brahim, L. (2021) · IEEE Access · doi:10.1109/access.2021.3064354 · Cited by 249 · see citing works
“The process is repeated until the termination state or time step has reached its limit [37].”
BackgroundMethodInfluentialHu, X., Liu, T., Qi, X., & Barth, M. (2019) · IEEE Industrial Electronics Magazine · doi:10.1109/mie.2019.2913015 · Cited by 240 · see citing works
“The first is “simplex algorithms,” meaning that only a single algorithm is used to derive the energy-management policy, such as the Q-learning, Dyna, and Sarsa algorithms [8].”
MethodDepression: a decision-theoretic analysis
Huys, Q. J. M., Daw, N. D., & Dayan, P. (2015) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071714-033928 · Cited by 221 · see citing works
A comparison of direct and model-based reinforcement learning
Atkeson, C. G., & Santamaría, J. C. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1997.606886 · Cited by 239 · see citing works
Reinforcement Learning-Based NOMA Power Allocation in the Presence of Smart Jamming
Xiao, L., Li, Y., Dai, C., Dai, H., & Poor, H. V. (2017) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2017.2782726 · Cited by 230 · see citing works
“Meanwhile, the Dyna architecture, which formulates a learned world model from real experience, can be used to increase the learning speed to derive the optimal policy [15].”
MethodTowards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
Reminders of past choices bias decisions for reward in humans
Bornstein, A. M., Khaw, M. W., Daphna Shohamy, A., & Daw, N. D. (2017) · Nature Communications · doi:10.1038/ncomms15958 · Cited by 222 · see citing works
“In this way, rather than computing sampled values separately, direct and sampled experiences might be mixed freely, buffered through a single prediction error signal and net value store [37, 38].”
BackgroundRobot learning towards smart robotic manufacturing: A review
Liu, Z., Liu, Q., Xu, W., Wang, L., & Zhou, Z. (2022) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2022.102360 · Cited by 233 · see citing works
Solving Semi-Markov Decision Problems Using Average Reward Reinforcement Learning
Das, T. K., Gosavi, A. A., Mahadevan, S., & Marchalleck, N. (1999) · Management Science · doi:10.1287/mnsc.45.4.560 · Cited by 221 · see citing works
Deep reinforcement learning based energy management for a hybrid electric vehicle
Du, G., Zou, Y., Zhang, X., Liu, T., Wu, J., & He, D. (2020) · Energy · doi:10.1016/j.energy.2020.117591 · Cited by 214 · see citing works
Liu, T., Hu, X., Hu, W., & Zou, Y. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2903098 · Cited by 209 · see citing works
“95]; CL1 , CL2 , and BK1 {0, TABLE II PSEUDOCODE OF THE DYNA ALGORITHM [37]”
BackgroundMethodInfluentialSelf-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation
Kahn, G., Villaflor, A., Ding, B., Abbeel, P., & Levine, S. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460655 · Cited by 308 · see citing works
Fast Sequences of Non-spatial State Representations in Humans
Kurth-Nelson, Z. L., Economides, M., Dolan, R. J., & Dayan, P. (2016) · Neuron · doi:10.1016/j.neuron.2016.05.028 · Cited by 206 · see citing works
“Another possible intermediate function is offline planning (Shohamy and Daw, 2015; Sutton, 1991), where values of paths or sub-paths might be calculated and stored in anticipation of future use.”
A Deeper Look at Experience ReplayPreprint
Zhang, S., & Sutton, R. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.01275 · Cited by 193 · see citing works
“However, the key difference is that Dyna only samples states , while experience replay samples full transitions , which may be biased and potentially harmful.”
MethodExperience replay is associated with efficient nonlocal learning
Liu, Y., Mattar, M. G., Behrens, T. E. J., Daw, N. D., & Dolan, R. J. (2021) · Science · doi:10.1126/science.abf1357 · Cited by 187 · see citing works
“These covert trajectories can substitute for direct experience and thereby span the gaps between actions and outcomes (11), a process known as experience replay.”
BackgroundLearning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159 · see citing works
Neuronal Computation Underlying Inferential Reasoning in Humans and Mice
Barron, H. C., Reeve, H. M., Koolschijn, R. S., Perestenko, P. V., Shpektor, A., Nili, H., Rothaermel, R., Campo-Urriza, N., O’Reilly, J. X., Bannerman, D. M., et al. (2020) · Cell · doi:10.1016/j.cell.2020.08.035 · Cited by 179 · see citing works
“One candidate mechanism involves using multiple memories to internally simulate and cache statistics of the environment (Sutton, 1991).”
BackgroundAnticipatory Behavior in Adaptive Learning Systems
Butz, M. V., Sigaud, O., Pezzulo, G., & Baldassarre, G. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74262-3 · Cited by 178 · see citing works
Deep learning in computational mechanics: a review
Herrmann, L., & Kollmannsberger, S. (2024) · Computational Mechanics · doi:10.1007/s00466-023-02434-4 · Cited by 169 · see citing works
Spatial goal coding in the hippocampal formation
Nyberg, N., Duvelle, É., Barry, C., & Spiers, H. J. (2022) · Neuron · doi:10.1016/j.neuron.2021.12.012 · Cited by 141 · see citing works
Stochastic latent actor–critic: Deep reinforcement learning with a latent variable model
Lee, A. X., Nagabandi, A., Abbeel, P., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1907.00953 · Cited by 159 · see citing works
A survey on model-based reinforcement learning
Luo, F.-M., Xu, T., Lai, H., Chen, X., Zhang, W., & Yu, Y. (2024) · Science China Information Sciences · doi:10.1007/s11432-022-3696-5 · Cited by 173 · see citing works
“Dynamic programming [Sutton and Barto, 2018], tabular Dyna [Sutton, 1990b, 1991], and prioritized sweeping [Moore and Atkeson, 1993] all belong to background planning methods.”
BackgroundMastering diverse control tasks through world models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2025) · Nature · doi:10.1038/s41586-025-08744-2 · Cited by 189 · see citing works
A Review on Data-Driven Constitutive Laws for Solids
Fuhg, J. N., Padmanabha, G. A., Bouklas, N., Bahmani, B., Sun, W., Vlassis, N. N., Flaschel, M., Carrara, P., & Lorenzis, L. D. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10196-2 · Cited by 151 · see citing works
“On the other hand, model-based RL algorithms such as Dyna [55] and AlphaGo Zero [56] depend on planning based on the learned model.”
A Bayesian formulation of behavioral control
Huys, Q. J. M., & Dayan, P. (2009) · Cognition · doi:10.1016/j.cognition.2009.01.008 · Cited by 144 · see citing works
“In reinforcement learning, it is called an exploration bonus (Dayan & Sejnowski, 1996; Sutton, 1991).”
BackgroundThe theory of social functions: challenges for computational social science and multi-agent learning
Castelfranchi, C. (2001) · Cognitive Systems Research · doi:10.1016/s1389-0417(01)00013-4 · Cited by 144 · see citing works
Integrating memories to guide decisions
Daphna Shohamy, A., & Daw, N. D. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.08.010 · Cited by 141 · see citing works
“In machine learning, such replay-driven learning has been proposed in an architecture called DYNA [45] , which demonstrates how actual, remembered, and model-simulated experiences can be blended together for learning.”
BackgroundFreezing revisited: coordinated autonomic and central optimization of threat coping
Karin Roelofs, A., & Dayan, P. (2022) · Nature reviews. Neuroscience · doi:10.1038/s41583-022-00608-2 · Cited by 141 · see citing works
Offline replay supports planning in human reinforcement learning
Momennejad, I., Otto, A. R., Daw, N. D., & Norman, K. A. (2018) · eLife · doi:10.7554/elife.32548 · Cited by 133 · see citing works
Deep Reinforcement Learning for Soft, Flexible Robots: Brief Review with Impending Challenges
Bhagat, S., Banerjee, H., Tse, Z. T. H., & Ren, H. (2019) · Robotics · doi:10.3390/robotics8010004 · Cited by 134 · see citing works
COMBO: Conservative Offline Model-Based Policy OptimizationPreprint
Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., & Finn, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08363 · Cited by 132 · see citing works
“While any RL or planning algorithm can be used to learn the optimal policy for (cid:99) M , we focus specifically on MBPO [20, 57] which was used in MOPO.”
MethodInfluentialAlgorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical GuaranteesPreprint
Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., & Ma, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03858 · Cited by 145 · see citing works
“…et al. (2018) also study the discrepancy bounds under Lipschitz condition of the MDP. Prior work explores a variety of ways of combining model-free and model-based ideas to achieve the best of the two methods (Sutton, 1991; 1990; Racanière et al., 2017; Mordatch et al., 2016; Sun et al., 2018).”
MethodPlanning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129 · see citing works
“World models summarize past experience into a representation of the environment that enables predicting imagined future sequences (Sutton, 1991; Watter et al., 2015; Ha & Schmidhuber, 2018).”
BackgroundThe temporal dynamics of opportunity costs: A normative account of cognitive fatigue and boredom.
Agrawal, M., Mattar, M. G., Cohen, J. D., & Daw, N. D. (2021) · Psychological Review · doi:10.1037/rev0000309 · Cited by 131 · see citing works
Robot Learning From Randomized Simulations: A Review
Muratore, F., Ramos, F., Turk, G., Yu, W., Gienger, M., & Peters, J. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.799893 · Cited by 128 · see citing works
“By describing the integration policy optimization via model-free RL, the authors created an online variant of the original method.”
BackgroundInfluentialA Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials
Chatzilygeroudis, K., Vassiliades, V., Stulp, F., Calinon, S., & Mouret, J.-B. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2958211 · Cited by 160 · see citing works
“This idea leads to model-based policy search algorithms [39, 120], in which the trajectory data are used to learn the dynamics model, then policy search is performed on the model [71, 144].”
MethodRapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
Allen, K. R., Smith, K. A., & Tenenbaum, J. B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912341117 · Cited by 120 · see citing works
“SSUP is inspired by the theory of “problem solving as search” (Newell & Simon, 1972), as well as Dyna and other model-based policy optimization methods (Deisenroth, Neumann, Peters, et al., 2013; Sutton, 1991).”
MethodVenkatasatish, R., & Dhanamjayulu, C. (2022) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2022.06.088 · Cited by 121 · see citing works
Varieties of Helmholtz Machine
Dayan, P., & Hinton, G. E. (1996) · Neural Networks · doi:10.1016/s0893-6080(96)00009-3 · Cited by 121 · see citing works
Mattar, M. G., & Lengyel, M. (2022) · Neuron · doi:10.1016/j.neuron.2021.12.018 · Cited by 113 · see citing works
“This 8 Neuron 110, March 16, 2022 strategy is particularly fruitful in stochastic environments, where algorithms can improve the agent’s policy by expanding a large number of such single-step trees, starting from all possible states (as in dynamic programming algorithms, such as value iteration or policy iteration) or…”
Zhao, X., Xia, L., Tang, J., & Yin, D. (2019) · ACM SIGWEB Newsletter · doi:10.1145/3320496.3320500 · Cited by 106 · see citing works
“These methods include popular algorithms such as the Dyna [Sutton 1991], Prioritized Sweeping [Moore and Atkeson 1993], Q -iteration [Buso-niu et al. 2010], Policy Gradient (PG) [Williams 1992], and the variation of PG [Baxter and Bartlett 2001; Kakade 2001].”
MethodZou, L., Xia, L., Du, P., Zhang, Z., Bai, T., Liu, W., Nie, J., & Yin, D. (2020) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3336191.3371801 · Cited by 105 · see citing works
“To these ends, our main contributions in this work are as follows: • We present Pseudo Dyna-Q (PDQ) for interactive recommendation, which provides a general framework that can be instantiated in different neural architectures, and tailored for specific recommendation tasks.”
MethodInfluentialHossain, M. A., Noor, R. M., Yau, K. A., Azzuhri, S. R., Z’aba, M. R., & Ahmedy, I. (2020) · IEEE Access · doi:10.1109/access.2020.2989870 · Cited by 105 · see citing works
Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey
Majid, A. Y., Majid, A. Y., Saaybi, S., van Rietbergen, T., François-Lavet, V., Prasad, R. R. V., & Verhoeven, C. J. M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3264540 · Cited by 108 · see citing works
“Dyna-Q [14] is a classical model-based RL algorithm.”
BackgroundMethodLee, H., Kim, K.-H., Kim, N., & Won, S. (2022) · Applied Energy · doi:10.1016/j.apenergy.2021.118460 · Cited by 103 · see citing works
Adaptive Experience Engine for Serious Games
Bellotti, F., Berta, R., Gloria, A. D., & Primavera, L. (2009) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2009.2035923 · Cited by 103 · see citing works
“• It allows domain experts and pedagogical authors to focus on content development and learning strategy specification, without asking them to do (or deeply interact with a software development team that does) game scripting, which typically involves hard-coding events, actions, locations, and…”
BackgroundKoenig, S., & Simmons, R. G. (1996) · Machine Learning · doi:10.1007/bf00114729 · Cited by 102 · see citing works
“Sutton's DYNA architecture ( Sutton, 1990 and 1991 ), for example, implements planning in the framework of Q- or Q-learning.”
MethodAn Improved Dyna-Q Algorithm for Mobile Robot Path Planning in Unknown Dynamic Environment
Pei, M., An, H., Liu, B., & Wang, C. (2021) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2021.3096935 · Cited by 101 · see citing works
“Hence, a model-based architecture, Dyna [20], has been extended to the classical 2168-2216 c (cid:2) 2021 IEEE.”
How to set the switches on this thing
Dayan, P. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.05.011 · Cited by 99 · see citing works
“This reason can in fact be criticized [31] on the grounds that curiosity is actually a perfectly normal facet of control in the face of uncertainty — rewards associated with it are forms of bonus [32,33] that allow agents to trade exploration of potential opportunities afforded by a partially known environment against…”
BackgroundDecoding cognition from spontaneous neural activity
Liu, Y., Nour, M. M., Schuck, N. W., Behrens, T. E. J., & Dolan, R. J. (2022) · Nature reviews. Neuroscience · doi:10.1038/s41583-022-00570-z · Cited by 100 · see citing works
Mastering Diverse Domains through World Models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023) · arXiv · doi:10.48550/arxiv.2301.04104 · Cited by 124 · see citing works
“The algorithm is based on the idea of learning a world model that equips the agent with rich perception and the ability to imagine the future 14,15,16 .”
BackgroundMethodReplay and compositional computation
Kurth-Nelson, Z. L., Behrens, T. E. J., Wayne, G., Miller, K. J., Luettgau, L., Dolan, R. J., Liu, Y., & Schwartenbeck, P. (2023) · Neuron · doi:10.1016/j.neuron.2022.12.028 · Cited by 93 · see citing works
IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks
Lee, Y., Hu, E. S., & Lim, J. J. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560986 · Cited by 102 · see citing works
“re assembly [42, 18]. Model-based reinforcement learning could be another direction for tackling the long-horizon problem by giving the model a mechanism to predict the future before taking an action [43]. Dexterous Control: Dexterous manipulation is challenging due to agent and object complexity. Real world objects h…”
BackgroundDiscovery of hierarchical representations for efficient planning
Tomov, M. S., Yagati, S., Kumar, A., Yang, W., & Gershman, S. J. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007594 · Cited by 97 · see citing works
Goal-directed control and its antipodes
Dayan, P. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.03.004 · Cited by 93 · see citing works
“…(‘dreamt’) by the generative model can be used during times (notably sleeping, grooming and eating for rats, Diba & Buzsáki, 2007; Foster & Wilson, 2006; Lee & Wilson, 2002; Louie &Wilson, 2001; Skaggs &Mcnaughton, 1996) that active control is suppressed (Hinton et al., 1995; Sutton, 1991).”
Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
Parker-Holder, J., Rajan, R., Song, X., Biedenkapp, A., Miao, Y., Eimer, T., Zhang, B., Nguyen, V., Calandra, R., Faust, A., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13596 · Cited by 98 · see citing works
“In contrast, Dyna (Sutton, 1991) is a canonical MBRL algorithm that essentially treats a learned dynamics model as an environment for model-free RL. Dyna-style approaches are popular since they facilitate faster inference at deployment time (forward-pass of the policy rather than an optimization loop) and have been sh…”
BackgroundInfluentialHierarchical Action Control: Adaptive Collaboration Between Actions and Habits
Balleine, B. W., & Dezfouli, A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02735 · Cited by 88 · see citing works
“For example, one collaborative view, Dyna (Sutton, 1991; see also Gershman et al., 2014; Momennejad et al., 2018), proposes that model-based replay can train the model-free system; a suggestion that devolves to something like rehearsal or perhaps consolidation.”
BackgroundLearning locomotion over rough terrain using terrain templates
Kalakrishnan, M., Buchli, J., Pástor, P., & Stefan Schaal, A. (2009) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2009.5354701 · Cited by 87 · see citing works
Q-Learning with Linear Function Approximation
Melo, F. S., & Ribeiro, I. (2007) · Springer eBooks · doi:10.1007/978-3-540-72927-3_23 · Cited by 88 · see citing works
“Value-based methods such as TD-learning [1], Q-learning [2], SARSA [3] and others [4,5,6] have been exhaustively covered in the reinforcement learning (RL) literature and, under mild assumptions, have been proven to converge to the desired solution [7].”
MethodOccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
Zheng, W., Chen, W., Huang, Y., Zhang, B., Duan, Y., & Lu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72624-8_4 · Cited by 93 · see citing works
Xu, J. L., Hou, Z., Liu, Z., & Qiao, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05240 · Cited by 87 · see citing works
“Additionally, some researchers have focused on exploring how to make full use of the learned dynamic model in addition to the commonly used Dyna architecture [97] and GPS-based methods [26] in order to simulate the entire trajectory every iteration.”
MethodAn object-oriented representation for efficient reinforcement learning
Diuk, C., Cohen, A., & Littman, M. L. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390187 · Cited by 226 · see citing works
Exploration of Multi-State Environments: Local Measures and Back-Propagation of Uncertainty
Meuleau, N., & Bourgine, P. (1999) · Machine Learning · doi:10.1023/a:1007541107674 · Cited by 85 · see citing works
“It has already been used by Sutton (1990) to deal with non-stationarity in an indirect algorithm, and mentioned by Thrun (1992a, 1992b).”
BackgroundMethodInfluentialExploring Model-based Planning with Policy NetworksPreprint
Wang, T., & Ba, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08649 · Cited by 98 · see citing works
“Dyna [38, 39] algorithm alternately performs sampling in the real environments and optimize the controllers on the learned model of the environments.”
BackgroundHuys, Q. J. M., Guitart‐Masip, M., Dolan, R. J., & Dayan, P. (2015) · Clinical Psychological Science · doi:10.1177/2167702614562040 · Cited by 81 · see citing works
“This follows a venerable computational suggestion (Sutton, 1991) for improving the performance of model-free control.”
BackgroundAutonomous robotic nanofabrication with reinforcement learning
Leinen, P., Esders, M., Schütt, K. T., Wagner, C., Müller, K.-R., & Tautz, F. S. (2020) · Science Advances · doi:10.1126/sciadv.abb6987 · Cited by 80 · see citing works
Model Based Reinforcement Learning for Atari
Kaiser, Ł., Babaeizadeh, M., Miłoś, P., Osiński, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. (2020) · International Conference on Learning Representations · Cited by 75 · see citing works
Inventory management of new products in retailers using model-based deep reinforcement learning
Demizu, T., Fukazawa, Y., & Morita, H. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120256 · Cited by 76 · see citing works
Natural communication with robots
Torrance, M. (1994) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 76 · see citing works
AoI-minimal UAV Crowdsensing by Model-based Graph Convolutional Reinforcement Learning
Dai, Z., Liu, C. H., Ye, Y., Han, R., Yuan, Y., Wang, G., & Tang, J. (2022) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom48880.2022.9796732 · Cited by 75 · see citing works
Anxiety, avoidance, and sequential evaluation
Zorowitz, S., Momennejad, I., & Daw, N. (2019) · bioRxiv · doi:10.1162/cpsy_a_00026 · Cited by 64 · see citing works
A multifaceted vision of the human-AI collaboration: a comprehensive review
Puerta-Beldarrain, M., Gómez-Carmona, O., Sánchez-Corcuera, R., Casado-Mansilla, D., López-de-Ipiña, D., & Chen, L. (2025) · IEEE Access · doi:10.1109/access.2025.3536095 · Cited by 73 · see citing works
“Popular model-free methods include Monte Carlo [123], Temporal Difference Learning [145], Q-learning [146], SARSA [131], and Dyna-Q [147].”
Temporal structure in associative retrieval
Kurth-Nelson, Z. L., Barnes, G. R., Sejdinović, D., Dolan, R. J., & Dayan, P. (2015) · eLife · doi:10.7554/elife.04919 · Cited by 73 · see citing works
“…autonomously, is suggested as central to a wide variety of processes and functions, including online and offline planning and model-learning (Sutton, 1991; Moore and Atkeson, 1993; Foster and Wilson, 2006; Johnson and Redish, 2007; Hasselmo, 2008; Lisman and Redish, 2009; Gupta et al.,…”
BackgroundThe roles of online and offline replay in planning
Eldar, E., Lièvre, G., Dayan, P., & Dolan, R. J. (2020) · eLife · doi:10.7554/elife.56911 · Cited by 74 · see citing works
Information generation as a functional basis of consciousness
Kanai, R., Chang, A., Yu, Y., Magrans de Abril, I., Biehl, M., & Guttenberg, N. (2019) · Neuroscience of Consciousness · doi:10.1093/nc/niz016 · Cited by 72 · see citing works
Learning model-based planning from scratchPreprint
Pascanu, R., Li, Y., Vinyals, O., Heess, N., Buesing, L., Racanière, S., Reichert, D., Weber, T., Wierstra, D., & Battaglia, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06170 · Cited by 72 · see citing works
“The classic “Dyna” algorithm learns a model, which is then used to train a policy [21].”
MethodWeinberg, D., Wang, Q., Timoudas, T. O., & Fischione, C. (2022) · Sustainable Cities and Society · doi:10.1016/j.scs.2022.104351 · Cited by 70 · see citing works
Learning and Querying Fast Generative Models for Reinforcement LearningPreprint
Buesing, L., Weber, T., Racanière, S., Eslami, S. M. A., Rezende, D. J., Reichert, D., Viola, F., Besse, F., Gregor, K., Hassabis, D., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03006 · Cited by 70 · see citing works
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- ACM SIGART Bulletin
- Published
- 1991-07
- Volume
- 2
- Issue
- 4
- Pages
- 160-163
- ISSN
- 0163-5719 (ISSN-L)
- OpenAlex
- W1980035368
- Cited by
- 777
- References
- 18
- Citations by year
- 2026: 56 · 2025: 71 · 2024: 57 · 2023: 63 · 2022: 57 · 2021: 79 · 2020: 115 · 2019: 62 · 2018: 42 · 2017: 25 · 2016: 9 · 2015: 16 · 2014: 5 · 2013: 13 · 2012: 11
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGART Bulletin (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- learning from examples, planning, model-based learning, incremental planning, learning-based planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
