The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Dyna, an integrated architecture for learning, planning, and reacting
Abstract
Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1991
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citationCraik, K. J. W. (1943) · Cited by 1,845
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 916
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 583
Learning and Sequential Decision Making
Barto, A. G., Sutton, R. S., & Watkins, C. (1989) · Cited by 349
Why the Law of Effect will not Go Away*
Dennett, D. C. (1975) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1975.tb00350.x · Cited by 330
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Research works printed but not linked: 10
- Dennett D. C., 1978, Brainstorms, by D. C. Dennett, 71--89
- Holland J. H., 1986, R. Michalski, J. Carbonell &
- Lin Ji., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, 297--305
- Riolo R., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, MIT Press.
- Russell S. J., 1989, Proceedings of IJCAI-89
- 10.1023/A:1022633531479 · DOI · Paper page
- 10.5555/101883.102055 · DOI
- Sutton R.S., 1981, An adaptive network that constructs and uses an internal model of its environment, Cognition and Brain Theory Quarterly, 4, 217
- Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department. Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department.
- 10.1023/A:1022619109594 · DOI · Paper page
Cited by
Sorted by citations
Items 301–400 of 764
Toward Lifelong Affordance Learning Using a Distributed Markov Model
Glover, A., & Wyeth, G. F. (2016) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2016.2612721 · Cited by 10
“The action policy is updated on-line in parallel with the model as first suggested in Sutton’s “Dyna” architecture [21].”
MethodAn Anti-Jamming Game in VANET Platoon with Reinforcement Learning
Fan, Y., Xiao, X., & Feng, W. (2018) · IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan) · doi:10.1109/icce-china.2018.8448435 · Cited by 10
Bayesian Reinforcement Learning: A Survey
Ghavamzadeh, M., Mannor, S., Pineau, J., & Tamar, A. (2016) · CERN Document Server (European Organization for Nuclear Research) · doi:10.48550/arxiv.1609.04436 · Cited by 10
“Some examples of these methods are the Dyna architecture [Sutton, 1991] and prioritized sweeping [Moore and Atkeson, 1993].”
Computational Neural Mechanisms of Goal-Directed Planning and Problem Solving
Fine, J. M., Zarr, N., & Brown, J. W. (2020) · Computational Brain & Behavior · doi:10.1007/s42113-020-00095-7 · Cited by 9
Mixed Reinforcement Learning for Efficient Policy Optimization in Stochastic Environments
Mu, Y., Peng, B., Gu, Z., Li, S. E., Liu, C., Nie, B., Zheng, J., & Zhang, B. (2020) · International Conference on Control, Automation and Systems (ICCAS) · doi:10.23919/iccas50221.2020.9268413 · Cited by 11
Butz, M. V. (2015) · Springer eBooks · doi:10.1007/978-3-662-43505-2_47 · Cited by 13
A framework for Anticipatory Machine Improvisation and Style Imitation
Cont, A., Dubnov, S., & Assayag, G. (2006) · HAL · Cited by 10
Jarvis: Moving Towards a Smarter Internet of Things
Mudgerikar, A., & Bertino, E. (2020) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs47774.2020.00020 · Cited by 10
“Formally, Jarvis can be defined as a model-based RL framework based on the Dyna-Q framework [12], where: (i) the model of the environment in terms of safety/security policies is learnt from actual user experiences through supervised learning, and (ii) the optimal policies for the learnt model are…”
MethodUsing Monte Carlo Tree Search as a Demonstrator within Asynchronous Deep RLPreprint
Kartal, B., Hernández-Leal, P., & Taylor, M. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00045 · Cited by 12
DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous ControlPreprint
Alvarez, V. M. M., Rosca, R., & Falcutescu, C. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.04278 · Cited by 32
“Ensembles: Model ensembles have been successfully used in Dyna-style algorithms (Sutton, 1991) to improve their performance.”
MethodLearning to act by integrating mental simulations and physical experimentsPreprint
Dasgupta, I., Smith, K. A., Schulz, E., Tenenbaum, J. B., & Gershman, S. J. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/321497 · Cited by 10
“This model is based on the theory that just as we can learn mappings between actions and outcomes by real-world observations, we can gain information about this mapping using simulations in place of observations (Gershman et al., 2017; Sutton, 1991).”
MethodComrie, A. E., Monroe, E. J., Kahn, A. E., Denovellis, E. L., Joshi, A., Guidera, J. A., Krausz, T. A., Berke, J. D., Daw, N. D., & Frank, L. M. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.23.613567 · Cited by 10
“That said, we expect that a more mechanistic implementation would approximate the HMM computations by augmenting a Q-learning style model with non-local activations and updates 74,124 .”
MethodModel-based trajectory stitching for improved behavioural cloning and its applications
Hepburn, C. A., & Montana, G. (2023) · Machine Learning · doi:10.1007/s10994-023-06392-z · Cited by 8
“Model-based algorithms rely on an approximation of the environment’s dynamics [34, 35], that is probability distributions where the next state and reward are predicted from a current state and action.”
A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making
Núñez-Molina, C., Mesejo, P., & Fernández-Olivares, J. (2024) · ACM Computing Surveys · doi:10.1145/3663366 · Cited by 10
“We will use Dyna [146] and AlphaZero [141] as illustrative examples of the different existing model-based RL algorithms.”
Tomasetto, M., Braghin, F., & Manzoni, A. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2025.118030 · Cited by 10
A survey and tutorial of reinforcement learning methods in Process Systems Engineering
Bloor, M., Mowbray, M., Chanona, E. A. D. R., & Tsay, C. (2025) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2025.109515 · Cited by 10
Asarkaya, A. S., Aksaray, D., & Yazıcıoğlu, Y. (2021) · International Conference on Unmanned Aircraft Systems (ICUAS) · doi:10.1109/icuas51884.2021.9476694 · Cited by 9
“A new breed of RL, also referred as hybrid RL (e.g., [19], [20], [21], [22]), has emerged to inherit the advantages of both methods.”
BackgroundLearning Pessimism for Reinforcement Learning
Cetin, E., & Çeliktutan, O. (2021) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25852 · Cited by 31
“This algorithm learns a large ensemble of world models with Dyna-style (Sutton 1991) optimization to train the policy.”
MethodReinforcement Learning for Optimal HVAC Control: From Theory to Real-World Applications
Sayed, K. A., Boodi, A., Broujeny, R. S., & Beddiar, K. (2023) · Annual Conference of the IEEE Industrial Electronics Society · doi:10.1109/iecon51785.2023.10312131 · Cited by 7
“Examples of model-based algorithms include: Dynamic programming (DP), Dyna [21], AlphaZero [22], imagination-augmented agents (I2A) [23], Model-Based Value Expansion (MBVE) [24], and Model-Based RL with Model-Free Fine-Tuning (MBMF) [25].”
MethodData-driven Construction of Symbolic Process Models for Reinforcement Learning
Derner, E., Kubalı́k, J., & Babuška, R. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461182 · Cited by 9
“To speed up learning, various model-based methods have been proposed [4], [5], [6], [7], [8], [9].”
MethodNeuroevolutive Control of Industrial Processes Through Mapping Elites
Langarica, S., & Núñez, F. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3019846 · Cited by 9
“A well-known model-based RL algorithm is Dyna [24], which consists of an RL agent that interacts with the real environment and a model of it.”
MethodReinforced iLQR: A Sample-Efficient Robot Locomotion Learning
Zong, T., Sun, L., & Liu, Y. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561223 · Cited by 9
Hierarchical reinforcement learning in continuous state and multi-agent environments
Ghavamzadeh, M., & Mahadevan, S. (2005) · Cited by 9
A Novel Model for Arbitration Between Planning and Habitual Control Systems
Fard, F. S., & Trappenberg, T. P. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00052 · Cited by 9
Learning to Plan Hierarchically From Curriculum
Morere, P., Ott, L., & Ramos, F. T. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2920285 · Cited by 9
“In model based RL [31], environment dynamics are learned from observed transitions, and the learned model can then be used in conjunction with a planning algorithm such as Monte-Carlo Tree Search [5], DESPOT [37] or RRT [10], [17] to find an optimal plan.”
Reminders of past choices bias decisions for reward in humansPreprint
Bornstein, A. M., Khaw, M. W., Shohamy, D., & Daw, N. D. (2015) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/033910 · Cited by 8
Uncertainty-aware Model-based Policy OptimizationPreprint
Vuong, T.-L., & Tran, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10717 · Cited by 9
“Unlike many other model-based approaches, we also learn the reward function, along the same design of classical MBRL algorithms [28].”
MethodEfficient reinforcement learning control for continuum robots based on Inexplicit Prior KnowledgePreprint
Liu, J., Shou, J., Fu, Z., Zhou, H., Xie, R., Zhang, J., Fei, J., & Zhao, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11573 · Cited by 9
“Model-based reinforcement learning (MBRL) began with Dyna [6] architecture.”
Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RLPreprint
Cang, C., Rajeswaran, A., Abbeel, P., & Laskin, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09119 · Cited by 9
“In dyna-style approaches [42, 39, 43], interactions with the environment are used to update the dynamics model and the RL policy is trained on synthetic rollouts from the dynamics model, often using a model-free RL algorithm like policy gradients or actor-critic.”
MethodExploring the limits of hierarchical world models in reinforcement learning
Schiewer, R., Subramoney, A., & Wiskott, L. (2024) · Scientific Reports · doi:10.1038/s41598-024-76719-w · Cited by 19
“Singh [48] presents a hierarchical DYNA [50] variant that learns hierarchical world models in MDPs with a factorised structure.”
A Survey on Learning Motion Planning and Control for Mobile Robots: Toward Embodied Intelligence
Wang, M., Niu, Y., Wang, B., Zhang, W., & Wang, C. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3656889 · Cited by 9
“TheDyna-style framework [143] utilizes learned models to generate virtual interaction experiences, which are used to augment the replay bu ff er D of any model-free algorithm to improve sample e ffi ciency.”
Jocas, M., Kurrek, P., Zoghlami, F., Gianni, M., & Salehi, V. (2019) · Proceedings of the ... International Conference on Engineering Design · doi:10.1017/dsi.2019.210 · Cited by 8
Automatic Music Playlist Generation via Simulation-based Reinforcement Learning
Tomasi, F., Cauteruccio, J., Kanoria, S., Ciosek, K., Rinaldi, M., & Dai, Z. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599777 · Cited by 8
“Our approach to model-based RL is loosely based on the Dyna architecture [28].”
MethodCARL: A Synergistic Framework for Causal Reinforcement Learning
Méndez-Molina, A., Morales, E. F., & Sucar, L. E. (2023) · IEEE Access · doi:10.1109/access.2023.3331728 · Cited by 8
“They took inspiration from model-based reinforcement learning (MBRL), e.g., [24], [25], [26], but the structure of the given models and the way to use them are different.”
MethodHill Climbing on Value Estimates for Search-control in Dyna
Pan, Y., Yao, H., Farahmand, A., & White, M. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/445 · Cited by 8
DARES: An Asynchronous Distributed Recommender System Using Deep Reinforcement Learning
Shi, B., Τράγος, Η., Özsoy, M. G., Dong, R., Hurley, N., Smyth, B., & Lawlor, A. (2021) · IEEE Access · doi:10.1109/access.2021.3087406 · Cited by 9
“Later, they proposed Pseudo Dyna-Q (PDQ) [62] which is based on Dyna-Q [30], [40].”
MethodGuiding Robot Exploration in Reinforcement Learning via Automated Planning
Hayamizu, Y., Amiri, S., Chandan, K., Takadama, K., & Zhang, S. (2021) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v31i1.16011 · Cited by 8
“Dyna-Q (Sutton 1991) is a model-based RL framework, and includes the two primary components of model-free RL (Q-learning) and probabilistic planning (e.”
MethodColas, J. T., Dundon, N. M., Gerraty, R. T., Saragosa‐Harris, N. M., Szymula, K. P., Tanwisuth, K., Tyszka, J. M., van Geen, C., Ju, H., Toga, A. W., et al. (2022) · Human Brain Mapping · doi:10.1002/hbm.25988 · Cited by 10
“This hippocampal replay is reminiscent of the model-free but quasi-model-based “Dyna” architecture from machine learning that approximates model-based dynamic programming as indirect RL with quasi-inferential iterations of simulated experiences (Sutton, 1990, 1991).”
MethodControl-Oriented Model-Based Reinforcement Learning with Implicit Differentiation
Nikishin, E., Abachi, R., Agarwal, R., & Bacon, P. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20758 · Cited by 8
Offline Replay Supports Planning: fMRI Evidence from Reward RevaluationPreprint
Momennejad, I., Otto, A. R., Daw, N. D., & Norman, K. A. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/196758 · Cited by 9
“In the formalism of reinforcement learning in artificial intelligence, the Dyna family of algorithms implements offline planning, which is accomplished by learning from replay of experiences that were initially encoded during online behavior (8) (6).”
BackgroundMethodResultPlan2Vec: Unsupervised Representation Learning by Latent PlansPreprint
Yang, G., Zhang, A., Morcos, A. S., Pineau, J., Abbeel, P., & Calandra, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03648 · Cited by 8
“Under the context of reinforcement learning, plan2vec treats the graph as a model of the state space, and learns from Dyna-styled unroll using graph-search as an expert policy [42; 2].”
MethodSystematic Evaluation of Causal Discovery in Visual Model Based Reinforcement LearningPreprint
Ke, N. R., Didolkar, A., Mittal, S., Goyal, A., Lajoie, G., Bauer, S., Rezende, D. J., Bengio, Y., Mozer, M. C., & Pal, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00848 · Cited by 44
“With a well-trained world model, these algorithms can then simulate the environment and look ahead to future events to establish better value estimates, without requiring expensive interactions with the environment [Sutton, 1991].”
MethodPrioritizing replay when future goals are unknownPreprint
Sagiv, Y., Akam, T. E., Witten, I. B., & Daw, N. D. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.02.29.582822 · Cited by 8
“Mattar and Daw [17] formalized a version of the value hypothesis in a reinforcement learning (RL) model, specifying the particular computation (a DYNA-Q [33] value function update) hypothetically accomplished by each individual replay event.”
BackgroundGenerating Fingerings for Piano Music with Model-Based Reinforcement Learning
Gao, W., Zhang, S., Zhang, N., Xiong, X., Shi, Z., & Ka, S. (2023) · Applied Sciences · doi:10.3390/app132011321 · Cited by 7
“Prioritized sweeping is essentially an improvement to the Dyna architecture [21].”
BackgroundModel-based Adversarial Imitation Learning from Demonstrations and Human Reward
Huang, J., Hao, J., Juan, R., Gómez, R., Nakarnura, K., & Li, G. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341411 · Cited by 7
“The most classic model-based RL algorithm is Dyna [27], which uses generated simulation experience to accelerate learning in a model-free algorithm.”
MethodGeometric Optimization Frameworks in Mobile Robot Path Planning
Nguyen, C. T., & Pham, H.-A. (2025) · IEEE Access · doi:10.1109/access.2025.3600708 · Cited by 7
Between planning and map building: Prioritizing replay when future goals are uncertain
Sagiv, Y., Akam, T., Witten, I. B., & Daw, N. D. (2025) · Neuron · doi:10.1016/j.neuron.2025.09.021 · Cited by 23
Using Partial-Policy Q-Learning to Plan Path for Robot Navigation in Unknown Enviroment
Zhang, J., Zhang, J., Ma, Z., & He, Z. (2017) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid.2017.132 · Cited by 7
“Some of these are model-based [3], [4], and others are model-free [5], [6].”
BackgroundHeuristics in risky decision-making relate to preferential representation of information
Russek, E. M., Moran, R., Liu, Y., Dolan, R. J., & Huys, Q. J. M. (2024) · Nature Communications · doi:10.1038/s41467-024-48547-z · Cited by 8
“Participants were then required to press an arrowkeyindicatingthedirectionofthearrowasquicklyastheycould.”
BackgroundTWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer
Yamada, J., Rigter, M. P., Collins, J., & Posner, I. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610450 · Cited by 14
“Model-based RL has emerged as a promising approach to solving complex control problems by leveraging a learned dynamics model [16], [17], [18].”
BackgroundPoint Cloud Models Improve Visual Robustness in Robotic Learners
Peri, S. V., Lee, I., Kim, C., Li, F., Hermans, T., & Lee, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610710 · Cited by 12
“One technique in sequential decision making is to learn a model of the environment [39] and use it for planning [40]–[43] or policy learning [11], [18], [44], [45].”
Incremental model-based reinforcement learning with model constraint
Yang, Z., Fu, M., Qu, H., Li, F., Shi, S., & Hu, W. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107245 · Cited by 7
Policy Improvement by a Model-Free Dyna Architecture
Hwang, K., & Lo, C.-Y. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2244100 · Cited by 7
Croonenborghs, T., Ramon, J., Blockeel, H., & Bruynooghe, M. (2006) · Lirias · Cited by 7
Indirect Reinforcement Learning for Incident-responsive Ramp Control
Lu, C., Chen, H., & Grant‐Muller, S. (2014) · Procedia - Social and Behavioral Sciences · doi:10.1016/j.sbspro.2014.01.146 · Cited by 7
A CMAC-Q-Learning based Dyna agent
Hsu, Y.-P., Jiang, W., & Lin, H.-Y. (2008) · SICE Annual Conference (SICE) · doi:10.1109/sice.2008.4655167 · Cited by 7
“To deal with this learning time problem, the Dyna algorithm was hence proposed [1] [ 2 ].”
MethodReinforcement-Learning Approach Guidelines for Energy Management
Rioual, Y., Laurent, J., & Diguet, J.-P. (2019) · Journal of Low Power Electronics · doi:10.1166/jolpe.2019.1618 · Cited by 7
Santos, M., José Antonio Martín, H., López, V., & Botella, G. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1101.4003 · Cited by 7
“A new algorithm based on the Dyna architecture (Sutton, 1991; Sutton and Barto, 1998), that combines heuristic on-line search and Q -Learning is presented.”
InfluentialThe Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving.Preprint
Allen, K. R., Smith, K. A., & Tenenbaum, J. B. (2019) · arXiv (Cornell University) · Cited by 8
Frequency-based Search-control in DynaPreprint
Pan, Y., Mei, J., & Farahmand, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05822 · Cited by 7
“Model-based reinforcement learning (MBRL) (Lin, 1992; Sutton, 1991b; Daw, 2012; Sutton & Barto, 2018) methods have successfully been applied to many benchmark domains (Gu et al., 2016; Ha, David and Schmidhuber, J ¨ urgen, 2018; Kaiser et al., 2020).”
MethodEfficient and Robust Reinforcement Learning with Uncertainty-based Value ExpansionPreprint
Zhou, B., Zeng, H., Wang, F., Li, Y., & Tian, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.05328 · Cited by 11
“A lot of efforts has been devoted to this field, including the Dyna algorithm [Sutton, 1991], model-based value expansion [Feinberg et al., 2018], I2A [Weber et al., 2017], etc.”
Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations OnlinePreprint
Pan, Y., Banman, K., & White, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08068 · Cited by 9
The roles of online and offline replay in planningPreprint
Eldar, E., Lièvre, G., Dayan, P., & Dolan, R. J. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.26.009571 · Cited by 10
Think Too Fast Nor Too Slow: The Computational Trade-off Between Planning And Reinforcement LearningPreprint
Moerland, T. M., Deichler, A., Baldi, S., Broekens, J., & Jonker, C. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07404 · Cited by 7
“Model-based RL, where we use observed data to approximate the dynamics model, has mostly focused on dealing with enhancing data efficiency and dealing with uncertainty in the learned models [Sutton, 1991; Chua et al. , 2018].”
MethodOnline Monitoring of Heterogeneous Partially Observable Data Streams Based on Q-Learning
Li, H., Ye, H., Cheng, J.-R. C., & Liu, K. (2024) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2024.3411770 · Cited by 6
“Other approaches such as the Dyna algorithm [19] learns a model of the environment through keeping interacting with the environment and then uses this model to generate simulated experience and update the value function.”
Learning 4D Embodied World Models
Zhen, H., Sun, Q., Zhang, H., Li, J., Zhou, S., Du, Y., & Gan, C. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00507 · Cited by 8
Learning Representations in Reinforcement Learning
Heravi, J. R. (2019) · Cited by 7
Data-Informed Residual Reinforcement Learning for High-Dimensional Robotic Tracking Control
Li, C., Liu, F., Wang, Y., & BUSS, M. J. (2024) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2024.3412275 · Cited by 6
“The common MBRL improves sample efficiency by first learning a latent model of environment dynamics, and then, using the learned latent model to simulate experience for policy learning [4], [5], [6].”
BIMRL: Brain Inspired Meta Reinforcement Learning
Rohani, S. R. R., Hedayatian, S., & Baghshah, M. S. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981250 · Cited by 6
“Dyna [11] and successor representation [12] are two such attempts.”
BackgroundSocially Assistive Robot-Based Intervention for Children with Autism Spectrum Disorder
Feil-Seifer, D. (2008) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · Cited by 6
An Active Exploration Method for Data Efficient Reinforcement Learning
Zhao, D., Liu, J., Wu, R., Cheng, D., & Tang, X. (2019) · International Journal of Applied Mathematics and Computer Science · doi:10.2478/amcs-2019-0026 · Cited by 6
“The Dyna papers are classical works in the model-based RL domain (Sutton, 1991; Silver et al., 2008).”
BackgroundFrom Reinforcement Learning Towards Artificial General Intelligence
Rocha, F. M., Costa, V. S., & Reis, L. P. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-45691-7_37 · Cited by 6
From non-deterministic to probabilistic planning with the help of statistical relational learning
Thon, I., Gutmann, B., van Otterlo, M., Landwehr, N., & Raedt, L. D. (2009) · Lirias · Cited by 6
Modification of Q-learning to Adapt to the Randomness of Environment
Luo, X., Gao, Y.-B., Huang, S., Zhao, Y., & Zhang, S. (2019) · International Conference on Control, Automation and Information Sciences (ICCAIS) · doi:10.1109/iccais46528.2019.9074718 · Cited by 6
A multiagent variant of Dyna-Q
Weiß, G. (2002) · International Conference on Multi Agent Systems · doi:10.1109/icmas.2000.858525 · Cited by 6
“Dyna-Q (e.g., [1] and [2, Chapter 9]) is a single-agent framework that integrates planning and reacting on the basis of learning.”
BackgroundSpeeding up Tabular Reinforcement Learning Using State-Action Similarities
Rosenfeld, A., Taylor, M. E., & Kraus, S. (2017) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/3091125.3091417 · Cited by 6
Kielak, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10181 · Cited by 23
“Most studies focused on the visually-rich Atari domain employ model-based strategies inspired by the classical Dyna approach (Sutton, 1991) and action-conditional prediction methods (Oh et al., 2015; Leibfried et al., 2016).”
Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor ModelsPreprint
Aminmansour, F., Jafferjee, T., Imani, E., Talvitie, E., Bowling, M., & White, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04363 · Cited by 6
“Early work on Dyna considered environments with deterministic dynamics and finite state spaces (Sutton, 1990, 1991).”
Learning Off-Policy with Online PlanningPreprint
Sikchi, H., Zhou, W., & Held, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10066 · Cited by 6
“Approaches such as MBPO [13] and MAAC [24] follow Dyna-style [25] learning where imagined short-horizon trajectories are used to provide additional transitions to the replay buffer leveraging model generalization.”
MethodImproving Computational Efficiency in Visual Reinforcement Learning via Stored EmbeddingsPreprint
Chen, L., Lee, K., Srinivas, A., & Abbeel, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02886 · Cited by 6
“To address this, several improvements in pixelbased off-policy RL have been proposed recently: algorithmic improvements such as Rainbow [15] and its data-efficient versions [48]; using ensemble approaches based on bootstrapping [34, 26]; combining RL algorithms with auxiliary predictive, reconstruction and contrastive…”
MethodModel-Based Episodic Memory Induces Dynamic Hybrid ControlsPreprint
Lê, H., George, T. K., Abdolshah, M., Tran, T., & Venkatesh, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02104 · Cited by 6
The Case for a Realistic Mobile Robot Simulator
Torrance, M. (1992) · Cited by 6
Deep Reinforcement Learning Based Control of Lower Limb Exoskeleton
Rv, M., & Rakshit, S. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651057 · Cited by 6
“In this work, the proposed framework follows the Dyna-style RL [33] algorithm, which utilizes data obtained from environment policy interactions to learn a dynamics model and use the virtual data from the learned model to improve the policy.”
MethodJiang, T., Guan, Y., Ma, L., Xu, J. L., Meng, J., Chen, W., Zeng, Z., Li, L., Wu, D., & Chen, R. (2025) · IEEE Transactions on Robotics · doi:10.1109/tro.2025.3584504 · Cited by 5
“Building an accurate and generalizable transition model of the environment capable of reacting to agent interactions has been a long-standing problem in optimal control and model-based RL [35], [36].”
A review of reinforcement learning approaches for pursuit-evasion games
Kun, Y. A. N. G., Shen, A., Xu, N., Deng, F. Y., Lu, M., & Chen, C. (2025) · Chinese Journal of Aeronautics · doi:10.1016/j.cja.2025.103940 · Cited by 5
Proximal policy optimization with model-based methods
Li, S., Zhang, W., Zhang, H., Zhang, X., & Leng, Y. (2022) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-211935 · Cited by 7
“TheDyna framework is the most basic method of combining model-based and model-free meth-ods [72] and can be applied to various model-free algorithms.”
Approximate Model-Based Shielding for Safe Reinforcement Learning
Goodall, A. W., & Belardinelli, F. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230357 · Cited by 5
“Dyna – “an integrated architecture for learning, planning and reacting” [38] proposed an architecture that learns a dynamics model of the environment in addition to a reward maximising policy.”
BackgroundDeep reinforcement learning in finite-horizon to explore the most probable transition pathway
Guo, J., Gao, T., Zhang, P., Han, J., & Duan, J. (2023) · Physica D Nonlinear Phenomena · doi:10.1016/j.physd.2023.133955 · Cited by 5
“There are some typical methods such as Dyna [26], Model-based policy optimization (MBPO) [27], Model-based value expansion (MVE) [28], etc Some efforts have been made to seek transition pathways via reinforcement learning in real-world applications.”
MethodFOSS: A Self-Learned Doctor for Query Optimizer
Zhong, K., Sun, L., Ji, T., Li, C., & Chen, H. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00330 · Cited by 7
“FOSS follows the Dyna-style [43] method, using the learned environment model to generate more simulated experiences for the agent.”
MethodDemonstration of the Dyna Reinforcement Learning Framework for Reactive Close Proximity Operations
Majumdar, R., Sternberg, D. C., Albee, K., & Jia-Richards, O. (2025) · AIAA SciTech Forum · doi:10.2514/6.2025-1002 · Cited by 5
“The broad context of this work is to introduce SmallSat Steward, a reactive and integrated architecture for online model learning and trajectory planning based on the Dyna reinforcement learning architecture [10].”
Matarić, M. J., & Feil-Seifer, D. (2012) · Cited by 5
A New Asynchronous Architecture for Tabular Reinforcement Learning Algorithms
Zhao, X., Ding, S., & An, Y. (2018) · Proceedings in adaptation, learning and optimization · doi:10.1007/978-3-030-01520-6_15 · Cited by 5
Augmented Memory Replay in Reinforcement Learning With Continuous Control
Ramičić, M., & Bonarini, A. (2021) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3050723 · Cited by 5
“…with the intersection between the neuroscientific models of hippocampal memory consolidation and the artificial replay memory in reinforcement learning: they implemented a prioritization of the agents replay memory to a DYNA-based reinforcement learning algorithm [46], based on its predicted utility.”
MethodHeuristic Reinforcement Learning Based Overtaking Decision for an Autonomous Vehicle
Du, G., Zou, Y., Zhang, X., Dong, G., & Yin, X. (2021) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2021.10.141 · Cited by 5
Modular Networks Prevent Catastrophic Interference in Model-Based Multi-task Reinforcement Learning
Schiewer, R., & Wiskott, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-95470-3_23 · Cited by 5
Reinforcement Learning Navigation for Robots Based on Hippocampus Episode Cognition
Yuan, J., Guo, W. P., Hou, Z., Zha, F., Li, M., Wang, P., & Sun, L. (2024) · Journal of Bionic Engineering · doi:10.1007/s42235-023-00454-7 · Cited by 5
“We refer to the algorithm framework of Dyna[26].”
Reinforcement Learning of User Preferences for a Ubiquitous Personal Assistant
Zaidenberg, S., & Reignier, P. (2011) · InTech eBooks · doi:10.5772/13723 · Cited by 5
“This is for instance the Dyna approach (Sutton, 1991) that we will present in subsection 4.4.”
MethodInfluentialA Tensor Factorization Approach to Generalization in Multi-agent Reinforcement Learning
Bromuri, S. (2012) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat.2012.21 · Cited by 5
Learning a Policy for Coordinated Sampling in Body Sensor Networks
Liu, S., Panangadan, A. V., Talukder, A., & Raghavendra, C. S. (2011) · Wearable and Implantable Body Sensor Networks · doi:10.1109/bsn.2011.12 · Cited by 5
“Future rewards are discounted by factor 6, i.e., a reward obtained steps in the future is worth 6 , 0 6 1, compared to receiving it in the current state.”
BackgroundCombined Use of Reinforcement Learning and Simulated Annealing: Algorithms and Applications
Péter, S. (2009) · Cited by 5
Metadata
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- ACM SIGART Bulletin
- Published
- 1991-07
- Volume
- 2
- Issue
- 4
- Pages
- 160-163
- ISSN
- 0163-5719 (ISSN-L)
- OpenAlex
- W1980035368
- Cited by
- 777
- References
- 18
- Citations by year
- 2026: 56 · 2025: 71 · 2024: 57 · 2023: 63 · 2022: 57 · 2021: 79 · 2020: 115 · 2019: 62 · 2018: 42 · 2017: 25 · 2016: 9 · 2015: 16 · 2014: 5 · 2013: 13 · 2012: 11
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGART Bulletin (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- learning from examples, planning, model-based learning, incremental planning, learning-based planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
