The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Dyna, an integrated architecture for learning, planning, and reacting
Abstract
Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1991
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citationCraik, K. J. W. (1943) · Cited by 1,845
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 916
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 583
Learning and Sequential Decision Making
Barto, A. G., Sutton, R. S., & Watkins, C. (1989) · Cited by 349
Why the Law of Effect will not Go Away*
Dennett, D. C. (1975) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1975.tb00350.x · Cited by 330
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Research works printed but not linked: 10
- Dennett D. C., 1978, Brainstorms, by D. C. Dennett, 71--89
- Holland J. H., 1986, R. Michalski, J. Carbonell &
- Lin Ji., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, 297--305
- Riolo R., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, MIT Press.
- Russell S. J., 1989, Proceedings of IJCAI-89
- 10.1023/A:1022633531479 · DOI · Paper page
- 10.5555/101883.102055 · DOI
- Sutton R.S., 1981, An adaptive network that constructs and uses an internal model of its environment, Cognition and Brain Theory Quarterly, 4, 217
- Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department. Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department.
- 10.1023/A:1022619109594 · DOI · Paper page
Cited by
Sorted by citations
Items 501–600 of 764
Neurocomputational Mechanisms for Adaptive Self-Preservative Robot Behaviour
Guerrero, N., & Ignacio, N. (2016) · Cited by 3
Collaborative Decision in Multi-Agent Learning of Action Models
Rodrigues, C., Soldano, H., Bourgne, G., & Rouveirol, C. (2016) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2016.0103 · Cited by 2
“In the indirect or model-based Reinforcement Learning framework [2], the agent explicitly learns such an action model, allowing it to predict the effect of actions, and uses it as an input of a symbolic planner, whose output is a plan to execute in order to reach its current goal.”
BackgroundPain and self-preservation in autonomous robots: From neurobiological models to psychiatric disease
Piccolo, L., Libera, F. D., Bonarini, A., Seymour, B., & Ishiguro, H. (2017) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2017.8329816 · Cited by 2
CostNet: An End-to-End Framework for Goal-Directed Reinforcement Learning
Andersen, P., Goodwin, M., & Granmo, O. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63799-6_7 · Cited by 2
“The Dyna algorithm from [26] trains in two steps.”
MethodPolicy Learning with Deep Koopman Operator Networks
Zhao, D., Li, Y., Yang, X., & Shi, Y. (2023) · International Conference on Advanced Robotics and Mechatronics (ICARM) · doi:10.1109/icarm58088.2023.10218748 · Cited by 2
“Among the black-box models, early value-based RL, such as Q-learning [18], SARA [19], etc., are learned by a tablebased model in a broad sense, and their models have large errors.”
MethodData-efficient model-based reinforcement learning with trajectory discrimination
Qu, T., Duan, F., Zhang, J., Zhao, B., & Huang, W. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01247-5 · Cited by 2
“An effective method to reduce sample complexity is model-basedreinforcementlearning(MBRL),whichrequires learning an environment model and then optimizing the policy based on the model [13, 30].”
Environment Transformer and Policy Optimization for Model-Based Offline Reinforcement Learning
Wang, P., Zhu, M., & Shen, S. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802094 · Cited by 2
“Sutton [27] proposes an architecture that maintains a dynamics model of the agent’s transitions.”
Banker, T., Lawrence, N. P., & Mesbah, A. (2025) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc63710.2025.11108095 · Cited by 2
“Crucially, RL looks to learn optimal policies through interactions with its environment in a “model-free” manner; however, these methods may also be applied to a simulation environment to boost efficiency [18], [19], [20].”
Free Energy Projective Simulation (FEPS): Active inference with interpretability
Pazem, J., Krumm, M., Vining, A. Q., Fiderer, L. J., & Briegel, H. J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0331047 · Cited by 4
An exploratory rollout policy for imagination-augmented agents
Liu, P., Zhao, Y., Zhao, W. Z., Tang, X., & Yang, Z. (2019) · Applied Intelligence · doi:10.1007/s10489-019-01484-7 · Cited by 2
A Survey: Limited Data Problem and Strategy of Reinforcement Learning
Yang, Z., Wu, F., & Zhao, J. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-16-6372-7_53 · Cited by 2
Combining Hindsight with Goal-enhanced Prediction for Multi-goal Reinforcement Learning
Yang, R., Luo, F., & Li, X. (2021) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai52525.2021.00052 · Cited by 2
Considering Unseen States as Impossible in Factored Reinforcement Learning
Kozlova, O., Sigaud, O., Wuillemin, P.-H., & Meyer, C. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04180-8_64 · Cited by 2
“An implementation of FRL is expressed in the SDYNA framework (Degris et al., 2006a; Degris et al., 2006b) as a structured version of the DYNA architecture (Sutton, 1991).”
Hyper-cubic discretization for TD learning based on autonomous decentralized approach
Kobayashi, Y., & Hosoe, S. (2004) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2003.1244453 · Cited by 2
Teaching by demonstration of robotic manipulators in non-stationary environments
Skoglund, A., Duckett, T., Iliev, B., Lilienthal, A. J., & Palm, R. (2006) · Örebro University (Örebro University) · Cited by 2
Backtracking for More Efficient Large Scale Dynamic Programming
Tripp, C. E., & Shachter, R. D. (2012) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2012.63 · Cited by 2
Trajectory Sampling Value Iteration: Improved Dyna Search for MDPs
Zhou, Y., Liu, Q., Fu, Q., & Zhang, Z. (2015) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/2772879.2773385 · Cited by 2
Efficient Practice for Deep Reinforcement Learning
Kancharla, V. S. S. R. T., & Lee, M. (2019) · IEEE Symposium Series on Computational Intelligence · doi:10.1109/ssci44817.2019.9003129 · Cited by 1
“Also, preparation and improvement go parallel in Dyna-Q which is not the case with practice.”
MethodInfluentialAddressing Task Prioritization in Model-based Reinforcement Learning
Zholus, A., Ivchenkov, Y., & Panov, A. I. (2022) · Studies in computational intelligence · doi:10.1007/978-3-031-19032-2_3 · Cited by 2
Ahmadi, M. A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-24010-2.00008-1 · Cited by 2
Shaping Model-Free Reinforcement-Learning with Model-Based Pseudorewards
Krueger, P. M., & Griffiths, T. L. (2018) · Conference on Cognitive Computational Neuroscience (CCN) · doi:10.32470/ccn.2018.1191-0 · Cited by 2
“Our first simulation involved an agent learning in a maze environment (Sutton, 1991).”
BackgroundMethodMapping Web Usage Patterns to MDP Model and Mining with Reinforcement Learning
Gao, Y., Luo, Z., & Li, N. (2005) · Lecture notes in computer science · doi:10.1007/11540007_85 · Cited by 2
“Q-learning is exploration insensitive, Q-learning method must converge with probability 1 to the optimal Q ∗ [ 10 ].”
MethodReinforcement Learning for Racecar Control
Cleland, B. G. (2006) · Research Commons (University of Waikato) · Cited by 2
Temporal Second Difference TracesPreprint
Bloch, M. K. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1104.4664 · Cited by 2
Reinforcement Learning for Object Manipulation Using Low-dimensional Mapping
Kobayashi, Y., FUJII, H., & Hosoe, S. (2006) · Transactions of the Society of Instrument and Control Engineers · doi:10.9746/sicetr1965.42.814 · Cited by 2
Intelligent Trainer for Model-Based Reinforcement LearningPreprint
Li, Y., Dong, L., Zhou, X., Wen, Y., & Guan, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09496 · Cited by 2
“To tackle this challenge with training data, researchers have previously proposed a model-based reinforcement learning (MBRL) [6].”
BackgroundA Short Survey on Probabilistic Reinforcement LearningPreprint
Russel, R. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.07010 · Cited by 2
Greenyer, J., Lochau, M., & Vogel, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11851 · Cited by 5
Bootstrapping the Expressivity with Model-based PlanningPreprint
Dong, K., Luo, Y., & Ma, T. (2019) · arXiv (Cornell University) · Cited by 2
Sample-Efficient Model-based Actor-Critic for an Interactive Dialogue TaskPreprint
Kudashkina, K., Chockalingam, V., Taylor, G. W., & Bowling, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13657 · Cited by 2
“Their endto-end architecture is close to the Dyna architecture (Sutton 1991).”
BackgroundMaximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding ErrorsPreprint
Zhang, C., Kuppannagari, S. R., & Prasanna, V. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04802 · Cited by 2
“The differences between MEMR and MBPO are: 1) MEMR follows Dyna (Sutton, 1991) that only generates single-step model rollouts while MBPO encourages generating short-horizon model rollouts.”
InfluentialZielinski, K., Teixeira, M., Ribeiro, R., & Casanova, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16035 · Cited by 3
“There is a wide variety of algorithms of RL, such as Q-Learning [36], H-Learning [33], Dyna [30], Sarsa [31] Deep Q [19], among others.”
MethodTDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?Preprint
Romoff, J., Henderson, P., Kanaa, D., Bengio, E., Touati, A., Bacon, P., & Pineau, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02786 · Cited by 2
“Similarly, performing expected TD updates using a learned model of the transition dynamics has been shown to improve sample efficiency in both the tabular (Sutton, 1991) and the Deep RL setting (Feinberg et al., 2018).”
Defining Digital Quadruplets in the Cyber-Physical-Social Space for Parallel DrivingPreprint
Liu, T., Xing, Y., Chen, L. F., Cao, D., & Wang, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14248 · Cited by 2
“The relevant approaches are adversarial learning [29], adjoint state [30], predictability minimization [31], actor-critic reinforcement learning [32] and Dyna architecture [33].”
BackgroundDeserno, L., Moran, R., Michely, J., Lee, Y., Dayan, P., & Dolan, R. J. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.01.15.426639 · Cited by 2
Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement LearningPreprint
Yang, R., Lyu, J., Yu, Y., Yan, J., Luo, F., Luo, D., Li, L., & Li, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12962 · Cited by 2
On Optimism in Model-Based Reinforcement LearningPreprint
Pacchiano, A., Ball, P. J., Parker-Holder, J., Choromanski, K., & Roberts, S. (2020) · arXiv.org · doi:10.48550/arxiv.2006.11911 · Cited by 29
“Dyna-style approaches [Sutton, 1991], are particularly sensitive to model bias [Deisenroth and Rasmussen, 2011], which often leads to catastrophic failure when policies are trained on inaccurate synthetic data.”
BackgroundPolicy Gradients Incorporating the FuturePreprint
Venuto, D., Lau, E., Precup, D., & Nachum, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02096 · Cited by 5
“Thus, credit assignment boils down to utilizing an appropriate planning algorithm that can perform multiple rollouts in the model, effectively allowing the agent to “look into the future” [32, 29] to determine cause-and-effect [40, 31, 2].”
BackgroundSelf-Consistent Models and ValuesPreprint
Farquhar, G., Baumli, K., Marinho, Z., Filos, A., Hessel, M., van Hasselt, H. P., & Silver, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12840 · Cited by 3
“They have been used to great effect in planning for action selection [45, 29, 49], and for learning policies or value functions more efficiently [51].”
InfluentialAbsraction for Efficient Reinforcement Learning
Telfar, A. (2020) · doi:10.26686/wgtn.17147651.v1 · Cited by 2
Bueno, T. P. (2021) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.45.2021.tde-29102021-133418 · Cited by 2
On the duality of pain and pleasure processing: Why two dimensions of valence may be better than onePreprint
Dulberg, Z., & Cohen, J. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.22.634365 · Cited by 2
Model Learning for Look-Ahead Exploration in Continuous Control
Agarwal, A., Muelling, K., & Fragkiadaki, K. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013151 · Cited by 1
Modular transfer learning with transition mismatch compensation for excessive disturbance rejection
Wang, T., Lu, W., Yu, H., & Liu, D. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01641-4 · Cited by 1
“Various modelbased RL methods have been proposed [45]–[50], and applied successfully on both simulated and real-world robots, such as inverted pendulums [45], manipulators [51], and legged robots [52].”
MethodUncertainty Aware Model Integration on Reinforcement Learning
Nagata, T., Xing, J., Kumazawa, T., & Neftci, E. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892778 · Cited by 1
“Dyna was proposed as an architecture that integrates both model-free and model-based reinforcement learning [7].”
Global Path Planning for Amphibious Robots Based on Improved Q-Learning
Jiang, W., Chen, B., Tu, F., & Wang, Y. (2024) · China Automation Congress (CAC) · doi:10.1109/cac63892.2024.10865651 · Cited by 1
“This idea is being used in the Dyna learning framework proposed by Sutton[20], which combines model-based learning and direct learning.”
Model-based algorithms shape automatic evaluative processing
Melnikoff, D. E., & Kurdi, B. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2417068122 · Cited by 2
“Research has revealed similarities between human reinforcement learning and a computational architecture called DYNA ( 76 ), in which model - based algorithms play a strictly indirect role in evaluative processing ( 15 , 17 – 19 ).”
Sample-efficient reinforcement learning of Koopman eNMPC
Mayfrank, D., Velioglu, M., Mitsos, A., & Dahmen, M. (2025) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2025.109240 · Cited by 1
Approximating Fixpoints of Approximated Functions
Baldan, P., Gurke, S., König, B., Padoan, T., & Wittbold, F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98679-6_9 · Cited by 1
Causal dreamer for partially observable model-based reinforcement learning
Gao, H., Xu, T., Zhang, T., Guo, Y., Zhao, C., Ren, J., Jiang, Y., Guo, S., & Chen, F. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131012 · Cited by 2
Metacognition for Unknown Situations and Environments (MUSE)
Valiente, R., & Pilly, P. K. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108131 · Cited by 5
Learning by doing: an online causal reinforcement learning framework with causal-aware policy
Cai, R., Huang, S., Qiao, J., Chen, W., Zeng, Y., Zhang, K., Sun, F., Yu, Y., & Hao, Z. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4403-7 · Cited by 3
Methodological Overview of Deep Learning in Computational Mechanics
Herrmann, L., Jokeit, M., Weeger, O., & Kollmannsberger, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-89529-6_10 · Cited by 1
Farheen, N., Jaman, G. G., & Schoen, M. P. (2026) · Machines · doi:10.3390/machines14020157 · Cited by 1
Ramakrishnan, S. A., Muliya, M., Chakravarthy, V. S., & Parvaz, M. A. (2026) · Springer eBooks · doi:10.1007/978-3-032-02925-6_33 · Cited by 1
Farajzadehahary, K., Hamzehlou, S., Ballard, N., & Asúa, J. M. (2026) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2026.109604 · Cited by 1
Computing Fixpoints of Learned Functions: Chaotic Iteration and Simple Stochastic Games
Baldan, P., Gurke, S., König, B., & Wittbold, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-22752-2_28 · Cited by 1
AI for battery-accelerated discovery of high-voltage electrolytes for advanced lithium batteries
Zhan, Y. Z., Chen, N., Li, L., Wu, F., & Chen, R. (2026) · Chemical Society Reviews · doi:10.1039/d4cs01250j · Cited by 1
TeXDYNA: Hierarchical Reinforcement Learning in Factored MDPs
Kozlova, O., Sigaud, O., & Meyer, C. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15193-4_46 · Cited by 1
“Structured-DYNA (SDYNA) [6] is a general framework that adapts indirect RL of the DYNA family [8] to the FMDP framework.”
BackgroundMethodOneiric Processing Utilising the Anticipatory Classifier System
Holley, J., Pipe, T., & Carse, B. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30217-9_111 · Cited by 1
“As the ACS interacts with the environment a generalised model is developed, this knowledge can be exploited to perform some model learning ‘Dyna style’ [ 23 ] and reduce environmental interaction.”
MethodCont, A. (2012) · Techniques et sciences informatiques · doi:10.3166/tsi.31.311-335 · Cited by 1
Collaborative Online Learning of an Action Model
Rodrigues, C., Soldano, H., Bourgne, G., & Rouveirol, C. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-41706-6_16 · Cited by 1
Data-Efficient Reinforcement Learning Using Active Exploration Method
Zhao, D., Liu, J., Wu, R., Cheng, D., & Tang, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-04182-3_24 · Cited by 1
Integrating Learning and Planning
Zhang, H., Huang, R., & Zhang, S. (2020) · Springer eBooks · doi:10.1007/978-981-15-4095-0_9 · Cited by 1
Variational Diversity Maximization for Hierarchical Skill Discovery
Zhao, Y., Liu, P., Zhao, W. Z., & Tang, X. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-10912-8 · Cited by 1
“Dyna [26]usedrealexperiencestotrainanenvironmentmodelandthenaugmentedrealexperiences withfictitiousonestotrainapolicy.”
Backward Imitation and Forward Reinforcement Learning via Bi-directional Model Rollouts
Pan, Y., & Lin, F. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981869 · Cited by 1
“In contrast, the Dyna-style algorithm [18] that is one branch of the model-based paradigms learns a dynamics model (forward model) pθ(st+1|st, at) parameterized by θ as the true dynamics from agent interactions with the environment by conducting the policy πφ(at|st) parameterized by φ.”
MethodReview of Deep Reinforcement Learning-Based Recommender Systems
Krishnamoorthi, S., & Shyam, G. K. (2022) · International Conference on Smart Technologies in Computing, Electrical and Electronics (ICSTCEE) · doi:10.1109/icstcee56972.2022.10099739 · Cited by 1
“The ideal value function is estimated using Monte Carlo (MC) methods, Temporal-difference (TD) learning techniques, including Dyna, Sarsa, and Deep Q-Networks.”
BackgroundKucharavy, A., Guerraoui, R., & Dolamic, L. (2023) · Conference on Artificial Life (ALIFE) · doi:10.1162/isal_a_00695 · Cited by 1
UNeC: Unsupervised Exploring In Controllable Space
Xiong, X., Meng, L., Ruan, J., Xu, S., & Xu, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447360 · Cited by 1
“Dyna [27] utilises data generated from model-based predictions for data augmentation in agent trajectories.”
MethodSelf-Playing RNA Inverse Folding
Obonyo, S., Jouandeau, N., & Owuor, D. O. (2024) · SN Computer Science · doi:10.1007/s42979-024-02659-x · Cited by 1
“In model-based the RL agent learns the model of the environment which then informs the policy [15, 20, 74].”
Vector-Based Navigation Inspired by Directional Place Cells
Espino, H., & Krichmar, J. L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71533-4_3 · Cited by 1
Model-Based Reinforcement Learning Inspired by Augmented PD for Robotic Control
Li, X. Y., Shang, W., & Cong, S. (2024) · Chinese Control Conference (CCC) · doi:10.23919/ccc63176.2024.10661623 · Cited by 1
GPI-tree search: algorithms for decision-time planning with the general policy improvement theorem
Bagot, L., D’eer, L., Latré, S., Schepper, T. D., & Mets, K. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11304-4 · Cited by 1
Adaptive, Goal-Oriented Navigation Using a Model of Directionally-Polarized Place Cells
Espino, H., & Krichmar, J. L. (2025) · Adaptive Behavior · doi:10.1177/10597123251364749 · Cited by 1
“The fi rst environment is the Dyna maze, which was introduced to test the Dyna Reinforcement Learning algo-rithm and has been used in previous navigation papers (Mattar & Daw, 2018; Sutton, 1991).”
Interpretable and efficient data-driven discovery and control of distributed systems
Wolf, F., Botteghi, N., Fasel, U., & Manzoni, A. (2025) · Data-Centric Engineering · doi:10.1017/dce.2025.10027 · Cited by 3
“We focus on Dyna-style [Sut91] RL algorithms which learn a surrogate model of the system dynamics, allowing the agent to train on an approximation of the environment and thus requiring fewer data.”
Realizing Undelayed N-step TD prediction with neural networks
Zuters, J. (2010) · IEEE Mediterranean Electrotechnical Conference (MELECON) · doi:10.1109/melcon.2010.5476332 · Cited by 1
“5 shows the Dyna algorithm[4] adapted for Sarsa learning and neural network as a value function.”
MethodModel-Based Reinforcement Learning for Alternating Markov Games
Mellor, D. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-24581-0_44 · Cited by 1
“is an example of this approach [17, 19 , 20].”
BackgroundMethodModelling Human Target Reaching using A novel predictive deep reinforcement learning technique
Fard, F. S. (2018) · Cited by 1
Approximate Dynamic Programming
Kamalapurkar, R., Walters, P., Rosenfeld, J. A., & Dixon, W. E. (2018) · Communications and control engineering/Communications and control engineering series · doi:10.1007/978-3-319-78384-0_2 · Cited by 1
Köpf, F., Nitsch, A., Flad, M., & Hohmann, S. (2020) · International Conference on Control, Automation and Robotics (ICCAR) · doi:10.1109/iccar49639.2020.9108080 · Cited by 1
“…proposed in the next section incorporates partner modeling into the paradigm to accelerate learning by using simulated data and can therefore be interpreted to extend powerful mechanisms such as the Dyna-architecture (Sutton, 1991) or Imagination Rollouts (Gu et al., 2016) to the multi-agent case.”
BackgroundMethodOvercoming Reinforcement Learning Limits with Inductive Logic Programming
Rocha, F. M., Costa, V. S., & Reis, L. P. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-45691-7_38 · Cited by 2
GAN-Based Planning Model in Deep Reinforcement Learning
Chen, S., Jiang, J., Zhang, X., Wu, J., & Lu, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61616-8_26 · Cited by 1
TDprop: Does Adaptive Optimization With Jacobi Preconditioning Help Temporal Difference Learning?
Romoff, J., Henderson, P., Kanaa, D., Bengio, E., Touati, A., Bacon, P., & Pineau, J. (2021) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/mpbl3656 · Cited by 1
BiES: Adaptive Policy Optimization for Model-Based Offline Reinforcement Learning
Yang, Y., Jiang, J. B., Wang, Z., Duan, Q., & Shi, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-97546-3_46 · Cited by 1
Mahmoud, S., Billing, E., Svensson, H., & Thill, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1098982 · Cited by 1
“One of the earliest approaches that use a replay buffer (also known as experience replay) in reinforcement learning was the Dyna architecture (Sutton, 1991).”
BackgroundRTDK-BO: High Dimensional Bayesian Optimization with Reinforced Transformer Deep kernels
Shmakov, A., Naug, A., Gundecha, V., Ghorbanpour, S., Gutiérrez, R. L., Babu, A. R., Guillén, A., & Sarkar, S. (2023) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case56687.2023.10260520 · Cited by 1
“…to construct a robust acquisition function for Bayesian Optimization. d) Model-Based RL: Model-based reinforcement learning presents a method for improving sample efficiency by estimating the environment’s dynamics and allowing the policy to train from the information learned by such a model [18].”
MethodError Reducing Sampling in Reinforcement Learning
Scherrer, B., & Mannor, S. (2008) · Cited by 1
Learning the structure of activities for a mobile robot
Schmill, M. D., & Cohen, P. R. (2004) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 1
APPLICATION OF AI-BASED REINFORCEMENT LEARNING TO ROBOT VEHICLE CONTROL
Madden, M. G., & Nolan, P. (1995) · Cited by 1
Incremental topological reinforcement learning agent in non-structured environments
Braga, A. P. S., Araujo, A. F. R., & Wyatt, J. (2005) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2004.1401080 · Cited by 1
Apprentissage par renforcement hiérarchique dans les MDP factorisés
Kozlova, O., Sigaud, O., Meyer, C., & Fre, C. (2006) · Cited by 1
Fast, Scalable Algorithms for Reinforcement Learning in High Dimensional Domains
Bellemare, M. G. (2013) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r3dj58t21 · Cited by 1
Open Loop Execution of Tree-Search Algorithms, extended versionPreprint
Lecarpentier, E., Infantes, G., Lesire, C., & Rachelson, E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.01367 · Cited by 2
“Tree-search algorithms use this model in order to build a tree of what may possibly occur in the current situation of the agent [Sutton, 1991; Sutton and Barto, 1998; Silver et al., 2008].”
MethodInfluentialDyna-AIL : Adversarial Imitation Learning by PlanningPreprint
Saxena, V., Sivanandan, S., & Mathur, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03234 · Cited by 1
“Dyna [2] based methods use a combined approach wherein the policy is learned by alternating between MB and MF methods.”
MethodIterative Model-Based Reinforcement Learning Using Simulations in the Differentiable Neural ComputerPreprint
Mufti, A., Penkov, S., & Ramamoorthy, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.07248 · Cited by 1
“AI agents can be trained to perform tasks in simulations generated by a predictive model of the environment (Sutton, 1991; Gu et al., 2016; Ha & Schmidhuber, 2018).”
The Application of Machine Learning to Optimise Live Migration in Cloud Data Centres
Duggan, M. (2019) · Arrow@dit (Dublin Institute of Technology) · doi:10.13025/17880 · Cited by 1
Contextual Policy Reuse using Deep Mixture Models.Preprint
Gimelfarb, M., Sanner, S., & Lee, C.-G. (2020) · arXiv (Cornell University) · Cited by 1
Guided Dyna-Q for Mobile Robot Exploration and NavigationPreprint
Hayamizu, Y., Amiri, S., Chandan, K., Zhang, S., & Takadama, K. (2020) · arXiv (Cornell University) · Cited by 1
On the Expressivity of Neural Networks for Deep Reinforcement LearningPreprint
Dong, K., Luo, Y., & Ma, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05927 · Cited by 2
Neural Network iLQR: A New Reinforcement Learning Architecture.Preprint
Cheng, Z., Ma, J., Zhang, X., Lewis, F. L., & Lee, T. H. (2020) · arXiv (Cornell University) · Cited by 1
Metadata
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- ACM SIGART Bulletin
- Published
- 1991-07
- Volume
- 2
- Issue
- 4
- Pages
- 160-163
- ISSN
- 0163-5719 (ISSN-L)
- OpenAlex
- W1980035368
- Cited by
- 777
- References
- 18
- Citations by year
- 2026: 56 · 2025: 71 · 2024: 57 · 2023: 63 · 2022: 57 · 2021: 79 · 2020: 115 · 2019: 62 · 2018: 42 · 2017: 25 · 2016: 9 · 2015: 16 · 2014: 5 · 2013: 13 · 2012: 11
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGART Bulletin (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- learning from examples, planning, model-based learning, incremental planning, learning-based planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
