The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Dyna, an integrated architecture for learning, planning, and reacting
Abstract
Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1991
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citationCraik, K. J. W. (1943) · Cited by 1,845
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 916
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 583
Learning and Sequential Decision Making
Barto, A. G., Sutton, R. S., & Watkins, C. (1989) · Cited by 349
Why the Law of Effect will not Go Away*
Dennett, D. C. (1975) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1975.tb00350.x · Cited by 330
Research works printed but not linked: 10
- Dennett D. C., 1978, Brainstorms, by D. C. Dennett, 71--89
- Holland J. H., 1986, R. Michalski, J. Carbonell &
- Lin Ji., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, 297--305
- Riolo R., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, MIT Press.
- Russell S. J., 1989, Proceedings of IJCAI-89
- 10.1023/A:1022633531479 · DOI · Paper page
- 10.5555/101883.102055 · DOI
- Sutton R.S., 1981, An adaptive network that constructs and uses an internal model of its environment, Cognition and Brain Theory Quarterly, 4, 217
- Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department. Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department.
- 10.1023/A:1022619109594 · DOI · Paper page
Cited by
Sorted by citations
Items 601–700 of 764
The Intersection of Planning and Learning
Moerland, T. M. (2021) · Research Repository (Delft University of Technology) · doi:10.4233/uuid:5437884e-0078-4b36-b2c7-c6edfea3b418 · Cited by 1
Eden: A Unified Environment Framework for Booming Reinforcement Learning AlgorithmsPreprint
Chen, R., Wu, X., Pan, Y., Yuan, K., Li, L., Ma, T., Liang, J., Zhang, R., Wang, K., Zhang, C., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.01768 · Cited by 2
“First, the model interacts with the agent to generate more dataSutton [1991], Kurutach et al. [2018], Kalweit and Boedecker [2017], Gu et al. [2016], which can be used as an extra data source to accelerate the training process. Second, during Q-value or V-value estimation, the model is used as additional context infor…”
MethodInfluentialMathematical Modeling of Human Learning and Decision Making
Xia, L. (2021) · eScholarship (California Digital Library) · Cited by 1
Machine learning applied to prediction, control and planning from dynamic epidemiological models
Bent, O. (2020) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.5287/ora-b7qkedzkb · Cited by 1
Absraction for Efficient Reinforcement Learning
Telfar, A. (2020) · doi:10.26686/wgtn.17147651 · Cited by 3
Wang, Z., Meng, H., Zhou, Z., Feng, Y., Gao, Y., & Yu, C. (2022) · Research Square · doi:10.21203/rs.3.rs-1780336/v1 · Cited by 1
Deep reinforcement learning for electricity energy trading
Sanayha, M. (2022) · doi:10.58837/chula.the.2022.91 · Cited by 1
Muthyala, P., Mayank, M., Ünver, B., Evren, S., Uzunoğlu, B., Toğuş, F., Kaydeci, A. B., Schaub, J., Virnich, L., & Pischinger, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4826756 · Cited by 1
Unravelling Repetitive Negative Thinking With Reinforcement LearningPreprint
Bedder, R. L., Hitchcock, P., & Sharp, P. B. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/cu46a · Cited by 3
Haberland, S., Ruge, H., & Frimmel, H. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.05.20.655119 · Cited by 1
Hierarchical and factored reinforcement learning
Kozlova, O. (2010) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 1
Dopamine ramps as a normative consequence of dual-process controlPreprint
Priestley, L., & Akam, T. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.18.706579 · Cited by 2
“This contrasts with offline replay mechanisms that refine cached value estimates guiding subsequent behaviour (Mattar and Daw, 2018; Sutton, 1991).”
Learning under uncertainty: a model-based approach for understanding gambling behaviour
Yu, E. C. (2011) · Doctoral thesis, UCL (University College London).
Learning Integrated Relational and Continuous Action Models for Continuous Domains.
Xu, J. Z. (2013) · Deep Blue (University of Michigan)
Applying topological maps to accelerate reinforcement learning in mobile robot navigation
Braga, A. P. S., & Araiijo, A. F. R. (2005) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2002.1173376
A self-learning motorway traffic control system for ramp metering
Lu, C. (2014) · White Rose eTheses Online (University of Leeds, The University of Sheffield, University of York)
Temporal plannability by variance of the episode lengthPreprint
Takács, B. M., Szita, I., & Lörincz, A. (2003) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0301006
“ence, attempts were made to extend classical AI planning methods to handle uncertainty. On the other hand, operations research has made important advances recently in solving decision making problems [23]. One of the earliest works, which tried to unify the two complementary approaches appeared in the seventies [6]. R…”
BackgroundMethodA Real Time Object-Oriented Rational Agent Development System
Wesley, L. P. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45484-5_35
Enhanced Temporal Difference Learning Using Compiled Eligibility Traces
Vamplew, P. W., Ollington, R., & Hepburn, M. (2006) · Lecture notes in computer science · doi:10.1007/11941439_18
“The ability to learn about sequences of states which are possible within an environment but yet to be observed can also be provided by model-based learning methods such as the Dyna methods [ 5 ].”
MethodAvoiding failure states during reinforcement learning
Diepen, M. D. M. V. (2011) · Research Repository (Delft University of Technology)
Agente topológico de aprendizado por reforço
Braga, A. P. S. (2004) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.18.2004.tde-21102011-081848
Walking bundles of habits (and response–outcome associations) (Commentary on Liljeholm et al.)
Bornstein, A. M., Shenhav, A., & Miller, K. J. (2015) · European Journal of Neuroscience · doi:10.1111/ejn.12906
“within an explicitly serial architecture with habits built by ‘training’ from the goal-directed system (Sutton, 1991).”
BackgroundAn efficient reinforcement learning algorithm for continuous actions
Bo, F., Chen, X., He, Y., & Wu, M. (2013) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc.2013.6560898
RoMPLA: An efficient robot motion and planning learning architecture
González-Quijano, J., Abderrahim, M., Bensalah, C., & Rodríguez-Jiménez, S. (2013) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2013.6696677
“As an example, we can mention the Dyna architecture [7].”
MethodAn algorithm for finding reliably schedulable plans
Takács, B. M., Szita, I., & Lörincz, A. (2005) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2004.1380973
Policy Gradient Methods: Variance Reduction and Stochastic Convergence
Greensmith, E. (2005) · ANU Open Research (Australian National University) · doi:10.25911/5d7a2a01dcebe
Self-organization of evaluation space for planning using state discretization resolution
Kobayashi, Y., & Hosoe, S. (2005) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2004.1389650
A Learning Architecture to Support Autonomous Resource Scheduling and Allocation in the Cloud
Barrett, E. (2013) · Arrow@dit (Dublin Institute of Technology) · doi:10.13025/15830
Autonomous hierarchical POMDP planning from low-level sensors
Squire, S., & desJardins, M. (2013) · National Conference on Artificial Intelligence
A Motion Control Method Based on the Knowledge Discovery and Data Mining for Humanoid Robots
Kuwayama, K., Kato, S., & Itoh, H. (2007) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.25.402
Trajectory Sampling Value Iteration: Improved Dyna Search for MDPs (Extended Abstract)
Zhou, Y., Liu, Q., Fu, Q., & Zhang, Z. (2015)
Adaptive resolution function approximation for TD learning: simple division and integration
Kobayashi, Y., & Hosoe, S. (2003) · Society of Instrument and Control Engineers of Japan
Proposal of a Generative Model of Event-based Representations for Grounded Language Understanding
Brodeur, S., Celotti, L., & Rouat, J. (2017) · doi:10.21437/glu.2017-14
“as occurring during the time interval of [25,30] seconds) when such interaction event is linked with large variations in linear accelerations.”
BackgroundModel-Based Action Exploration.Preprint
Berseth, G., & van de Panne, M. (2018) · arXiv (Cornell University)
Deep Reinforcement Learning with Hidden Layers on Future States
Kameko, H., Suzuki, J., Mizukami, N., & Tsuruoka, Y. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-75931-9_4
Utilization of Smartphone Technology of Students in Making Videos On Motorcycle Chassis Learning
Firmansyah, R., Hamdani, R. A., Kuswardhana, D., & Kusuma, I. H. (2019) · UPI International Conference on Technical and Vocational Education and Training (ICTVET) · doi:10.2991/ictvet-18.2019.44
“The initial stage in the implementation of smartphonebased learning is to make learning plans, so that they can determine learning methods and learning strategies that will be used in order to create a reactive learning process [15].”
BackgroundImproving Deep Reinforcement Learning Using Graph Convolution and Visual Domain Transfer
Niu, S. (2018) · TigerPrints (Clemson University)
Combined use of reinforcement learning and simulated annealing
Péter, S. (2003)
Adversarial Imitation Learning between Agents with Different Numbers of State Dimensions
Taketo, Y., & Kuniyoshi, Y. (2019) · IEEE International Conference on Artificial Intelligence and Knowledge Engineering (AIKE) · doi:10.1109/aike.2019.00040
“GCL is based on the guided policy search [19] [20], which is one of the methods of modelbased reinforcement learning [21] [22] [23] [24] [25] [26], and is thus is a model-based approach of IRL.”
MethodOptimal Learning Theory and Approximate Optimal Learning Algorithms
Song, H. (2019) · UWSpace (University of Waterloo)
Hasz, B. M. (2020) · University of Minnesota Digital Conservancy (University of Minnesota)
Domain Knowledge Integration By Gradient Matching For Sample-Efficient Reinforcement LearningPreprint
Chadha, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13778
“t al. (2000)), however, it often requires large amounts of interaction with the environment and doesn’t generalize across tasks with similar environmental dynamics. On the other hand, model-based RL (Sutton (1991)) attempts to learn a model of an environment from experience gained during training and uses this model t…”
Approximation Methods for Monte Carlo Tree Search
Aksenov, K., & Panov, A. I. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-50097-9_8
A Hybrid PAC Reinforcement Learning AlgorithmPreprint
Zehfroosh, A., & Tanner, H. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02602
“The search for a connection between model-free and model-based RL algorithms begins with the Dyna-Q algorithm [15], in which synthetic generated experiences based on the learned model are used to expedite Q-learning.”
Dynamic Horizon Value Estimation for Model-based Reinforcement LearningPreprint
Wang, J., Zhang, Q., Zhao, D., Zhao, M., & Hao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09593 · Cited by 1
“For model-based RL methods, a dynamics model is usually bulit to help the learning process of an agent in two directions: direct policy learning (Sutton 1991) and value expansion (Feinberg et al. 2018).”
NEARL: Non-Explicit Action Reinforcement Learning for Robotic ControlPreprint
Lin, N., Li, Y., Zhu, Y., Wang, R., Zhang, X., Ji, J., Tang, K., Chen, X., & Zhang, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01046
“For example, Dyna [17] learns policy from both real and simulated experience; more recently, Guided Policy Search [9], [10] uses guiding samples to optimize the policy search.”
Learning from limited experience: Real-world robot navigation from a single traversal
Bruce, J. (2020) · Queensland University of Technology · doi:10.5204/thesis.eprints.205492
Habituation and goal-directed arbitration mechanisms and failures under partial observabilityPreprint
Sánchez-Fibla, M. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.11.24.396630
“(a) Dyna-Q [31] is the prototypical Model-Based algorithm.”
MethodModel-Augmented Q-learningPreprint
Oh, Y., Shin, J., Yang, E., & Hwang, S. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.03866 · Cited by 1
“There are different types of Model-based RL (MBRL), and the most popular approach is Dyna-style (Sutton, 1991), in which algorithms generate fictitious experiences to train agents.”
Robust and Efficient Planning using Adaptive Entropy Tree Search.Preprint
Kozakowski, P., Pacek, M., & Miłoś, P. (2021) · arXiv (Cornell University)
Machine Learning in Algorithmic Trading leading to Reinforced Deep Kalman Filters
Ferreira, T. A. (2020) · TSpace (University of Toronto)
Planning with Expectation Models for ControlPreprint
Kudashkina, K., Wan, Y., Naik, A., & Sutton, R. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08543
“Learned parametric models are used in the Dyna architecture (Sutton, 1991), in normalized advantage functions method that incorporates the learned model into the Q-learning algorithm based on imagination rollouts (Gu et al.”
MethodA Feature Analysis Tool for Batch RL Datasets
Xu, R., & Chen, Z. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442442.3453147
Driving exploration through particle Q-distributions
Likmeta, A. (2018)
Zhang, W., Xiao, X., Yao, Y., Chen, M., & Luo, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01295
“…deployed in the environment P , i.e., V π, P , has a discrepancy bound: where and Since MBDP algorithm is an extension of the Dyna-style algorithm [27]: a series of model-based reinforcement learning methods which jointly optimize the policy and transition model, it can be written in a general…”
MethodLearning Predictive Models for Efficient Policy Learning
Xu, H. (2021) · eScholarship (California Digital Library)
Deep Reinforcement Learning Survey from the Peopective of Action Planning
Ariki, Y. (2021) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.39.601
Trust, but Verify: Alleviating Pessimistic Errors in Model-Based Exploration
Czechowski, K., Odrzygóźdź, T., Izworski, M., Zbysiński, M., Kuciński, Ł., & Miłoś, P. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534062
Planning and Learning using Adaptive Entropy Tree Search
Kozakowski, P., Pacek, M., & Miłoś, P. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892556
Multitask Adaptation by Retrospective Exploration with Learned World ModelsPreprint
Zholus, A., & Panov, A. I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13241
“…were proposed for using the environment model in the agent learning process: to generate additional “virtual" agent steps in the environment [17], to simulate full episodes to more accurately approximate the Q-value [18, 14], and to directly learn the policy only on “imaginary" trajectories…”
ORACLE: End-to-End Model Based Reinforcement Learning
Andersen, P., Goodwin, M., & Granmo, O. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91100-3_4
Multi-step Prediction for Learning Invariant Representations in Reinforcement Learning
Xu, X., Lv, K., Dong, X., Han, S., & Lin, Y. (2021) · International Conference on High Performance Big Data and Intelligent Systems (HPBD&IS) · doi:10.1109/hpbdis53214.2021.9658436
Setoodeh, P., Habibi, S. R., & Haykin, S. (2022) · Wiley eBooks · doi:10.1002/9781119078166.ref
Koenig, S., & Simmons, R. G. (2007) · Springer eBooks · doi:10.1007/978-0-585-33656-5_10
(2011) · Elsevier eBooks · doi:10.1016/b978-0-12-372512-7.00020-1
Faculty Opinions recommendation of Fast Sequences of Non-spatial State Representations in Humans.
Kiebel, S. J. (2019) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.726435693.793559771
Model Learning and Tactical Maneuver Planning for Automatic Driving
Helbig, M., Hoedt, J., & Konigorski, U. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-19-1610-6_62
Reducing the Planning Horizon Through Reinforcement Learning
Dunbar, L., Rosman, B., Cohn, A. G., & Leonetti, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26412-2_5
“[8] showed that a simple implementation of the DYNA architecture [26] was able to replicate their psychological findings, lending credence to the idea of a cooperative model-based and model-free system.”
Deep Q-Network Updates for the Full Action-Space Utilizing Synthetic Experiences
von Pilchau, W. P., Pätzel, D., Stein, A., & Hähner, J. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191853
“A lot of work has been done in the area of model-based RL. Dyna-Q [8] generates a tabular model of the environment and was extended to a version that utilizes NNs instead [9].”
BackgroundReinforcement Learning with Goal Relabeling and Dynamic Model for Robotic Tasks
Dong, K., Luo, Y., Wang, Y., Fang, S. C., Liu, Y., Cheng, E., Sun, Z., Zhang, Q., & Song, B. (2023) · IEEE International Conference on Real-time Computing and Robotics (RCAR) · doi:10.1109/rcar58764.2023.10250060
“Different from the standard model-based reinforcement that only use the data generated from the dynamic network model to train the policy, the policy in this paper is optimized using both real data and model-generated data which can be seen as a Dyna-style method [24], [25].”
MethodNavigating Black Swan Events in Algorithmic Trading: A Reinforcement Learning Perspective
Díaz, F. V., & Guerrero-Mosquera, C. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230667
MA-TDMPC: Multi-Agent Temporal Difference for Model Predictive Control
Wang, R., Hao, X., Feng, Y., Huang, J., & Zhang, P. (2023) · International Conference on Machine Learning, Cloud Computing and Intelligent Mining (MLCCIM) · doi:10.1109/mlccim60412.2023.00043
“In terms of model utilization, Model Predictive Control (MPC) methods have some advantages over Dyna-like methods [4], such as higher computational efficiency and online optimization which enables itself to dynamically adapt to environmental changes.”
MethodCooperative control strategy of shared energy storage system for smart grid
Pagri, M. I. F., Hwang, J., Kim, B., & Suh, D. (2023) · Building Simulation Conference proceedings · doi:10.26868/25222708.2023.1586
GPTMORE: Generative Pre-trained Transformer for Model-based Offline Reinforcement Learning
Li, Y., & Wang, P. (2023) · Asia Conference on Cognitive Engineering and Intelligent Interaction (CEII) · doi:10.1109/ceii60565.2023.00032
Mitigating Value Hallucination in Dyna-Style Planning via Multistep Predecessor Models
Aminmansour, F., Jafferjee, T., Imani, E., Talvitie, E. J., Bowling, M., & White, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15155 · Cited by 9
“Early work on Dyna considered environments with deterministic dynamics and finite state spaces (Sutton, 1990, 1991).”
Mutschler, C., Kontes, G. D., & Rietsch, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-64832-8_3 · Cited by 1
Model Based Reinforcement Learning Pre-Trained with Various State Data
Ono, M., & Ichise, R. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00169
“World Models in MBRL One of the pivotal advances in the RL domain has been the integration of world models in MBRL [8], [10].”
BackgroundA Pedestrian Trajectory Prediction Model Based on Generative Adversarial Mimicry Learning
Li, M., Fei, R., & Li, A. (2024) · International Conference on Machine Intelligence and Digital Applications (MIDA) · doi:10.1145/3662739.3662757
Policy Optimization by Looking Ahead for Model-based Offline Reinforcement Learning
Liu, Y., & Hofert, M. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610966
“In contrast, model-based methods like Dyna [14] learn a transition (dynamics) model of the environment in a supervised way, thus providing extra information about the environment.”
Method单, 国. (2024) · Software Engineering and Applications · doi:10.12677/sea.2024.135072
Advanced Applications of Symbol-Based AI: Planning and Learning
Luger, G. F. (2024) · Springer eBooks · doi:10.1007/978-3-031-57437-5_11
García, J., Rañó, I., Burés, J. M., Fdez-Vidal, X. R., & Iglesias, R. (2024) · Applied Intelligence · doi:10.1007/s10489-024-06190-7
“Therefore, Algorithm 1 assumes there is a previous learning process in the source task where a policy π and a transition model ˆ T s are learned at the same time, e.g., using Dyna approach [31, 32].”
Quality-Aware Experience Exploitation in Model-Based Reinforcement Learning
Yang, G., Li, J., Geng, Z., & Luo, C. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825395
“Many experience generation methods, which are known as the Dyna-style method [22], have leveraged learned dynamics models to generate simulated experiences.”
Path Planning of Unmanned Surface Vessel Based on Improved Dyna-Sarsa Algorithm
Zhao, P., Zhao, E., Yang, Y., & Yuan, K. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-2248-1_12
Robust Ensembled Model Based Reinforcement Learning for Better Tomato Yield in Greenhouse
Gandhimathi, S., Senthilkumaran, B., Devakanth, J. J. M. A., Nithya, K., & Chinnadurai, J. (2025) · Journal of Machine and Computing · doi:10.53759/7669/jmc202505090
Zhao, K., Hao, Y., & Liang, Z. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.121083
Comparative Study on Curiosity with Attention, Memory and Empowerment
Duarte, F. F., Lau, N., Pereira, A., & Reis, L. P. (2025) · IEEE International Conference on Autonomous Robot Systems and Competitions · doi:10.1109/icarsc65809.2025.10970179
“The objective of RL is to derive a policy that maximizes the expected (usually discounted) sum of these reward signals [1].”
Zhang, Q., Zhao, F., Wang, T., Zhao, J., & Wang, W. (2025) · IEEE Data Driven Control and Learning Systems Conference (DDCLS) · doi:10.1109/ddcls66240.2025.11065680
“In recent years, Dyna-Q [9] is the earliest algorithm to be proposed in the research of joint model-based and model-free RL.”
Neural Motion Simulator Pushing the Limit of World Models in Reinforcement Learning
Hao, C., Lu, W., Xu, Y., & Chen, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02571
“A world model [11, 23, 38] is an inner model that simulates how the world evolves and in essence it does one thing: given the current state and action, it predicts the future state [38].”
Memory and Planning in Brains and Machines
Momennejad, I. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192882547.003.0012 · Cited by 12
Learning Active Tactile Perception Through Belief-Space Control
Tremblay, J.-F., Meger, D., Hogan, F. R., & Dudek, G. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11127425
“Model-based reinforcement learning: Model-based re-inforcement learning methods [10]–[12] learn a dynamics model for the system and use it to perform control (either through a sample based controller or by leveraging the model to learn a value function), similarly to our method.”
Hybrid reward-driven reinforcement learning for efficient quantum circuit synthesis
Giordano, S., Sen, K., & Martin-Delgado, M. (2025) · Quantum Machine Intelligence · doi:10.1007/s42484-026-00359-8 · Cited by 6
“Our training structure aligns with Q-learning based on action sequences and planning-inspired exploration, similar to models like Dyna-Q [32]–[34].”
Asymmetric learning and adaptability to changes in relational structure during transitive inference
Graham, T. A., & Spitzer, B. (2025) · Communications Psychology · doi:10.1038/s44271-025-00352-0
Personalizing Dialog Policy in Health Guidance via Model-based Multitask Reinforcement Learning
Sano, T., Kohjima, M., Takahashi, M., Fujimura, K., Sato, T., & Katagiri, Y. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228619
Leveraging World Model Disentanglement in Value-Based Multi-Agent Reinforcement Learning
Wang, Z., & Meger, D. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228644
“These methods optimize the policy by directly learning from the model-based imaginary experiences, in a style analogous to the Dyna method [14].”
Metadata
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- ACM SIGART Bulletin
- Published
- 1991-07
- Volume
- 2
- Issue
- 4
- Pages
- 160-163
- ISSN
- 0163-5719 (ISSN-L)
- OpenAlex
- W1980035368
- Cited by
- 777
- References
- 18
- Citations by year
- 2026: 56 · 2025: 71 · 2024: 57 · 2023: 63 · 2022: 57 · 2021: 79 · 2020: 115 · 2019: 62 · 2018: 42 · 2017: 25 · 2016: 9 · 2015: 16 · 2014: 5 · 2013: 13 · 2012: 11
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGART Bulletin (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- learning from examples, planning, model-based learning, incremental planning, learning-based planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
