The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Dyna, an integrated architecture for learning, planning, and reacting
Abstract
Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1991
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citationCraik, K. J. W. (1943) · Cited by 1,845
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 916
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 583
Learning and Sequential Decision Making
Barto, A. G., Sutton, R. S., & Watkins, C. (1989) · Cited by 349
Why the Law of Effect will not Go Away*
Dennett, D. C. (1975) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1975.tb00350.x · Cited by 330
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Research works printed but not linked: 10
- Dennett D. C., 1978, Brainstorms, by D. C. Dennett, 71--89
- Holland J. H., 1986, R. Michalski, J. Carbonell &
- Lin Ji., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, 297--305
- Riolo R., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, MIT Press.
- Russell S. J., 1989, Proceedings of IJCAI-89
- 10.1023/A:1022633531479 · DOI · Paper page
- 10.5555/101883.102055 · DOI
- Sutton R.S., 1981, An adaptive network that constructs and uses an internal model of its environment, Cognition and Brain Theory Quarterly, 4, 217
- Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department. Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department.
- 10.1023/A:1022619109594 · DOI · Paper page
Cited by
Sorted by citations
Items 401–500 of 764
Open Loop Execution of Tree-Search Algorithms
Lecarpentier, E., Infantes, G., Lesire, C., & Rachelson, E. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/327 · Cited by 7
Uncertainty maximization in partially observable domains: A cognitive perspective
Ramičić, M., & Bonarini, A. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.02.044 · Cited by 5
Intelligent Security Aware Routing: Using Model-Free Reinforcement Learning
Mudgerikar, A., & Bertino, E. (2023) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn58024.2023.10230195 · Cited by 5
“Formally, STE-SDN can be defined as a model-based RL framework based on the Dyna-Q framework [9], where: (i) a part of the model of the environment is learnt, in terms of the placement and types”
MethodAn indirect reinforcement learning approach for ramp control under incident-induced congestion
Lu, C., Chen, H., & Grant‐Muller, S. (2013) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2013.6728359 · Cited by 5
Pricing in Prosumer Aggregations using Reinforcement Learning
Agwan, U., Spangher, L., Arnold, W., Srivastava, T., Poolla, K. R., & Spanos, C. J. (2021) · ACM International Conference on Future Energy Systems (e-Energy) · doi:10.1145/3447555.3464853 · Cited by 5
“regression or neural nets), could train with the agent and help augment data [22].”
BackgroundA Fast Reinforcement Learning Technique via Multiple Lookahead Levels.
Al-Dayaa, H., & Megherbi, D. B. (2006) · MLMTA · Cited by 5
Al-Dayaa, H., & Megherbi, D. B. (2012) · The Journal of Supercomputing · doi:10.1007/s11227-011-0738-6 · Cited by 5
Machine Learning and System Identification for Estimation in Physical SystemsPreprint
Carlson, F. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02003 · Cited by 9
“, 2000], 3[Watkins and Dayan, 1992], 4[Sutton, 1991], 5[Silver et al.”
Symbolic Regression for Constructing Analytic Models in Reinforcement LearningPreprint
Derner, E., Kubalı́k, J., Ancona, N., & Babuška, R. (2019) · arXiv (Cornell University) · Cited by 5
Baconian: A Unified Open-source Framework for Model-Based Reinforcement LearningPreprint
Dong, L., Gao, G., Zhang, X., Chen, L., & Wen, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.10762 · Cited by 6
“The other one is Dyna-like control flow [Sutton, 1991”
Meta-Model-Based Meta-Policy OptimizationPreprint
Hiraoka, T., Imagawa, T., Tangkaratt, V., Osa, T., Onishi, T., & Tsuruoka, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02608 · Cited by 7
“, 2019) Branched rollouts are Dyna-style rollouts (Sutton, 1991), in which model-based rollouts are run as being branched from real trajectories.”
BackgroundPhysically Embedded Planning Problems: New Challenges for Reinforcement LearningPreprint
Mirza, M., Jaegle, A., Hunt, J. J., Guez, A., Tunyasuvunakool, S., Muldal, A., Weber, T., Karkus, P., Racanière, S., Buesing, L., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05524 · Cited by 12
“…instance, research in model-based planning (e.g., Chua et al., 2018; Co-Reyes et al., 2018; Finn and Levine, 2017; Silver et al., 2016, 2017, 2018; Sutton, 1991; Wang and Ba, 2020) studies how models can be used to identify good actions either to update the policy or to structure interactions…”
BackgroundAnalog Circuit Design with Dyna-Style Reinforcement LearningPreprint
Lee, W., & Oliehoek, F. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07665 · Cited by 5
“To further enhance the sample efficiency, a unified Dyna-style optimization scheme is proposed by intermixing both the model-free and model-based RL methods [8].”
MethodLearning Dynamics Models for Model Predictive AgentsPreprint
Lutter, M., Hasenclever, L., Byravan, A., Dulac-Arnold, G., Trochim, P., Heess, N., Merel, J., & Tassa, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.14311 · Cited by 27
“Model-based RL algorithms use the model as a simulator to generate additional data [39, 2, 40], to evaluate the reward of an action sequence [1, 18, 41, 19], to improve the value function estimate [42, 21] or backpropagate the policy gradients through time [43, 44, 45, 20, 46].”
Learning Pessimism for Robust and Efficient Off-Policy Reinforcement LearningPreprint
Cetin, E., & Celiktutan, O. (2021) · arXiv.org · doi:10.48550/arxiv.2110.03375 · Cited by 7
“This algorithm learns a large ensemble of world models with Dyna-style (Sutton 1991) optimization to train the policy.”
MethodProcedural Generalization by Planning with Self-Supervised World Models
Anand, A., Walker, J., Li, Y., Vértes, E., Schrittwieser, J., Ozair, S., Weber, T., & Hamrick, J. B. (2021) · arXiv · doi:10.48550/arxiv.2111.01587 · Cited by 5
“Third, the model can also be used to train a policy or value function by simulating training data [66] or constructing more informative losses [21, 30], again enabling faster learning (possibly entirely in simulation).”
BackgroundMethodReinforcement Twinning: From Digital Twins to Model-Based Reinforcement LearningPreprint
Schena, L., Marques, P. A. D. M., Poletti, R., Ahizi, S., den Berghe, J. V., & Mendez, M. A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4761240 · Cited by 6
Critic as Lyapunov function (CALF): a model-free, stability-ensuring agent
Osinenko, P., Yaremenko, G., Zashchitin, R., Bolychev, A., Ibrahim, S., & Dobriborsci, D. (2024) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc56724.2024.10886034 · Cited by 4
“Notice, not every reinforcement learning agent is model-free, e. g., Dyna [6], [7].”
Qu, K., Zhang, H., Zhou, X., Ferrando, M., & Causone, F. (2025) · Building Simulation · doi:10.1007/s12273-025-1288-9 · Cited by 5
“Meanwhile, the agent refines its policy using a mix of real and synthetic data (Sutton 1991).”
Guided Cooperation in Hierarchical Reinforcement Learning via Model-Based Rollout
Wang, H., Tang, Z., Sun, Y., Wang, F., Zhang, S., & Chen, Y. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3425809 · Cited by 4
“The well-known MBRL algorithm, Dyna [43], leveraged a dynamics model to generate one-step transitions and then updated the value function using these imagined data, thus accelerating learning.”
Memory consolidation from a reinforcement learning perspective
Lee, J. W., & Jung, M. W. (2025) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1538741 · Cited by 4
“To address these challenges, the Dyna algorithm, introduced by Richard Sutton in 1991, provides an integrated approach that significantly enhances learning efficiency.”
Looking deeper into the algorithms underlying human planning
Kuperwajs, I., Russek, E. M., Mattar, M. G., Ma, W. J., & Griffiths, T. L. (2025) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2025.06.006 · Cited by 5
“A related problem is how these systems can bene fi t from each other ’ s computations, for which one solution is to allow the model-free system to learn not only from direct experience, but also from computations made by the model-based system [17,18].”
SR-AIF: Solving Sparse-Reward Robotic Tasks From Pixels with Active Inference and World Models
Nguyen, V. D., Yang, Z., Buckley, C. L., & Ororbia, A. G. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11127713 · Cited by 10
“From the ‘Dyna’ framework [76], model-based RL has evolved into powerful modern-day models, including latent dynamics models [6], imagination-augmented RL [65], or recurrent-based state dynamics models capable of playing Atari games [89].”
MethodA Fast Learning Agent Based on the Dyna Architecture
Hsu, Y.-P., & Jiang, W. (2014) · doi:10.6688/jise.2014.30.6.8 · Cited by 4
“4 shows the general architecture of a Dyna agent in [1] [2].”
BackgroundMethodInfluentialCMAX++ : Leveraging Experience in Planning and Execution using Inaccurate Models
Vemula, A., Bagnell, J. A., & Likhachev, M. A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16765 · Cited by 4
Learning Controllers for Adaptive Spreading of Carbon Fiber Tows
Krützmann, J., Schiendorfer, A., Beratz, S., Moosburger‐Will, J., Reif, W., & Horn, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64580-9_6 · Cited by 4
Counterexample-Guided Repair for Symbolic-Geometric Action Abstractions
Thomason, W., & Kress‐Gazit, H. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3294918 · Cited by 4
“Although this is a rich area of research [29], [30], [31], [32], [33], [34], [35], we learn a different kind of model than those typically considered.”
BackgroundModel-Based Reinforcement Learning with Automated Planning for Network Management
Ordóñez, A., Rendón, O. M. C., Jácome, W. F. V., Rodriguez-Vivas, A., & da Fonseca, N. L. S. (2022) · Sensors · doi:10.3390/s22166301 · Cited by 4
Strategy Synthesis in Markov Decision Processes Under Limited Sampling Access
Christel Baier, A., Dubslaff, C., Wienhöft, P., & Kiebel, S. J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33170-1_6 · Cited by 4
Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuring
Krale, M., Simão, T. D., & Jansen, N. (2023) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v33i1.27197 · Cited by 4
A Deep Reinforcement Learning-based Routing Algorithm for Unknown Erroneous Cells in DMFBs
Kawakami, T., Shiro, C., Nishikawa, H., Kong, X., Tomiyama, H., & Yamashita, S. (2023) · IEEE Interregional NEWCAS Conference (NEWCAS) · doi:10.1109/newcas57931.2023.10198067 · Cited by 4
“Reinforcement Learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward [11].”
BackgroundYu, Y., Yang, R., Lyu, J., Yan, J., Luo, F., Luo, D., Li, X., & Li, L. (2023) · International Conference on Frontiers of Robotics and Software Engineering (FRSE) · doi:10.1109/frse58934.2023.00028 · Cited by 4
“Dyna [24] utilizes a trained model to generate virtual transitions for accelerating learning value functions.”
MethodHuang, Y., Ren, B., Xu, Z., & Wu, L. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650959 · Cited by 4
“Inspired by humans’ ability to predict future trajectories, Dyna [13] was the first model-based RL al-gorithm to increase training data by generating virtual samples with a learned dynamics model.”
MethodAn adaptive inventory control for a supply chain
Xu, J., Zhang, J., & Liu, Y. (2009) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc.2009.5195218 · Cited by 4
“Currently, prioritized sweeping [21] and Dyna-Q [22] are known as representative learning architectures for boosting the learning speed.”
BackgroundSchneider, M. (2018) · Cognitive Systems Research · doi:10.1016/j.cogsys.2018.07.005 · Cited by 4
Wang, T., Lu, W., & Liu, D. (2018) · UTS ePRESS (University of Technology Sydney) · Cited by 4
Shaping Model-Free Habits with Model-Based Goals.
Krueger, P. M., & Griffiths, T. L. (2018) · eScholarship (California Digital Library) · Cited by 4
Fine-Tuning Offline Reinforcement Learning with Model-Based Policy Optimization
Villaflor, A., Dolan, J. M., & Schneider, J. (2021) · Cited by 4
Wen, J., Zhao, Z., Cui, J., & Chen, B. M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30108-7_27 · Cited by 5
Reinforcement Learning for Manipulation Using Constraint between Object and Robot
Kobayashi, Y., Fujii, H., & Hosoe, S. (2006) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2005.1571256 · Cited by 4
“Another possibility to accelerate reinforcement learning is to have an explicit approximation of system dynamics and reward function [6].”
BackgroundMethodFlexible Heuristic Dynamic Programming for Reinforcement Learning in Quad-Rotors
Helmer, A., de Visser, C. C., & Kampen, E.-J. V. (2018) · AIAA Information Systems-AIAA Infotech @ Aerospace · doi:10.2514/6.2018-2134 · Cited by 4
Koenig, S., & Simmons, R. G. (1996) · Machine Learning · doi:10.1023/a:1018068507504 · Cited by 4
Cooperative learning and planning for multiple robots
van der Zwaan, S., Moreira, J. A. A., & Lima, P. U. (2002) · International Symposium on Intelligent Control · doi:10.1109/isic.2000.882949 · Cited by 4
Feliu, J. L. (2013) · doi:10.23860/thesis-feliu-jose-2013 · Cited by 4
Kobayashi, Y., Yuasa, H., & Hosoe, S. (2004) · Transactions of the Society of Instrument and Control Engineers · doi:10.9746/sicetr1965.40.849 · Cited by 4
Is prioritized sweeping the better episodic control?Preprint
Brea, J. (2017) · arXiv.org · doi:10.48550/arxiv.1711.06677 · Cited by 8
“without interaction with the environment (”trying things in your head” [9, 5]).”
BackgroundMethodThe LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement LearningPreprint
van Seijen, H., Nekoei, H., Racah, E., & Chandar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03158 · Cited by 4
“The other option is to use the model in Dyna-style [18] where the model is used to generate samples which will be used to improve the model-free components of the algorithm.”
BackgroundMethodSelective Dyna-style Planning Under Limited Model CapacityPreprint
Abbas, Z., Sokota, S., Talvitie, E. J., & White, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02418 · Cited by 7
“Dyna (Sutton, 1991) is an approach to model-based reinforcement learning that combines learning from real experience and experience simulated from a learned model.”
BackgroundRevisiting Design Choices in Model-Based Offline Reinforcement Learning.Preprint
Lu, C., Ball, P., Parker-Holder, J., Osborne, M. A., & Roberts, S. (2021) · arXiv (Cornell University) · Cited by 4
Chen, H. I., & van der Meer, M. A. A. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.05.09.593332 · Cited by 6
“…upcoming behavior (Eldar et al., 2020; Ólafsdóttir et al., 2017; Pfeiffer & Foster, 2013) and the idea that replay implements “fictive learning” that fixes those reward prediction errors in value maps that are most useful for future behavior (Mattar & Daw, 2018; Sagiv et al., 2025; Sutton, 1991) .”
A Survey on Safe Model Predictive Control and Reinforcement Learning
Ham, H., & Ahn, H. (2025) · Journal of Institute of Control Robotics and Systems · doi:10.5302/j.icros.2025.24.0264 · Cited by 3
Data-assimilated model-informed reinforcement learning
Ozan, D. E., Nóvoa, A., Rigas, G., & Magri, L. (2025) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2025.0476 · Cited by 3
“In contrast to model-driven control, this model is used to generate additional samples of the environment for training, such as the Dyna algorithm [24], or to plan over finite horizons, i.e., the agent uses the model to simulate the consequences of candidate action sequences, updating its policy…”
Salim, A., Aziz, N. F. B. A., Mohammed, K. K., Rahmat, N. A., & Mekhilef, S. (2026) · Journal of Power Sources · doi:10.1016/j.jpowsour.2026.239752 · Cited by 3
A Hybrid PAC Reinforcement Learning Algorithm for Human-Robot Interaction
Zehfroosh, A., & Tanner, H. G. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.797213 · Cited by 5
“The search for a connection between model-free and modelbased RL algorithms begins with the Dyna-Q algorithm (Sutton, 1991), in which synthetic generated experiences based on the learned model are used to expedite Q-learning.”
MethodData-efficient Deep Reinforcement Learning for Vehicle Trajectory Control
Frauenknecht, B., Ehlgen, T., & Trimpe, S. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10422451 · Cited by 3
“Fourth, dyna-style algorithms [25], [52]–[54] follow the scheme of Dyna-Q [55], where synthetic experience generated by a transition model is used to train a model-free learner.”
MethodWerner, S., & Peitz, S. (2024) · European Control Conference (ECC) · doi:10.23919/ecc64448.2024.10590945 · Cited by 3
“A popular approach to tackle these issues is the usage of model-based algorithms [7], see also [8] for an overview and a taxonomy.”
BackgroundXia, L., Chen, S., Zhou, W., Wang, Q., & Xu, J. (2025) · Memetic Computing · doi:10.1007/s12293-025-00468-0 · Cited by 3
“Using the Dyna-Q model[8] as a learner, the algorithm analyzes the state of the current solution population during each iteration and adaptively selects the most appropriate evolutionary algorithm to effectively balance exploration and exploitation.”
Towards a supervised dyna-Q application on a robotic manipulator
Skoglund, A., Palm, R., & Duckett, T. (2005) · Cited by 3
A hybrid reinforcement learning algorithm for policy-based autonomic management
Wang, Z., Qiu, X., & Wang, T. (2012) · International Conference on Service Systems and Service Management · doi:10.1109/icsssm.2012.6252294 · Cited by 3
AI based intelligent system on the EDISON platform
Ma, J., Park, S. C., Shin, J. H., Kim, N. G., Seo, J. H., Lee, J. S. R., & Sa, J. H. (2018) · Artificial Intelligence and Cloud Computing Conference (AICCC) · doi:10.1145/3299819.3299843 · Cited by 3
“A specific action changes the state of the world, and the agent receives a reward [50].”
BackgroundTowards Self-confidence-based Adaptive Learning for Lunar Exploration
Mellinkoff, B., Ahmed, N. R., & Burns, J. O. (2020) · AIAA SciTech Forum · doi:10.2514/6.2020-1378 · Cited by 3
Model-Based Reinforcement Learning
Plaat, A. (2022) · Springer eBooks · doi:10.1007/978-981-19-0638-1_5 · Cited by 3
Visual Foresight with a Local Dynamics Model
Kohler, C., & Platt, R. (2023) · Springer proceedings in advanced robotics · doi:10.1007/978-3-031-25555-7_6 · Cited by 3
Reinforcement learning for finance: A review
Leon, D. (2023) · ODEON · doi:10.18601/17941113.n24.02 · Cited by 4
“There is the E-greedy strategy, optimistic initial values, action selection methods based on the Boltzmann distribution, interval estimation method, exploration bonus used in Dyna, and competition maps (Sutton, 1990, 1991).”
MethodDiffusion-Based Approximate Value Functions
Klissarov, M., & Precup, D. (2018) · Cited by 3
A New Sample-Efficient PAC Reinforcement Learning Algorithm
Zehfroosh, A., & Tanner, H. G. (2020) · Mediterranean Conference on Control and Automation (MED) · doi:10.1109/med48518.2020.9182985 · Cited by 3
“The results reported in this paper are along an established line of research that seeks connections between model-free and model-based RL algorithms, and which has begun with the the Dyna-Q algorithm [17].”
When will's wont wants wanting
Dayan, P. (2021) · Behavioral and Brain Sciences · doi:10.1017/s0140525x20001508 · Cited by 2
Semi-model-Based Reinforcement Learning in Organic Computing Systems
von Pilchau, W. P., Stein, A., & Hähner, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21867-5_16 · Cited by 3
Reinforcement learning in policy-driven autonomic management
Bahati, R. M., & Bauer, M. (2008) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms.2008.4575242 · Cited by 3
An Architectural Framework for Integrated Multiagent Planning, Reacting, and Learning
Weiß, G. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44631-1_22 · Cited by 3
Shirinov, E., & Butz, M. V. (2009) · IEEE Symposium on Artificial Life (ALIFE) · doi:10.1109/alife.2009.4937696 · Cited by 3
“The base system that is somewhat closest to our approach is the DYNA-PI architecture, which combines the learning of a (originally tabular) state-transition model of the environment with a policy learning system, which uses experiences from the state-transition model in a dynamic programming-like fashion to improve it…”
BackgroundMethodTasfi, N., & Capretz, M. A. M. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533459 · Cited by 4
Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction
Tang, H., Meng, Z., Chen, G., Chen, P., Chen, C., Yang, Y., Zhang, L., Liu, W., & Hao, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i11.17182 · Cited by 4
Discover Life Skills for Planning as Bandits via Observing and Learning How the World Works
Lai, T. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981158 · Cited by 4
Model-Based Action Exploration for Learning Dynamic Motion Skills
Berseth, G., Kyriazis, A., Zinin, I., Choi, W., & van de Panne, M. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593588 · Cited by 3
“There has been success in incorporating model-based methods to generate synthetic data or locally approximate the dynamics [15], [16], [17].”
MethodPolicy Search Based Relational Reinforcement Learning using the Cross-Entropy Method
Sarjant, S. (2013) · Research Commons (University of Waikato) · Cited by 3
Al-Dayaa, H., & Megherbi, D. B. (2006) · Cited by 3
Sample Complexity of Estimating the Policy Gradient for Nearly Deterministic Dynamical SystemsPreprint
Bastani, O. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08562 · Cited by 2
“Leveraging model-based reinforcement learning has been proposed as a potential way to reduce variance (Sutton, 1991; Deisenroth & Rasmussen, 2011; Chebotar et al., 2017; Ha & Schmidhuber, 2018).”
BackgroundDisentangling Dynamics and Returns: Value Function Decomposition with Future PredictionPreprint
Tang, H., Hao, J., Chen, G., Chen, P., Meng, Z., Yang, Y., & Wang, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11100 · Cited by 4
“The idea of future prediction is also studied in model-based RL [2, 26].”
BackgroundLearning Powerful Policies by Using Consistent Dynamics ModelPreprint
Sodhani, S., Goyal, A., Deleu, T., Bengio, Y., Levine, S., & Tang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04355 · Cited by 3
“One of the earliest examples of this interaction is the classic Dyna algorithm (Sutton, 1991), which takes advantage of the model of the environment to generate simulated experiences, which get included in the training data of a model-free algorithm (like Q-learning, with Dyna-Q).”
Planning in entropy-regularized Markov decision processes and gamesPreprint
Grill, J.-B., Domingues, O. D., Ménard, P., Munos, R., & Valko, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.19695 · Cited by 3
“An agent thinks using a world model that it has built from prior experience [Sutton, 1991, Sutton and Barto, 2018].”
Deep Adversarial Koopman Model for Reaction-Diffusion systemsPreprint
Balakrishnan, K., & Upadhyay, D. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05547 · Cited by 3
“Even the field of model-based Reinforcement Learning [33, 34] can use a Koopman-type model for learning the system dynamics p ( x t +1 | x t , a t ) .”
MethodContinuous Control for Searching and Planning with a Learned ModelPreprint
Yang, X., Duvaud, W., & Wei, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07430 · Cited by 6
“A classic approach is to directly model the dynamics of the observations [31, 9, 39, 11].”
BackgroundControl as Hybrid InferencePreprint
Tschantz, A., Millidge, B., Seth, A. K., & Buckley, C. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.05838 · Cited by 9
“Previous work has considered using a learned model to generate additional data for training a model-free policy (Gu et al., 2016; Sutton, 1990; 1991).”
MethodRobust Hierarchical Planning with Policy DelegationPreprint
Lai, T., & Morere, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13033 · Cited by 3
“When transition dynamics are learned from observed transitions, model-based RL methods [8] can generate optimal plans maximising rewards.”
MethodAgarwal, A., Alomar, A., Alumootil, V., Shah, D., Shen, D., Xu, Z., & Yang, C. (2021) · Neural Information Processing Systems · doi:10.48550/arxiv.2102.06961 · Cited by 22
“Note that using model-based methods to augment the data used by a model-free method has been explored in the literature [46, 38, 19].”
MethodData augmentation for regularizing learned world models in reinforcement learning
van Bree, R. J. (2021) · Cited by 3
Applications of the Free Energy Principle to Machine Learning and NeurosciencePreprint
Millidge, B. (2021) · arXiv (Cornell University) · doi:10.7488/era/1501 · Cited by 13
Baconian : a unified model-based reinforcement learning library
Dong, L. (2021) · doi:10.32657/10356/146557 · Cited by 3
Deep Reinforcement Learning as Guidance for Aerospace Robotics
Hovell, K. (2022) · doi:10.22215/etd/2022-14922 · Cited by 3
The construction and use of cognitive maps in model-based controlPreprint
Karagoz, A. B., Reagh, Z. M., & Kool, W. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ngqwa · Cited by 3
Internet Appendix for "AI-Powered Trading, Algorithmic Collusion, and Price Efficiency"Preprint
Dou, W. W., Itay Goldstein, B., & Ji, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4815650 · Cited by 3
Online model adaptation in Monte Carlo tree search planning
Zuccotto, M., Fusa, E., Castellini, A., & Farinelli, A. (2024) · Optimization and Engineering · doi:10.1007/s11081-024-09896-2 · Cited by 2
Yang, X. H., Wu, L., Wang, L., Ying, C., Su, H., & Zhu, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3585726 · Cited by 2
“In real-world scenarios [44], [45], accurately modeling environment dynamics in a differentiable manner is often infeasible due to the complexity and unpredictability of physical systems.”
A Proposal for a Leaner Narrative of Learning Classifier Systems
Lanzi, P. L., & Loiacono, D. (2025) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3712255.3735661 · Cited by 2
“Note that we did not modify XCS but only how we computed the expected value used to update XCS. Dyna-Q [39, 40] is an architecture that integrates Q-learning with a model of the environment that can be used for planning or self-training.”
FlowDreamer: A RGB-D World Model With Flow-Based Motion Representations for Robot Manipulation
Guo, J., Ma, X., Wang, Y., Yang, M., Liu, H., & Li, Q. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3653273 · Cited by 2
“These models can serve as simulators for the environment and facilitate approaches such as model-based reinforcement learning [7], [8], [9], [27], [28], [29], [30], [31] and model predictive control [10], [32].”
Integrating reinforcement learning with visual generative models: foundations and advances
Liang, Y., Fang, Y., Li, R., Ni, Z., Su, R., & Zhang, C. (2026) · Vicinagearth. · doi:10.1007/s44336-025-00030-z · Cited by 6
Tong, J., Wang, Y., Huang, T., Zhao, X., & Liu, G. (2026) · Computer Networks · doi:10.1016/j.comnet.2026.112056 · Cited by 2
World Models for Robotic Manipulation: A Survey
Wang, F., Wang, Z., Pei, G., Zhang, M., 梁燦錫, Hu, J., Li, Z., Wu, J., Han, N., Zhang, Z., et al. (2026) · SmartBot · doi:10.1002/smb2.70053 · Cited by 11
Curiosity-based topological reinforcement learning
Hafez, M. B., & Kiong, L. C. (2014) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2014.6974211 · Cited by 2
“9, with the number of the selected configuration drawn randomly from a uniform distribution over the range [1,4].”
Background
Metadata
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- ACM SIGART Bulletin
- Published
- 1991-07
- Volume
- 2
- Issue
- 4
- Pages
- 160-163
- ISSN
- 0163-5719 (ISSN-L)
- OpenAlex
- W1980035368
- Cited by
- 777
- References
- 18
- Citations by year
- 2026: 56 · 2025: 71 · 2024: 57 · 2023: 63 · 2022: 57 · 2021: 79 · 2020: 115 · 2019: 62 · 2018: 42 · 2017: 25 · 2016: 9 · 2015: 16 · 2014: 5 · 2013: 13 · 2012: 11
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGART Bulletin (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- learning from examples, planning, model-based learning, incremental planning, learning-based planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
