The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Dyna, an integrated architecture for learning, planning, and reacting
Abstract
Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1991
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citationCraik, K. J. W. (1943) · Cited by 1,845
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 916
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 583
Learning and Sequential Decision Making
Barto, A. G., Sutton, R. S., & Watkins, C. (1989) · Cited by 349
Why the Law of Effect will not Go Away*
Dennett, D. C. (1975) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1975.tb00350.x · Cited by 330
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Research works printed but not linked: 10
- Dennett D. C., 1978, Brainstorms, by D. C. Dennett, 71--89
- Holland J. H., 1986, R. Michalski, J. Carbonell &
- Lin Ji., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, 297--305
- Riolo R., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, MIT Press.
- Russell S. J., 1989, Proceedings of IJCAI-89
- 10.1023/A:1022633531479 · DOI · Paper page
- 10.5555/101883.102055 · DOI
- Sutton R.S., 1981, An adaptive network that constructs and uses an internal model of its environment, Cognition and Brain Theory Quarterly, 4, 217
- Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department. Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department.
- 10.1023/A:1022619109594 · DOI · Paper page
Cited by
Sorted by citations
Items 201–300 of 764
Reinforcement Learning for Process Control: Review and Benchmark Problems
Park, J., Jung, H., Kim, J. W., & Lee, J. M. (2025) · International Journal of Control Automation and Systems · doi:10.1007/s12555-024-0990-1 · Cited by 22
“…methods MPC RS [146], Mb-Mf [147], PETS [148], PlaNet [149] MCTS ExIt [150], FBTS [151] Model-based data generation Dyna-style Dyna [152,153], MB-TRPO [154], MB-MPO [155], SLBO [156], MBPO [157] Value function MVE [158], STEVE [159] enhance the effectiveness and efficiency of RL.…”
A review of learning-based dynamics models for robotic manipulation
Ai, B., Tian, S., Shi, H., Wang, Y., Pfaff, T., Tan, C., Christensen, H. I., Su, H., Wu, J., & Li, Y. (2025) · Science Robotics · doi:10.1126/scirobotics.adt1497 · Cited by 41
“Learned dynamics models facilitate this process by simulating transitions ( 108 ), allowing policies to be trained with a reduced or negligible number of real-environment interactions.”
Hashmi, U. S., Akbar, S., Adve, R., Moo, P. W., & Ding, J. B. (2022) · IET Radar Sonar & Navigation · doi:10.1049/rsn2.12337 · Cited by 26
A multimodal adaptive session manager for physical rehabilitation exercising
Tsiakas, K., Huber, M., & Makedon, F. S. (2015) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/2769493.2769507 · Cited by 22
“This architecture is called Dyna architecture and Dyna-Q is an algorithm that follows this architecture [24].”
BackgroundNonparametric Model-Based Reinforcement Learning
Atkeson, C. G. (1997) · neural information processing systems · Cited by 22
Model-Based Planning with Discrete and Continuous ActionsPreprint
Henaff, M., Whitney, W. F., & LeCun, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07177 · Cited by 60
Intelligent games meeting with multi-agent deep reinforcement learning: a comprehensive review
Wang, Y., Wang, Y., Tian, F., Ma, J., & Jin, Q. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11166-1 · Cited by 25
Towards Safe, Explainable, and Regulated Autonomous Driving
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. G. (2023) · CRC Press eBooks · doi:10.1201/9781003324140-2 · Cited by 21
Research on Cooperative Collision Avoidance Problem of Multiple UAV Based on Reinforcement Learning
Fang, B., Feng, X.-F., & Xu, S. (2017) · International Conference on Intelligent Computation Technology and Automation · doi:10.1109/icicta.2017.30 · Cited by 21
MACHINE INTELLIGENCE. ESSAYS ON THE THEORY OF MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE
Шумский, С., & Shumskiy, S. (2019) · Publishing Center RIOR eBooks · doi:10.29039/02011-1 · Cited by 22
Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation
Yao, Y., Li, X., An, Z., Zhang, W., & Luo, D. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561842 · Cited by 20
“For example, Dyna-style methods [9], [11], [15] use the models to generate augmented samples.”
MethodInfluentialBayesian Learning of Recursively Factored Environments
Bellemare, M. G., Veness, J., & Bowling, M. (2013) · Cited by 21
A topological reinforcement learning agent for navigation
Braga, A. P. S., & Ara�jo, A. F. R. (2003) · Neural Computing and Applications · doi:10.1007/s00521-003-0385-9 · Cited by 21
“The three eligibility trace-based algorithms presented very similar trajectory lengths from 65% to 80% longer than Dyna-Q paths.”
BackgroundMethodInfluentialModel-Based Planning in Discrete Action Spaces.
Henaff, M., WHITNEY, W. F., & LeCun, Y. (2017) · Cited by 22
Yang, N., Han, L., Zhou, X., Liu, R., Liu, H., & Xiang, C. (2023) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2023.3295651 · Cited by 20
“To improve the existing drawbacks, this article proposes an online-learning adaptive strategy to handle the energy management problem in various driving scenarios by introducing an RL framework, Dyna [24].”
MethodOffline Reinforcement Learning of Robotic Control Using Deep Kinematics and Dynamics
Li, X., Shang, W., & Cong, S. (2023) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2023.3336316 · Cited by 20
“MBRL algorithms can be stratified into categories: shooting algorithms [7], [8], backpropagation-through-time policy search algorithms [5], [9], and Dyna-style algorithms [10], [11], [12], as used in this article.”
MethodHybrid Reinforcement Learning with Expert State Sequences
Guo, X., Chang, S., Yu, M., Tesauro, G. J., & Campbell, M. S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013739 · Cited by 23
Synthesis of model predictive control and reinforcement learning: Survey and classification
Reiter, R., Hoffmann, J., Reinhardt, D. P., Messerer, F., Baumgärtner, K., Sawant, S., Bödecker, J., Diehl, M., & Gros, S. (2026) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2026.101045 · Cited by 19
Liu, Y., Yan, S., Zhao, Y., Song, C., & Li, F. (2022) · Drones · doi:10.3390/drones6110365 · Cited by 22
Reinforcement Learning Approaches for Traffic Signal Control under Missing Data
Mei, H., Li, J., Shi, B., & Wei, H. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/251 · Cited by 19
“Thus we combine the model-based reinforcement learning (MBRL) with the reward model and train a shared policy in the Dyna-Q style framework [ Sutton, 1991; Zhao et al. , 2020 ] .”
MethodAutomatic MAC protocol selection in wireless networks based on reinforcement learning
Gomes, A., Macedo, D. F., & Vieira, L. F. M. (2019) · Computer Communications · doi:10.1016/j.comcom.2019.10.023 · Cited by 19
Model Learning and Knowledge Sharing for Cooperative Multiagent Systems in Stochastic Environment
Jiang, W., Narayanan, V., & Li, J.-S. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2958912 · Cited by 19
“The Dyna architecture is a well-known model-based RL approach [11], [12] in which an agent reconstructs a model of the environment from actual experience (through direct learning), and after the learning process, when the model is fed with the current state and an action, it will predict the next…”
BackgroundActive Learning of Relational Action Models
Rodrigues, C., Gérard, P., Rouveirol, C., & Soldano, H. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31951-8_26 · Cited by 19
“Indirect RL [6], in which an action model is explicitly learned, is proved to be very efficient with relational representations [1].”
BackgroundUltrasound Image-Based Average $Q$-Learning Control of Magnetic Microrobots
Liu, J., Ma, G., Fu, S., Huang, C., Wu, X., & Xu, T. (2025) · IEEE Transactions on Robotics · doi:10.1109/tro.2025.3543261 · Cited by 18
A Local Approach to Forward Model Learning: Results on the Game of Life Game
Lucas, S., Dockhorn, A., Volz, V., Bamford, C., Gaina, R. D., Bravi, I., Pérez-Liébana, D., Mostaghim, S., & Kruse, R. (2019) · 2019 IEEE Conference on Games (CoG) · doi:10.1109/cig.2019.8848002 · Cited by 19
“Deterministic forward models in some cases can be learned by tabular methods [6] [7].”
BackgroundMapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks
Zhu, M., Liu, M., Shen, J. G., Zhang, Z., Chen, S., Zhang, W., Ye, D., Yu, Y. H., Fu, Q., & Yang, W. P. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/480 · Cited by 19
Role of cerebellum in sleep-dependent memory processes
Jackson, A., & Xu, W. (2023) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2023.1154489 · Cited by 21
“This shares similarities with the Dyna architecture of Sutton (1991) in which reinforcement learning is applied to simulated experience generated by a forward model that predicts the consequences of hypothetical actions.”
MethodDreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical RepresentationsPreprint
Deng, F., Jang, I., & Ahn, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14565 · Cited by 18
“Model-Based Reinforcement Learning (MBRL, Sutton & Barto, 2018; Sutton, 1991) provides a so-lution to many problems in contemporary reinforcement learning.”
BackgroundDeep reinforcement learning for collision avoidance in unmanned surface vehicles: State-of-the-art
Li, Y., Wu, D., You, Z., Chen, G., & Wu, D. (2025) · Applied Ocean Research · doi:10.1016/j.apor.2025.104778 · Cited by 25
Zhao, C., Liu, G., Zhang, R., Liu, Y., Wang, J., Kang, J., Niyato, D. T., Li, Z., Shen, X., Han, Z., et al. (2026) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2026.3658762 · Cited by 31
“In the 1990s, Sutton’s Dyna architecture [60] demonstrated that learning an internal model of environment dynamics enables agents to simulate “imagined” experiences, improving planning and sample e ffi - ciency.”
Wu, G., Fang, W., Wang, J., Ge, P., Cao, J., Yang, P., & Gou, P. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04354-x · Cited by 17
“As an integrated architecture for learning and planning, Dyna [41] was first proposed by Sutton in 1991.”
MethodHuman-Flow-Aware Long-Term Mobile Robot Task Planning Based on Hierarchical Reinforcement Learning
Liu, Y., Palmieri, L., Georgievski, I., & Aiello, M. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3280816 · Cited by 17
“In this letter, we combine H-Dyna into the option framework to learn additionally from the “replayed” experiences [17].”
MethodJun, Z., Jia, R., Liu, S., He, D., Li, K., & Wang, F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124402 · Cited by 18
Topological Q-learning with internally guided exploration for mobile robot navigation
Hafez, M. B., & Loo, C. K. (2015) · Neural Computing and Applications · doi:10.1007/s00521-015-1861-8 · Cited by 17
“the selected configuration drawn randomly from a uniform distribution over the integer range [1, 3].”
BackgroundIntelligent Trainer for Dyna-Style Model-Based Deep Reinforcement Learning
Dong, L., Li, Y., Zhou, X., Wen, Y., & Guan, K. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3008249 · Cited by 20
“To address this challenge with training data, researchers have previously proposed a model-based reinforcement learning (MBRL) framework, Dyna [7].”
MethodTight Regret Bounds for Model-Based Reinforcement Learning with Greedy PoliciesPreprint
Efroni, Y., Merlis, N., Ghavamzadeh, M., & Mannor, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11527 · Cited by 17
“cations, the high computational complexity of model-based RL makes them infeasible. Thus, practical model-based approaches alleviate this computational burden by using short-term planning e.g., Dyna [Sutton, 1991], instead of full-planning. To the best of our knowledge, there are no regret guarantees for such algorith…”
InfluentialUsing computational models of learning to advance cognitive behavioral therapy
Berwian, I. M., Hitchock, P., Pisupati, S., Schoen, G., & Niv, Y. (2025) · Communications Psychology · doi:10.1038/s44271-025-00251-4 · Cited by 16
Cognitive Models Are Distinguished by Content, Not Format
Butlin, P. (2020) · Philosophy of Science · doi:10.1086/710030 · Cited by 16
“In this section I argue against the format conception of cognitive models by reference to Dyna, a simple model-based RL algorithm for ordinary computers (Sutton 1991).”
MethodModel-based learning retrospectively updates model-free values
Doody, M., van Swieten, M. M. H., & Manohar, S. (2022) · Scientific Reports · doi:10.1038/s41598-022-05567-3 · Cited by 22
Deep reinforcement learning with planning guardrails for building energy demand response
Jang, D., Spangher, L., Nadarajah, S., & Spanos, C. J. (2022) · Energy and AI · doi:10.1016/j.egyai.2022.100204 · Cited by 18
Mice identify subgoal locations through an action-driven mapping process
Shamash, P., Lee, S., Saxe, A., & Branco, T. (2023) · Neuron · doi:10.1016/j.neuron.2023.03.034 · Cited by 14
“One possibility is that mice construct a value function or successor representation through a more data-efficient, model-based learningalgorithm than the purelymodel-free updatingmechanismsweusedhere.(29,47) Another possibility is thatmice simply imbue certain actions (e.”
Deserno, L., Moran, R., Lee, Y., Michely, J., Dayan, P., & Dolan, R. J. (2021) · eLife · doi:10.7554/elife.67778 · Cited by 18
Parallel Online Temporal Difference Learning for Motor Control
Caarls, W., & Schuitema, E. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2442233 · Cited by 16
An Experience Replay Method Based on Tree Structure for Reinforcement Learning
Jiang, W., Hwang, K., & Lin, J.-L. (2019) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2018.2890682 · Cited by 16
“Dyna architecture, which combines policy learning and model learning, is an extension of reinforcement learning (RL) [13], [15].”
BackgroundMendonca, R., Geng, X., Finn, C., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07178 · Cited by 36
“This resembles methods that combine model learning with model-free RL in single-tasks settings [32, 12].”
MethodMHER: Model-based Hindsight Experience ReplayPreprint
Yang, R., Fang, M., Han, L., Du, Y., Luo, F., & Li, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00306 · Cited by 16
“Dyna [32] generates virtual samples with a trained dynamics model to augment training data.”
MethodA New Glycemic closed-loop control based on Dyna-Q for Type-1-Diabetes
Giorno, S. D., D’Antoni, F., Piemonte, V., & Merone, M. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.104492 · Cited by 15
Jang, J., Han, J., & Kim, J. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120136 · Cited by 17
Sample-efficient model-based reinforcement learning for quantum control
Khalid, I., Weidner, C. A., JONCKHEERE, E. A., Schirmer, S. G., & Langbein, F. C. (2023) · Physical Review Research · doi:10.1103/physrevresearch.5.043002 · Cited by 15
Reinforcement Learning in Image Classification: A Review
Alrebdi, N., Alrumiah, S. S., Almansour, A. F., & Rassam, M. A. (2022) · International Conference on Computing and Information Technology (ICCIT) · doi:10.1109/iccit52419.2022.9711620 · Cited by 15
“In contrast, model-based approaches learn an explicit model of the system simultaneously with a value function and policy [11,19-22].”
Associative versus predictive processes in Pavlovian conditioning
Jozefowiez, J. (2017) · Behavioural Processes · doi:10.1016/j.beproc.2017.12.016 · Cited by 17
Constructing future behaviour in the hippocampal formation through composition and replayPreprint
Bakermans, J. J. W., Warren, J., Whittington, J. C. R., & Behrens, T. E. J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.04.07.536053 · Cited by 14
“For example, if replay performs Bellman back-ups (as in the dyna algorithm (Sutton, 1991)) it can only update the value of any state by comparing to its neighbour, so is exquisitely sensitive to the trajectory of the replay.”
BackgroundReinforcement learning in non-stationary environments
Lecarpentier, E. (2020) · theses.fr (ABES) · doi:10.70675/0c2168aaz74edz47a7z87b1z55403db85295 · Cited by 16
WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigation
Nie, D., Guo, X., Duan, Y., Zhang, R., & Chen, L. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11246684 · Cited by 52
“World models originated in classical reinforcement learning with Dyna’s simulated experiences [32], evolving into frameworks for predicting state transitions to improve sample efficiency [33], [34].”
Gómez, M. M., Gonzalez, R., Castro‐González, Á., Malfáz, M., & Salichs, M. A. (2021) · IEEE Access · doi:10.1109/access.2021.3095392 · Cited by 13
“Dyna [17] can be defined as a probabilistic model, which improves the convergence speed and stability of classical algorithms, such as one-step Q-learning [18] or multi-step Q-( λ ) [19].”
BackgroundMethodFear and hope emerge from anticipation in model-based reinforcement learning
Moerland, T. M., Broekens, J., & Jonker, C. M. (2016) · International Joint Conference on Artificial Intelligence · Cited by 14
A Sample Aggregation Approach to Experiences Replay of Dyna-Q Learning
Shi, H., Yang, S., Hwang, K.-S., Chen, J., Hu, M., & Zhang, H. (2018) · IEEE Access · doi:10.1109/access.2018.2847048 · Cited by 14
“The virtual experience can be used to update the policy, which is called indirect reinforcement learning or planning [23].”
MethodModel-Based Reinforcement Learning For Robot Control
Li, X., Shang, W., & Cong, S. (2020) · International Conference on Advanced Robotics and Mechatronics (ICARM) · doi:10.1109/icarm49381.2020.9195341 · Cited by 15
“First is Dyna-style Algorithms [19] [17] [16].”
Hierarchical intrinsically motivated agent planning behavior with dreaming in grid environments
Dzhivelikian, E., Latyshev, A., Kuderov, P., & Panov, A. I. (2022) · Brain Informatics · doi:10.1186/s40708-022-00156-6 · Cited by 14
“The learned model can be utilized to support the learning of an agent’s global policy by supplying it with additional trajectories generated in imagination (Sutton, 1991).”
MethodThe pursuit of happiness: A reinforcement learning perspective on habituation and comparisons
Dubey, R., Griffiths, T. L., & Dayan, P. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010316 · Cited by 18
“A promising direction in this vein could be studying how aspiration levels might be shaped via a functional relationship between a model-free and model-based system [82, 83].”
BackgroundPlanning and Execution using Inaccurate Models with Provable Guarantees
Vemula, A., Oza, Y., Bagnell, J. A., & Likhachev, M. A. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.001 · Cited by 14
Computational Psychiatry for Computers
Schulz, E., & Dayan, P. (2020) · iScience · doi:10.1016/j.isci.2020.101772 · Cited by 14
“, 2017); and there is speculation that interactions between the systems, with samples drawn from the model being used to train model-free mechanisms (Sutton 1991; Mattar and Daw 2018) could themselves be associated with psychiatric conditions involving issues such as rumination (Gagne et al.”
BackgroundButz, M. V. (2007) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/1274000.1274104 · Cited by 14
On Optimism in Model-Based Reinforcement Learning.Preprint
Pacchiano, A., Ball, P., Parker-Holder, J., Choromański, K., & Roberts, S. (2020) · arXiv (Cornell University) · Cited by 14
Model-Invariant State Abstractions for Model-Based Reinforcement LearningPreprint
Tomar, M., Zhang, A., Calandra, R., Taylor, M. E., & Pineau, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.09850 · Cited by 14
“by a model-free solver (Sutton, 1991), or 3) be used as an auxiliary task while training (Lee et al.”
Method(2023) · doi:10.3390/s23218924 · Cited by 13
“The main purpose of RL is to solve problems, utilizing intelligent agents that perform actions and aim to maximize the total rewards within a given environment [21].”
BackgroundEntry guidance for spatial no-fly zones avoidance via model-based reinforcement learning
Li, X., Wang, X., & Zhou, H. (2024) · Aerospace Science and Technology · doi:10.1016/j.ast.2024.109405 · Cited by 13
Awake replay: off the clock but on the job
van der Meer, M. A. A., & Bendor, D. (2025) · Trends in Neurosciences · doi:10.1016/j.tins.2025.02.006 · Cited by 19
“This idea blurs the distinction between online and of fl ine roles of replay [64,69,71].”
A Computational Neuroscience Perspective on the Change Process in Psychotherapy
Smith, R. P., Lane, R. D., Nadel, L., & Moutoussis, M. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780190881511.003.0015 · Cited by 13
Pagalyte, E., Mancini, M., & Climent, L. (2020) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3383652.3423868 · Cited by 13
“Wender & Watson [16] and Amato & Shani [1] target the game Civilization IV to study RL approaches [13] in strategy games.”
BackgroundMarkov Decision Processes: Concepts and Algorithms
van Otterlo, M., Wiering, M., Wiering, M., & van Otterlo, M. (2012) · Cited by 13
Attractors in recurrent behavior networks
Goetz, P. (1998) · Cited by 13
Goal-Directed Acting with Incomplete Information
Koenig, S. (1997) · Cited by 13
DREAM Architecture: a Developmental Approach to Open-Ended Learning in RoboticsPreprint
Doncieux, S., Bredèche, N., Goff, L. K. L., Girard, B., Coninx, A., Sigaud, O., Khamassi, M., Díaz-Rodríguez, N., Filliat, D., Hospedales, T. M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06223 · Cited by 18
“One recent hypothesis is that such a replay might be useful to bootstrap reinforcement learning by using an internal model of the task structure to update state-action value functions offline [174, 59], similarly to the Dyna architecture [175].”
Reward Propagation Using Graph Convolutional NetworksPreprint
Klissarov, M., & Precup, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02474 · Cited by 14
“Future work could consider a more sophisticated approach by taking inspiration form grid cell-like constructs [O’Keefe and Dostrovsky, 1971] or by combining our sampling strategy with model roll-outs [Sutton, 1991] to construct the MDP’s underlying graph.”
Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a SurveyPreprint
Plaat, A., Kosters, W. A., & Preuß, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05598 · Cited by 36
“This approach was introduced by Sutton [Sutton, 1990, 1991] in the Dyna system, long before deep learning was used widely.”
BackgroundMethodLearning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 346
“Learning an accurate dynamics model in reaction to control inputs has been a long-standing challenge in system identification [50], model-based reinforcement learning [51], and optimal control [52, 53].”
BackgroundA probabilistic successor representation for context-dependent learning
Geerts, J. P., Gershman, S. J., Burgess, N., & Stachenfeld, K. L. (2024) · Psychological Review · doi:10.1037/rev0000414 · Cited by 14
“…1992), and simulated experience, in which possible experienced sequences are generated from a transition model (Momennejad, Otto, et al., 2017; Sutton, 1991); rather, multiple states are potentially activated and updated simultaneously (like the form of reactivation described in Manning,…”
BackgroundSelf-managed patient-game interaction using the barrett WAM arm for motion analysis
Lioulemes, A., Sassaman, P., Gieser, S. N., Karkaletsis, V., Makedon, F. S., & Metsis, V. (2015) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/2769493.2769517 · Cited by 12
“We will use Reinforcement Learning for the problem formulation and Dyna-Q algorithm [21] for the system training.”
MethodWu, Z., Yu, C., Chen, C., Hao, J., & Zhuo, H. H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26241 · Cited by 12
Bayesian Reinforcement Learning With Limited Cognitive Load
Arumugam, D., Ho, M. K., Goodman, N. D., & Roy, B. V. (2024) · Open Mind · doi:10.1162/opmi_a_00132 · Cited by 17
Muthyala, P., Sharma, M., Ünver, B., Evren, S., Uzunoğlu, B., Kaydeci, A. B., Toğuş, F., Schaub, J., Virnich, L., & Pischinger, S. (2024) · Energy Conversion and Management X · doi:10.1016/j.ecmx.2024.100679 · Cited by 12
Velasco, P. F., Griesbauer, E., Brunec, I. K., Morley, J. G., Manley, E., McNamee, D., & Spiers, H. J. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2407814122 · Cited by 14
Fast Learning-based Control for Energy Management of Hybrid Electric Vehicles
Liu, T., Du, G., Zou, Y., & Cao, D. (2018) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2018.10.143 · Cited by 12
Anticipation by multi-modal association through an artificial mental imagery process
Gaona, W., Escobar, E., Hermosillo, J., & Lara, B. (2014) · Connection Science · doi:10.1080/09540091.2014.956289 · Cited by 12
“The idea of off-line training via simulated ‘experience’ is not new and can be traced back to DYNA (Sutton, 1991), under the classical framework of reinforcement learning.”
BackgroundProgramming by demonstration of robot manipulators
Skoglund, A. (2009) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 12
Wittkuhn, L., Krippner, L. M., Koch, C., & Schuck, N. W. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.02.02.478787 · Cited by 12
Robotics: Science and Systems XVI
(2020) · doi:10.15607/rss.2020.xvi · Cited by 12
Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
Muller-Brockhausen, M., Preuss, M., & Plaat, A. (2021) · 2021 IEEE Conference on Games (CoG) · doi:10.1109/cog52621.2021.9619000 · Cited by 11
“One trend that could help overcome this problem, at least for navigation-related tasks, is the fusion of learning and planning, as in model-based reinforcement learning [70].”
BackgroundAl-Dayaa, H., & Megherbi, D. B. (2010) · The Journal of Supercomputing · doi:10.1007/s11227-010-0451-x · Cited by 11
“12a Q-Learning vs. Distance-and-Frequency (actual data) Fig.”
MethodInfluentialA global framework for a systemic view of brain modeling
Alexandre, F. (2021) · Brain Informatics · doi:10.1186/s40708-021-00126-4 · Cited by 17
Influence zones: A strategy to enhance reinforcement learning
Braga, A. P. S., & Araujo, A. F. R. (2006) · Neurocomputing · doi:10.1016/j.neucom.2006.07.010 · Cited by 11
How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy OptimizationPreprint
D’Oro, P., & Jaśkowski, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14309 · Cited by 11
“Our approach is inspired by Dyna [46], and employs an approximate dynamics model for generating 1-step imaginary on-policy transitions starting from observed states stored in a replay buffer.”
MethodModel-Based Deep Reinforcement Learning for High-Dimensional Problems, a Survey.Preprint
Plaat, A., Kosters, W. A., & Preuß, M. (2020) · arXiv (Cornell University) · Cited by 12
Experience replay supports non-local learningPreprint
Liu, Y., Mattar, M. G., Behrens, T. E. J., Daw, N. D., & Dolan, R. J. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.10.20.343061 · Cited by 11
“These covert trajectories can substitute for direct experience and thereby span the gaps between actions and outcomes (11), a process referred to as experience replay.”
BackgroundMahmud, S., Sayed, A. N., Himeur, Y., Nhlabatsi, A., & Bensaali, F. (2025) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2025.116481 · Cited by 10
Causally aware reinforcement learning agents for autonomous cyber defence
Purves, T., Kyriakopoulos, K. G., Jenkins, S., Phillips, I. W., & Dudman, T. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112521 · Cited by 10
Residential Energy Scheduling With Solar Energy Based on Dyna Adaptive Dynamic Programming
Xiong, K., Wei, Q., & Li, H. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124809 · Cited by 10
Antonov, G. K., & Dayan, P. (2025) · Nature Communications · doi:10.1038/s41467-025-56731-y · Cited by 10
Megherbi, D. B., & Malayia, V. (2010) · The Journal of Supercomputing · doi:10.1007/s11227-010-0510-3 · Cited by 10
“The most popular example of the use of reenforcement learning has been the DYNA-Q algorithm [28, 29].”
Method
Metadata
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- ACM SIGART Bulletin
- Published
- 1991-07
- Volume
- 2
- Issue
- 4
- Pages
- 160-163
- ISSN
- 0163-5719 (ISSN-L)
- OpenAlex
- W1980035368
- Cited by
- 777
- References
- 18
- Citations by year
- 2026: 56 · 2025: 71 · 2024: 57 · 2023: 63 · 2022: 57 · 2021: 79 · 2020: 115 · 2019: 62 · 2018: 42 · 2017: 25 · 2016: 9 · 2015: 16 · 2014: 5 · 2013: 13 · 2012: 11
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGART Bulletin (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- learning from examples, planning, model-based learning, incremental planning, learning-based planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
