The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Dyna, an integrated architecture for learning, planning, and reacting
Abstract
Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1991
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citationCraik, K. J. W. (1943) · Cited by 1,845
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 916
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 583
Learning and Sequential Decision Making
Barto, A. G., Sutton, R. S., & Watkins, C. (1989) · Cited by 349
Why the Law of Effect will not Go Away*
Dennett, D. C. (1975) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1975.tb00350.x · Cited by 330
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Research works printed but not linked: 10
- Dennett D. C., 1978, Brainstorms, by D. C. Dennett, 71--89
- Holland J. H., 1986, R. Michalski, J. Carbonell &
- Lin Ji., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, 297--305
- Riolo R., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, MIT Press.
- Russell S. J., 1989, Proceedings of IJCAI-89
- 10.1023/A:1022633531479 · DOI · Paper page
- 10.5555/101883.102055 · DOI
- Sutton R.S., 1981, An adaptive network that constructs and uses an internal model of its environment, Cognition and Brain Theory Quarterly, 4, 217
- Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department. Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department.
- 10.1023/A:1022619109594 · DOI · Paper page
Cited by
Sorted by citations
Items 701–764 of 764
Wang, S., Liu, M., Zheng, R., Dong, S., & Wei, P. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11245885
“Early foundations trace back to the Dyna [11] framework, which pioneers the integration of model learning and policy optimization for sample efficiency.”
Haberland, S., Ruge, H., & Frimmel, H. (2025) · PLoS ONE · doi:10.1371/journal.pone.0338034 · Cited by 2
“79 seconds for data preprocessing of the human and DQN time series.”
Sample-Efficient Reinforcement Learning for Walking Robots
Vennemann, B. (2013) · Research Repository (Delft University of Technology)
Reinforcement Learning: Shaping Behaviour Through Rewards
Akerkar, R. A. (2026) · Springer eBooks · doi:10.1007/978-3-031-91084-5_6
AndroidWMSearch: Mobile Agents Tree Search with World Model
Jia, X., Zhang, L., & Xu, M. (2025) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads67057.2025.11323094
“World models have long been fundamental to model-based reinforcement learning (RL) [10], [11], as they learn state transitions to improve sample efficiency and support planning [12].”
Sharma, M., Singh, R., & Singh, M. (2025) · International Conference on Automation, Computing and Renewable Systems (ICACRS) · doi:10.1109/icacrs67045.2025.11324403
“Williams [22] introduced policy gradient methods that directly optimize the expected reward, while Schulman et al. [23] developed Proximal Policy Optimization (PPO) a robust and efficient algorithm for continuous control.”
Gao, S., Yang, X., Huang, J., Tu, R., Zhang, T., & Li, Q. (2026) · IET Control Theory and Applications · doi:10.1049/cth2.70104
Diversity-Driven Model Ensemble Adaptive Trust Region Policy Optimization
Xu, H., Yan, Z., Xuan, J., Zhang, G., & Jie Lu, B. (2026) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2026.3650850
“During 1990 and 1991, the RL pioneer Sutton proposed the first model-based RL algorithm Dyna [27], [28], [29], in which the training procedure iterates between two key steps.”
Li, T., Liu, Q., & Peng, P. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00400
“In the field of Artifical Intelligence (AI), model-based RL enhances sample efficiency and planning capabilities by explicitly reconstructing the environment dynamics [28]– [31], commonly referred to as world models.”
Introduction of Artificial Intelligence in Healthcare
Tayebi, L., Shetty, S. P., & Imani, S. (2026) · Springer eBooks · doi:10.1007/978-3-032-15977-9_1 · Cited by 1
Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models
Pappa, M., Romani, L., Sacco, V., Palma, A., Lathuilière, S., Galasso, F., Alameda-Pineda, X., & Spinelli, I. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37350-2_26
Reinforcement-Trained Neural Network for Various Manufacturers AV Mixed Traffic Flow Simulation
Бобровская, О. П., Владимирович, Г. Т., & Galkin, V. A. (2026) · Optical Memory and Neural Networks · doi:10.3103/s1060992x25603173
Micro-Data Reinforcement Learning for Adaptive Robots
Chatzilygeroudis, K. I. (2018) · theses.fr (ABES) · doi:10.70675/7990d9edz7f40z4344z9318z6da4e46000c0
“This idea leads to model-based policy search algorithms (Deisenroth et al., 2013; Polydoros and Nalpantidis, 2017), in which the trajectory data are used to learn the dynamics model, then policy search is performed on the model (Sutton, 1991; Kaelbling et al., 1996).”
MethodGénération automatique de modèle pour l'optimisation énergétique des systèmes cyber-physiques.
Rioual, Y. (2019) · theses.fr (ABES) · doi:10.70675/c09ada33z0ac7z4f01z93dfz0991eb4681ea
Multimodal and interactive models for visually grounded language learning
Strub, F. (2020) · theses.fr (ABES) · doi:10.70675/565f6b44z5446z433azb85dz57d7ca36bccf
Mamba World Model for RL: Enhancing Long-term Memory in Model-based Reinforcement Learning
Cao, M., Huang, Y., Gümüşsoy, S., & Ogras, U. Y. (2025) · International Conference on Machine Learning and Applications · doi:10.1109/icmla66185.2025.00094
“The concept of integrating model-based components into deep reinforcement learning can be traced back to the Dyna architecture [21], where model-free learning and model-based planning are combined.”
Improving Cooperation in Multi-agent Deep Reinforcement Learning for Mobile Robotics
Toquebiau, M. (2024) · theses.fr (ABES) · doi:10.70675/3385d0bbzc66cz4fceza9e0z8696ff61ff4e
Automatic Symbolic Goal Abstraction via Reachability Analysis in Hierarchical Reinforcement Learning
Zadem, M. (2024) · theses.fr (ABES) · doi:10.70675/f273f52az0906z443cza443z172023a62cf5
Wang, Y., Shi, D., & Yang, H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464085
Liu, Y., Qiao, P., Li, R., He, Z., & Sun, T. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463569
Zhang, X., Li, G., Shokouhi, S., & Thein, M.-W. L. (2026) · Actuators · doi:10.3390/act15050235
MultiModal Action Conditioned Video Simulation
Yichen, L., & Torralba, A. B. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01315 · Cited by 1
Default Feature Representations of the Cognitive MapPreprint
Bazarjani, A., & Piray, P. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.11.724436 · Cited by 2
“Next, we evaluate our model against Dyna-Q [38, 39], an integrated planning algorithm that builds an explicit model of the environment from sampled transitions and interleaves real experience with simulated backups (Figure 1c, f).”
Reinforcement Learning based Personalized Training Method for Computer Programming Courses
Han, L., Lin, L., Wang, Y., & Hu, R. (2025) · International Conference on Information Education and Artificial Intelligence (ICIEAI) · doi:10.1145/3799457.3799592
“Its primary objective is to overcome the fundamental limitation of traditional Q-learning—namely, its excessive reliance on real environmental interactions and low sample efficiency [1].”
Portfolio Optimization Methods for the Digital Asset Market: A Comprehensive Survey
Demosthenous, G., & Georgiou, C. (2026) · ACM Computing Surveys · doi:10.1145/3819577
“This model is used to predict future states and rewards which can be incorporated into the learning process (e.g., Dyna-Q [79]).”
Ji, B., Yu, R., Zhang, L., & Zhu, M. (2026) · Green Energy and Intelligent Transportation · doi:10.1016/j.geits.2026.100434
Dopamine ramps as a normative consequence of dual-process controlPreprint
Priestley, L., & Akam, T. (2026) · eLife · doi:10.7554/elife.111458
Dopamine ramps as a normative consequence of dual-process controlPreprint
Priestley, L., & Akam, T. (2026) · eLife · doi:10.7554/elife.111458.1
Park, H., & Lee, S.-H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37167-6_3
Deep Reinforcement Learning for Adaptive Decision-Making in Real-World Applications
Touati, C., & Kemmar, A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7082-8.ch005
Perspective Chapter: Monte Carlo for Artificial Intelligence, Foundations and Emerging Trends
Chafi, F. Z. (2026) · IntechOpen eBooks · doi:10.5772/intechopen.1016392
LeCropFollow: Latent Space Planning for Navigation in Unstructured Crop Fields
Tommaselli, F. A. G., Pinto, F. A. C., Rocha, A., Capezzuto, G., Sivakumar, A. N. V., Chowdhary, G. V., & Becker, M. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3710052
“Reinforcement learning (RL) provides a principled alternative by mapping high-dimensional observations directly to actions, allowing the policy to retain uncertainty and semantic context [8].”
Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models
Li, H., Fu, W., Cong, X., Li, Z., Huang, B., Jiang, H., He, X., Liang, Y., Fu, R., Lu, T., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37447-9_9 · Cited by 1
RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futures
Gani, H., Kulkarni, T., Chodavarapu, M., Hansen, N., & Chandraker, M. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37447-9_26
Zhang, Z., Tian, W., Liao, Z., & Yuan, Z. (2026) · Water Research · doi:10.1016/j.watres.2026.126458
Decision‐Making Under Uncertainty Using Artificial Intelligence
Caers, J., Arief, M., Scheidt, C., & Li, L. (2026) · Geophysical monograph · doi:10.1002/9781394306824.ch2
Li, T., Peng, P., Liu, Q., Zhang, S., & Song, L. (2026) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2026.3715905
“1, compared with conventional RL methods (MFRL), MBRL improves sample efficiency by learning an environment dynamics model [25], commonly referred to as the world model [26].”
Wang, W., Li, M., Wang, B., Hu, J., & Zhang, T. (2026) · Journal of Marine Science and Engineering · doi:10.3390/jmse14161477
Uncertainty-Weighted Optimisation for Model-Based Reinforcement Learning
Qiao, T., Williams, H. A., & MacDonald, B. (2026) · IEEE International Conference on Mechatronics and Automation (ICMA) · doi:10.1109/icma69663.2026.11647556
“We focus on enhancing a classic and prevalent Dyna-Q framework [26] with Soft Actor-Critic (SAC) [9] as its DP (model-free) component, hereafter referred to as Dyna-Q-SAC, as a demonstration of IW.”
Wang, J., Li, L., & Yu, Y. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.116026
Structural requirements for intelligent clinical digital twins in feedback-driven care
Wang, Y., Smith, G. E., Lipori, G. P., Sun, R. C., Shenkman, E. A., Liu, M., Wu, Y., & Guo, Y. (2026) · npj Health Systems · doi:10.1038/s44401-026-00143-7
A survey of world models for physical AI with uncertainty representation and control
Kirchner, S., Purschke, N., & Knoll, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02122-1
“The classical Dyna architecture formalizes this by mixing real transitions with model-generated transitions for planning and policy/value learning [77].”
DreamerNBV: Model-Based Active View Planning for 3D Reconstruction with 3D Gaussian Splatting
Luo, X., Li, Y., Zheng, W., Wang, C., & Zhao, L. (2026) · International Conference on Intelligent Systems and Robotics (CISR) · doi:10.1109/cisr70722.2026.11680152
“Model-based reinforcement learning (MBRL) offers a principled alternative by learning an internal world model that summarizes history into a compact latent state and supports imagination-based planning [12]–[16].”
Chen, X., & Cheng, K. (2026) · International Conference on Automation and Computing (ICAC) · doi:10.1109/icac70325.2026.11712323
Zhou, L., & Ceyhan, E. (2026) · Computational Statistics · doi:10.1007/s00180-026-01808-5
“This hybrid architecture, similar to Dyna-based methods (Sutton 1991; Silver et al. 2008), systematically handles uncertainty while balancing exploration and exploitation.”
Bridging the Sim2Real Gap via Physics-Constrained Gating and Disturbance Dictionary Learning
Deyong, H., Wang, J., & Li, J. (2026) · International Conference on Emerging Techniques in Computational Intelligence (ICETCI) · doi:10.1109/icetci71772.2026.11710440
Natural and engineered safe reinforcement learning
Mahajan, P., & Seymour, B. (2026) · Communications Psychology · doi:10.1038/s44271-026-00542-4
“Hybrid approaches , such as Dyna 15 , integrate both strategies: a learned model generates simulated experiences to train model-free value learning.”
Yang, B., Zheng, L., Lyu, D., Liu, J., & Chen, B. (2026) · Remote Sensing · doi:10.3390/rs18203517
Shi, D., Zhuang, Z., & Wang, D. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728762
“Many works have been proposed: Dyna [19] and Dreamer [20] leveraged dynamics and reward model to generate new transition samples for data augmentation.”
Shi, C., Luan, X., Jiang, S., & He, B. (2026) · SmartBot · doi:10.1002/smb2.70074
Incremental Learning of Trust while Reacting and Planning
Froelich, W., Kisiel‐Dorohinicki, M., & Nawarecki, E. (2007) · Springer eBooks · doi:10.1007/978-1-4020-6264-3_85
“One of the most interesting is the Dyna system [ 3 ][4] and it’s multi-agent extension M-Dyna-Q [5].”
BackgroundTwin: Playing an Unknown Game with a Test-Time Digital Twin
Skoutnev, A., Acharya, K., Longhitano, G., Udell, M., Ellis, K., & Drori, I. (2026) · arXiv · doi:10.48550/arxiv.2608.14490
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824
“Dynamic programming methods, such as Q-learning algorithms, in principle can offer a more attractive option for offline reinforcement learning as compared to pure policy gradients.”
MethodInfluentialDream to Control: Learning Behaviors by Latent Imagination
Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019) · arXiv · doi:10.48550/arxiv.1912.01603 · Cited by 558
“Imagination environment The latent dynamics define a Markov decision process (MDP; Sutton, 1991) that is fully observed because the compact model states s t are Markovian.”
BackgroundModel-Based Reinforcement Learning for Atari
Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1903.00374 · Cited by 467
Reinforcement Learning with Augmented Data
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., & Srinivas, A. (2020) · arXiv · doi:10.48550/arxiv.2004.14990 · Cited by 299
“While usually not viewed as a data augmentation technique, the idea of generating fake or synthetic rollouts to improve the data-efficiency of RL agents has been proposed in the Dyna framework [32].”
BackgroundMastering Atari with Discrete World Models
Hafner, D., Lillicrap, T., Norouzi, M., & Ba, J. (2020) · arXiv · doi:10.48550/arxiv.2010.02193 · Cited by 247
“Compared to model-free reinforcement learning that learn through trial and error, world models facilitate generalization and can predict the outcomes of potential actions to enable planning (Sutton, 1991).”
BackgroundBenchmarking Model-Based Reinforcement Learning
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., & Ba, J. (2019) · arXiv · doi:10.48550/arxiv.1907.02057 · Cited by 226
“In the Dyna algorithm [45, 46, 47], training iterates between two steps.”
MethodA Survey of Exploration Methods in Reinforcement Learning
Amin, S., Gomrokchi, M., Satija, H., van Hoof, H., & Precup, D. (2021) · arXiv · doi:10.48550/arxiv.2109.00157 · Cited by 41
Counterfactual Data Augmentation using Locally Factored Dynamics
Pitis, S., Creager, E., & Garg, A. (2020) · arXiv · doi:10.48550/arxiv.2007.02863 · Cited by 26
“Using the same model to generate data directly (a la Dyna [82]) performs poorly, suggesting significant model bias.”
Method
Metadata
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- ACM SIGART Bulletin
- Published
- 1991-07
- Volume
- 2
- Issue
- 4
- Pages
- 160-163
- ISSN
- 0163-5719 (ISSN-L)
- OpenAlex
- W1980035368
- Cited by
- 777
- References
- 18
- Citations by year
- 2026: 56 · 2025: 71 · 2024: 57 · 2023: 63 · 2022: 57 · 2021: 79 · 2020: 115 · 2019: 62 · 2018: 42 · 2017: 25 · 2016: 9 · 2015: 16 · 2014: 5 · 2013: 13 · 2012: 11
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGART Bulletin (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- learning from examples, planning, model-based learning, incremental planning, learning-based planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
