The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Dyna, an integrated architecture for learning, planning, and reacting
Abstract
Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1991
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citationCraik, K. J. W. (1943) · Cited by 1,845
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 916
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 583
Learning and Sequential Decision Making
Barto, A. G., Sutton, R. S., & Watkins, C. (1989) · Cited by 349
Why the Law of Effect will not Go Away*
Dennett, D. C. (1975) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1975.tb00350.x · Cited by 330
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Research works printed but not linked: 10
- Dennett D. C., 1978, Brainstorms, by D. C. Dennett, 71--89
- Holland J. H., 1986, R. Michalski, J. Carbonell &
- Lin Ji., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, 297--305
- Riolo R., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, MIT Press.
- Russell S. J., 1989, Proceedings of IJCAI-89
- 10.1023/A:1022633531479 · DOI · Paper page
- 10.5555/101883.102055 · DOI
- Sutton R.S., 1981, An adaptive network that constructs and uses an internal model of its environment, Cognition and Brain Theory Quarterly, 4, 217
- Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department. Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department.
- 10.1023/A:1022619109594 · DOI · Paper page
Cited by
Sorted by citations
Items 101–200 of 764
Learning and Querying Fast Generative Models for Reinforcement LearningPreprint
Buesing, L., Weber, T., Racanière, S., Eslami, S. M. A., Rezende, D. J., Reichert, D., Viola, F., Besse, F., Gregor, K., Hassabis, D., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03006 · Cited by 118
Episodic Memory Deep Q-Networks
Lin, Z., Zhao, T., Yang, G., & Zhang, L. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/337 · Cited by 92
(2020) · RWTH Publications (RWTH Aachen) · Cited by 68
Yang, N., Han, L., Xiang, C., Liu, H., & Hou, X. (2021) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2021.3064407 · Cited by 65
“Dyna is based on the old idea that planning is like trial-and-error learning from hypothetical experience [22], and it’s an integrated architecture for learning, planning and reacting [23].”
BackgroundChen, L., Meng, F., & Zhang, Y. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3164023 · Cited by 65
“The original Dyna algorithm [24], [25] derived a control policy via value function estimation.”
MethodA Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to ImitationPreprint
Tai, L., Zhang, J., Liu, M., Boedecker, J., & Burgard, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07139 · Cited by 111
“There also exists another branch of model-based algorithms where a model is learned from experiences. with which the agent can interact and collect imaginary rollouts (Sutton, 1991), It has also been extended with DRL methods (Weber et al., 2017; Kalweit and Boedecker, 2017).”
Nyong-Bassey, B. E., Giaouris, D., Patsios, C., Papadopoulou, S., Παπαδόπουλος, Α. Ι., Walker, S. L., Voutetakis, S. S., Seferlis, P., & Gadoue, S. M. (2019) · Energy · doi:10.1016/j.energy.2019.116622 · Cited by 64
“In [62] the Dyna AI architecture was proposed to integrate both learning, and experience, based on online plan ning, as well as reactive execution in a stochastic environm e t.”
Fussell, L., Bergamin, K., & Holden, D. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480527 · Cited by 66
“These World Models [Ha and Schmidhuber 2018], can take many forms, but are often trained to approximate the transition function of the environment - to predict the next state given the previous state and the action taken [Chiappa et al. 2017; Dosovitskiy and Koltun 2016; Sutton 1991].”
BackgroundOn a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient
Tang, J., & Abbeel, P. (2010) · Cited by 59
Kahil, H., Sharma, S., Välisuo, P. O., & Elmusrati, M. S. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.125734 · Cited by 71
Experience Selection in Deep Reinforcement Learning for Control
de Bruin, T., Kober, J., Tuyls, K., & Babuška, R. (2018) · Research Repository (Delft University of Technology) · doi:10.5555/3291125.3291134 · Cited by 56
A survey of reinforcement learning in relational domains
van Otterlo, M. (2005) · CTIT technical reports series · Cited by 58
Deployment-Efficient Reinforcement Learning via Model-Based Offline OptimizationPreprint
Matsushima, T., Furuta, H., Matsuo, Y., Nachum, O., & Gu, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03647 · Cited by 65
“Model-Based RL There are many types of model-based RL algorithms (Sutton, 1991; Deisenroth & Rasmussen, 2011; Heess et al., 2015).”
MethodInfluentialIncreased and biased deliberation in social anxiety
Hunter, L. E., Meer, E. A., Gillan, C. M., Hsu, M., & Daw, N. D. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01180-y · Cited by 67
High-accuracy model-based reinforcement learning, a survey
Plaat, A., Kosters, W. A., & Preuß, M. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10335-w · Cited by 63
“By then using the internal model for policy updates, the number of necessary environment samples can be reduced substantially (Sutton 1991).”
Sharma, H., Kumar, N., & Tekchandani, R. (2022) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2022.3212966 · Cited by 53
“For a dynamic game, a fast Q-based NOMA power allocation system was proposed in [33] which combines the Dyna architecture [34] and hotboot-ing technique to speed the learning process.”
MethodModel-based reinforcement learning for biological sequence design
Angermueller, C., Dohan, D., Belanger, D., Deshpande, R., Murphy, K. J., & Colwell, L. J. (2020) · International Conference on Learning Representations · Cited by 51
Data-driven switching modeling for MPC using Regression Trees and Random Forests
Smarra, F., Girolamo, G. D. D., Iuliis, V. D., Jain, A., Mangharam, R., & D’Innocenzo, A. (2020) · Nonlinear Analysis Hybrid Systems · doi:10.1016/j.nahs.2020.100882 · Cited by 52
Reinforcement learning with Gaussian processes for condition-based maintenance
Peng, S., & Feng, Q. (2021) · Computers & Industrial Engineering · doi:10.1016/j.cie.2021.107321 · Cited by 52
Hybrid predictive coding: Inferring, fast and slow
Tscshantz, A., Millidge, B., Seth, A. K., & Buckley, C. L. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011280 · Cited by 68
“This idea of combining model-free and model-based planning methods (interpreted as iterative and amortized inference) in reinforcement learning also has a long history [59, 60].”
BackgroundMethodChung, J., Fayyad, J., Younes, Y. A., & Najjaran, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10670-6 · Cited by 60
“For example, in the Dyna framework (Sutton 1991), planning is conducted only at previously visited states, and the model is sampled to generate 100 one-step transitions for each action derivation iteration.”
MethodSantos, M., José Antonio Martín, H., López, V., & Botella, G. (2011) · Knowledge-Based Systems · doi:10.1016/j.knosys.2011.09.008 · Cited by 51
Efficient reinforcement learning: model-based Acrobot control
Boone, G. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1997.620043 · Cited by 51
Agentic AI: The age of reasoning—A review
Nisa, U., Shirazi, M., Saip, M. A., & Pozi, M. S. M. (2025) · Journal of Automation and Intelligence · doi:10.1016/j.jai.2025.08.003 · Cited by 83
Abstraction and Generalization in Reinforcement Learning: A Summary and Framework
Ponsen, M., Taylor, M. E., & Tuyls, K. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-11814-2_1 · Cited by 50
“There exist several model-based learning methods, such as D yna-Q [38, 51] and RMax [6], but we will not go into much detail here because we are most interested in domains where the model is assumed to be both unknown and too c omplex to easily”
MethodAdaptive and intelligent navigation of autonomous planetary rovers — A survey
Wong, C., Yang, E., Yan, X.-T., & Gu, D. (2017) · NASA/ESA Conference on Adaptive Hardware and Systems (AHS) · doi:10.1109/ahs.2017.8046384 · Cited by 50
“Dyna was proposed by Sutton in 1991 as an artificial intelligence architecture which combines learning, planning and reactive execution [29].”
BackgroundZhang, W., Wang, J., Liu, Y., Gao, G., Liang, S., & Ma, H. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.115401 · Cited by 50
Wittkuhn, L., Chien, S., Hall-McMaster, S., & Schuck, N. W. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.08.002 · Cited by 49
Anticipations Control Behavior: Animal Behavior in an Anticipatory Learning Classifier System
Butz, M. V., & Hoffmann, J. (2002) · Adaptive Behavior · doi:10.1177/1059712302010002001 · Cited by 50
Hosseinloo, A. H., Ryzhov, A., Bischi, A., Ouerdane, H., Turitsyn, K. S., & Dahleh, M. A. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.115451 · Cited by 49
Structured agents for physical constructionPreprint
Bapst, V., Sánchez‐González, Á., Doersch, C., Stachenfeld, K., Kohli, P., Battaglia, P., & Hamrick, J. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03177 · Cited by 59
“Here we combine our DQN agent with MCTS, in the spirit of Sutton (1991) and Azizzadenesheli et al. (2018).”
MethodAlgorithms and applications of intelligent swarm cooperative control: A comprehensive survey
Xu, X., Yan, X., Yang, W.-Y., An, K., Huang, W., & Wang, Y. (2022) · Progress in Aerospace Sciences · doi:10.1016/j.paerosci.2022.100869 · Cited by 52
Maier, C., Hartung, N., Kloft, C., Huisinga, W., & de Wiljes, J. (2021) · CPT Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.12588 · Cited by 51
Constructing future behavior in the hippocampal formation through composition and replay
Bakermans, J. J. W., Warren, J., Whittington, J. C. R., & Behrens, T. E. J. (2025) · Nature Neuroscience · doi:10.1038/s41593-025-01908-3 · Cited by 75
“This is unlike alternative interpretations of replay—if replay performs Bellman backups 38 (as in the Dyna algorithm 39 ), it can only update the value of any state by comparing to its neighbor, so it is exquisitely sensitive to the trajectory of the replay.”
Pontryagin Differentiable Programming: An End-to-End Learning and Control FrameworkPreprint
Jin, W., Wang, Z., Yang, Z., & Mou, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12970 · Cited by 47
“The use of a model can assist to augment experience data [32, 33], perform back-propagation through time [12], or test policies before deployment.”
BackgroundBenchmarking Reinforcement Learning Techniques for Autonomous Navigation
Xu, Z., Liu, B., Xiao, X., Nair, A., & Stone, P. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160583 · Cited by 46
“error data, has achieved tremendous progress on multiple fronts, including safety [4]–[6], generalizability [7]–[10], sample efficiency [11], [12], and addressing temporal data [13]–[15].”
BackgroundMethodTowards RL-Based Hydraulic Excavator Automation
Egli, P., & Hutter, M. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341598 · Cited by 49
“This can be done in a Dyna setup [20], by improving the actuator model with new data and continue training of the controller based on the updated actuator model.”
MethodTemporal Difference Variational Auto-EncoderPreprint
Gregor, K., Papamakarios, G., Besse, F., Buesing, L., & Weber, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03107 · Cited by 88
“, 1995) or simply learn from simulated experience (Sutton, 1991).”
BackgroundFaster reinforcement learning after pretraining deep networks to predict state dynamics
Anderson, C. W., Lee, M., & Elliott, D. L. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280824 · Cited by 43
“Since the pretraining phase results in a predictive model of the system to be controlled, it would be a simple matter to use it to generate hypothetical samples as in Sutton’s DYNA framework [5].”
MethodOnline learning and exploiting relational models in reinforcement learning
Croonenborghs, T., Ramon, J., Blockeel, H., & Bruynooghe, M. (2007) · Lirias · Cited by 42
Reinforcement Twinning: From digital twins to model-based reinforcement learning
Schena, L., Marques, P. A. D. M., Poletti, R., Ahizi, S., den Berghe, J. V., & Mendez, M. A. (2024) · Journal of Computational Science · doi:10.1016/j.jocs.2024.102421 · Cited by 45
High-level reinforcement learning in strategy games
Amato, C., & Shani, G. (2010) · doi:10.5555/1838206.1838217 · Cited by 40
Hester, T., & Stone, P. (2012) · Adaptation, learning, and optimization · doi:10.1007/978-3-642-27645-3_4 · Cited by 40
“DYNA-2 is an architecture that extends the DYNA idea of updating its value function using both real and simulated experiences to using sample-based planning (Silver et al, 2008).”
BackgroundInfluentialAsynchronous reinforcement learning algorithms for solving discrete space path planning problems
Zhao, X., Ding, S., An, Y., & Jia, W. (2018) · Applied Intelligence · doi:10.1007/s10489-018-1241-z · Cited by 39
The anticipatory classifier system and genetic generalization
Butz, A. M. V., Goldberg, B. D. E., & Stolzmann, C. W. (2002) · Natural Computing · doi:10.1023/a:1021330114221 · Cited by 39
“In the reinforcement learning field, the Dyna architecture (Sutton, 1991b) reveals similarities with the ACS architecture.”
MethodDeep reinforcement learning towards real-world dynamic thermal management of data centers
Zhang, Q., Zeng, W., Lin, Q., Chng, C.-B., Chui, C., & Lee, P. S. (2022) · Applied Energy · doi:10.1016/j.apenergy.2022.120561 · Cited by 44
Harder, N., Qussous, R., & Weidlich, A. (2023) · Energy and AI · doi:10.1016/j.egyai.2023.100295 · Cited by 38
Ekundayo, O., & Ezugwu, A. E.-S. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113378 · Cited by 56
Novel Selection Schemes for Cuckoo Search
Abed-alguni, B. H., & Alkhateeb, F. Q. (2017) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-017-2663-3 · Cited by 38
“As recommended in [29,30], the learning rate α = 0 .”
BackgroundRealigning Models of Habitual and Goal-Directed Decision-Making
Miller, K. J., Ludvig, E. A., Pezzulo, G., & Shenhav, A. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-812098-9.00018-8 · Cited by 46
XCS classifier system with experience replay
Stein, A., Maier, R., Rosenbauer, L., & Hähner, J. (2020) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3377930.3390249 · Cited by 37
“The prototypical scheme of an ALCS can be found in Sutton’s Dyna architecture [43].”
MethodDeep Reinforcement Learning for Navigation in AAA Video Games
Alonso, E., Peter, M., Goumard, D., & Romoff, J. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/294 · Cited by 59
Latent Imagination Facilitates Zero-Shot Transfer in Autonomous Racing
Brunnbauer, A., Berducci, L., Brandstatter, A., Lechner, M., Hasani, R., Rus, D., & Grosu, R. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9811650 · Cited by 36
“by either planning or generating new training experience [20].”
BackgroundFlexible control of Discrete Event Systems using environment simulation and Reinforcement Learning
Zielinski, K., Hendges, L. V., Florindo, J. B., Lopes, Y. K., Ribeiro, R., Teixeira, M., & Casanova, D. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107714 · Cited by 35
Generative Temporal Models with MemoryPreprint
Gemici, M., Hung, C.-C., Santoro, A., Wayne, G., Mohamed, S., Rezende, D. J., Amos, D., & Lillicrap, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04649 · Cited by 78
“…also important components of intelligent agents, as they permit counterfactual reasoning, physical predictions, robot localisation, and simulation-based planning among other capacities (Sutton, 1991; Deisenroth and Rasmussen, 2011; Watter et al., 2015; Levine and Abbeel, 2014; Assael et al., 2015).”
BackgroundEppe, M., Nguyen, P. D. H., & Wermter, S. (2019) · Frontiers in Robotics and AI · doi:10.3389/frobt.2019.00123 · Cited by 37
“The Dyna Architecture (Sutton, 1991) and its derived methods, e.g., Dyna-Q (Sutton, 1990), queue-Dyna (Peng and Williams, 1993), RTP-Q (Zhao et al., 1999), aim to speed up the learning procedure of the agent by unifying reinforcement learning and incremental planning within a single framework.”
BackgroundMethodProcedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning
Bi, J., Luo, J., & Xu, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01532 · Cited by 37
“This approach brings following advantages: a) it helps policy to flexibly pursue a goal by leveraging causal knowledge of the actions’ potential consequences; b) when model-based simulations produce states with alternative actions, the discrimination and selection between actions allow an agent to find the currently m…”
BackgroundOn the role of planning in model-based deep reinforcement learningPreprint
Hamrick, J. B., Friesen, A. L., Behbahani, F., Guez, A., Viola, F., Witherspoon, S., Anthony, T., Buesing, L., Veličković, P., & Weber, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.04021 · Cited by 56
“Moreover, as discussed in the ∗Correspondence addressed to: {jhamrick,theophane}@google.com next section, MuZero has direct connections with many other MBRL methods, including Dyna [67], MPC [11], and policy iteration [33].”
BackgroundMethodInfluentialCombined model-free and model-sensitive reinforcement learning in non-human primates
Miranda, B., Malalasekera, W. M. N., Behrens, T. E. J., Dayan, P., & Kennerley, S. W. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007944 · Cited by 33
A Survey of Planning and Learning in Games
Duarte, F. F., Lau, N., Pereira, A., & Reis, L. P. (2020) · Applied Sciences · doi:10.3390/app10134529 · Cited by 35
Model-based deep reinforcement learning for wind energy bidding
Sanayha, M., & Vateekul, P. (2021) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2021.107625 · Cited by 34
Sanayha, M., & Vateekul, P. (2022) · IEEE Access · doi:10.1109/access.2022.3224460 · Cited by 33
“In conjunction with MBRL, Dyna-architecture has been used to improve interaction in a modeled environment through learning and planning based on real-world and simulated experiences [49].”
MethodParallel and hierarchical neural mechanisms for adaptive and predictive behavioral control
Macpherson, T., Matsumoto, M., Gomi, H., Morimoto, J., Uchibe, E., & Hikida, T. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.09.009 · Cited by 30
MBMF: Model-Based Priors for Model-Free Reinforcement LearningPreprint
Bansal, S., Calandra, R., Chua, K., Levine, S., & Tomlin, C. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.03153 · Cited by 32
“This work follows on a long line of research into using learned models for generating synthetic experience, including the classic Dyna framework [17].”
MethodLearning Deployable Navigation Policies at Kilometer Scale from a Single TraversalPreprint
Bruce, J., Sünderhauf, N., Mirowski, P., Hadsell, R. T., & Milford, M. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.05211 · Cited by 33
“Models can be built from the real environment in which the agent is embodied in order to fit directly the specifics of the robot [20, 21, 22, 23].”
BackgroundSecurity and Privacy Issues in Deep Reinforcement Learning: Threats and Countermeasures
Mo, K., Ye, P.-G., Ren, X., Wang, S., Li, W., & Li, J. (2024) · ACM Computing Surveys · doi:10.1145/3640312 · Cited by 37
“Commonly used model-based algorithms include DYNA [134], AlphaZero[128], and MuZero [120].”
MethodKnowledge-Aided Model-Based Reinforcement Learning for Anti-Jamming Strategy Learning
Li, K., Liu, H., Jiu, B., Pu, W., Peng, X., & Yan, J. (2024) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2024.3358779 · Cited by 33
Saeed, M. H., Kazmi, H., & Deconinck, G. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114879 · Cited by 30
Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation
Cao, X., Xu, M., Yu, X., Yao, J., Ye, W., Huang, S.-J., Zhang, M.-L., Tsang, I. W.-H., Ong, Y.-S., Kwok, J. T.-Y., et al. (2025) · ACM Computing Surveys · doi:10.1145/3773075 · Cited by 37
Action and Perception as Divergence MinimizationPreprint
Hafner, D., Ortega, P. A., Ba, J. L., Parr, T., Friston, K. J., & Heess, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.01791 · Cited by 52
“For example, past inputs can be stored in an experience dataset and future inputs can be approximated by either planning with a learned world model, on-policy trajectories, or replay of past inputs (Sutton, 1991).”
BackgroundInfluentialThe role of Reinforcement Learning in software testing
Abo-eleneen, A., Palliyali, A. W., & Catal, C. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107325 · Cited by 38
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term FuturePreprint
Ke, N. R., Singh, A., Touati, A., Goyal, A., Bengio, Y., Parikh, D., & Batra, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.01599 · Cited by 34
“Many of these prior methods aim to learn the dynamics model of the environment which can then be used for planning, generating synthetic experience, or policy search (Atkeson & Schaal, 1997; Peters et al., 2010; Sutton, 1991).”
MethodNeural evidence for the successor representation in choice evaluationPreprint
Russek, E. M., Momennejad, I., Botvinick, M., Gershman, S. J., & Daw, N. D. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.08.29.458114 · Cited by 34
“For example, whereas some approaches to valuation construct reward predictions directly from simulation or replay (e.g. “Dyna-Q” (Sutton, 1991) and basic MB value iteration), other recent approaches use either simulation or replay to update an SR and construct reward predictions from this (e.g.…”
MethodMoneta, N., Grossman, S., & Schuck, N. W. (2024) · Trends in Neurosciences · doi:10.1016/j.tins.2024.10.005 · Cited by 37
Reinforcement Learning Based Network Coding for Drone-Aided Secure Wireless Communications
Xiao, L., Li, H., Yu, S., Zhang, Y., Wang, L., & Ma, S. (2022) · IEEE Transactions on Communications · doi:10.1109/tcomm.2022.3194074 · Cited by 28
“Instead of using the standard Dyna architecture as presented in [35], a virtual model consists of four online planning functions is used to generate W simulated secure communication experiences, which are applied to provide additional updates for Q-values.”
MethodDo recent advancements in model-based deep reinforcement learning really improve data efficiency?Preprint
Kielak, K. (2019) · arXiv (Cornell University) · Cited by 29
Off-Dynamics Reinforcement Learning: Training for Transfer with Domain ClassifiersPreprint
Eysenbach, B., Asawa, S., Chaudhari, S., Levine, S., & Salakhutdinov, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.13916 · Cited by 79
World Model as a Graph: Learning Latent Landmarks for PlanningPreprint
Zhang, L., Yang, G., & Stadie, B. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.12491 · Cited by 32
“with a probabilistic and unpredictable world, and the action space for robots is oftentimes continuous. As a result, planning in robotics presents a much harder problem. One general class of methods (Sutton, 1991) seeks to combine model-based planning and deep RL. These methods can be thought of as an extension of mod…”
BackgroundMethodInfluentialProgress and summary of reinforcement learning on energy management of MPS-EV
Lin, Y., Chu, L., Hu, J., Hou, Z., Li, J., Jiang, J., & Zhang, Y. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e23014 · Cited by 28
“Dyna is an artificial intelligence (AI) architecture that integrates learning, planning, and reacting [60], as shown in Fig.”
BackgroundModel-Based Transfer Reinforcement Learning Based on Graphical Model Representations
Sun, Y., Zhang, K., & Sun, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3107375 · Cited by 28
A Method Integrating Q-Learning With Approximate Dynamic Programming for Gantry Work Cell Scheduling
Ou, X., Chang, Q., & Chakraborty, N. (2020) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2020.2984739 · Cited by 27
“Dyna, an integrated architecture for learning, planning, and reacting, is introduced in [28].”
BackgroundRobust Model-based Reinforcement Learning for Autonomous Greenhouse ControlPreprint
Zhang, W., Cao, X., Yao, Y., An, Z., Xiao, X., & Luo, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.11645 · Cited by 28
“And with the idea of Dyna-style RL (Sutton (1991)), we continually transmit the collected real samples to the simulator for further optimization of the tomato growth model in the policy optimization cycle, which enables the simulator to provide simulation samples for the framework more efficiently…”
MethodConstructing parsimonious analytic models for dynamic systems via symbolic regression
Derner, E., Kubalı́k, J., Ancona, N., & Babuška, R. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106432 · Cited by 26
“Various model-based methods have been proposed to speed up learning [11, 12, 13, 14, 15].”
Intelligent Motion Video Guidance for Unmanned Air System Ground Target Surveillance
Valasek, J., Kirkpatrick, K., May, J., & Harris, J. (2015) · Journal of Aerospace Information Systems · doi:10.2514/1.i010198 · Cited by 26
“In this case, more advanced reinforcement learning techniques such asmodel-based RL techniques, which learn a systemmodel from the environmental data; function approximation techniques; and hierarchal learning can be used [14–17].”
MethodReinforcement learning and artificial agency
Butlin, P. (2024) · Mind & Language · doi:10.1111/mila.12458 · Cited by 40
Automatic Generation Control in a Distributed Power Grid Based on Multi-Step Reinforcement Learning
Zhao, W., Zeng, T., Liu, Z., Xie, L., Xi, L., & Ma, H. (2024) · Protection and Control of Modern Power Systems · doi:10.23919/pcmp.2023.000220 · Cited by 25
Adapting to Run-Time Changes in Policies Driving Autonomic Management
Bahati, R. M., & Bauer, M. A. (2008) · International Conference on Networking and Services · doi:10.1109/icas.2008.47 · Cited by 25
“In this paper, we make use of an algorithm called Dyna-Q [11] which combines the strength of direct reinforcement learning in Q-learning [6], [12] and model learning to guide the learning process.”
MethodReinforcement Learning with a Disentangled Universal Value Function for Item Recommendation
Wang, K., Zou, Z., Deng, Q., Tao, J., Wu, R., Fan, C., Chen, L., & Cui, P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16569 · Cited by 26
Ready Policy One: World Building Through Active LearningPreprint
Ball, P., Parker-Holder, J., Pacchiano, A., Choromański, K., & Roberts, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02693 · Cited by 29
“In the Dyna-style approach (Sutton, 1991) that we adopt, this means training our sampling policy πs in the world model exclusively, then using this policy to collect samples in the real world.”
MethodModel-Based Visual Planning with Self-Supervised Functional DistancesPreprint
Tian, S., Nair, S., Ebert, F., Dasari, S., Eysenbach, B., Finn, C., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15373 · Cited by 37
“…McAllister & Rasmussen, 2016; Chua et al., 2018; Amos et al., 2018; Hafner et al., 2019b; Nagabandi et al., 2018; Kahn et al., 2020; Dong et al., 2020) or policy optimization (Sutton, 1991; Weber et al., 2017; Ha & Schmidhuber, 2018; Janner et al., 2019; Wang & Ba, 2019; Hafner et al., 2019a).”
BackgroundProspective Role of Foundation Models in Advancing Autonomous Vehicles
Wu, J., Gao, B., Gao, J., Yu, J., Chu, H., Yu, Q., Gong, X., Chang, Y. M., Tseng, H. E., Chen, H., et al. (2024) · Research · doi:10.34133/research.0399 · Cited by 26
“The idea of world model can be traced back to Dyna, proposed by Sutton [129] in 1991, to observe the state of the world and take appropriate actions accordingly to learn interactively with the world [130].”
BackgroundAsynchronous action-reward learning for nonstationary serial supply chain inventory control
Kim, C. O., Kwon, I.-H., & Baek, J. (2007) · Applied Intelligence · doi:10.1007/s10489-007-0038-2 · Cited by 24
“Currently, prioritized sweeping [24] and Dyna-Q [25] are known as representative learning architectures for boosting the learning speed.”
BackgroundDeep Reinforcement Learning with Sarsa and Q-Learning: A Hybrid Approach
Xu, Z., Cao, L., Chen, X., Li, C., Zhang, Y., & Lai, J. (2018) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2017edp7278 · Cited by 24
“Compared with Qlearning, which uses the maximum action-state value for iteration, whereas Sarsa learning use the actual action-state value for iteration, and strictly updates the value function according to the experience gained by executing a strategy [11], [12].”
Li, L., Liu, L., Bai, J., Chang, H.-H., Chen, H., Ashdown, J., Zhang, J., & Yi, Y. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2988268 · Cited by 25
“In [5], Dyna introduces a new class of RL algorithms that use models to accelerate MFRL.”
MethodLearning to Fly via Deep Model-Based Reinforcement LearningPreprint
Becker-Ehmck, P., Karl, M., Peters, J., & van der Smagt, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08876 · Cited by 45
“In Dyna [22] and derivative work, the learnt dynamics model is used exclusively as a data generator for optimising the policy using standard model-free algorithms.”
MethodCarè, A., Ruggero Carli, A., Libera, A. D., Romeres, D., & Pillonetto, G. (2023) · IEEE Control Systems · doi:10.1109/mcs.2023.3291625 · Cited by 23
“The first example of the MBRL algorithm can most likely be attributed to [207] under the name of Dyna.”
MethodHettiarachchi, C., Malagutti, N., Nolan, C. J., Suominen, H., & Daskalaki, E. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105839 · Cited by 31
The Development of Goal-Directed Decision-Making
Raab, H. A., & Hartley, C. A. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-812098-9.00013-9 · Cited by 25
“Importantly, such a mechanism might enable behavioral adaptation to changes in outcome valuation or identification of a newly rewarding action (Sutton, 1991).”
BackgroundPeters, J., Lee, D. D., Kober, J., Nguyen-Tuong, D., Bagnell, J. A., & Schaal, S. (2016) · Springer handbooks · doi:10.1007/978-3-319-32552-1_15 · Cited by 27
Dynamics-aware EmbeddingsPreprint
Whitney, W., Agarwal, R., Cho, K., & Gupta, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.09357 · Cited by 26
“which facilitate exploration via intrinsic motivation (Kim et al., 2018). Another line of work couples the process of learning a model of the environment with training a policy on imagined rollouts (Sutton, 1991; Deisenroth & Rasmussen, 2011; Ha & Schmidhuber, 2018; Clavera et al., 2018; Kaiser et al., 2019; H…”
Background
Metadata
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- ACM SIGART Bulletin
- Published
- 1991-07
- Volume
- 2
- Issue
- 4
- Pages
- 160-163
- ISSN
- 0163-5719 (ISSN-L)
- OpenAlex
- W1980035368
- Cited by
- 777
- References
- 18
- Citations by year
- 2026: 56 · 2025: 71 · 2024: 57 · 2023: 63 · 2022: 57 · 2021: 79 · 2020: 115 · 2019: 62 · 2018: 42 · 2017: 25 · 2016: 9 · 2015: 16 · 2014: 5 · 2013: 13 · 2012: 11
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGART Bulletin (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- learning from examples, planning, model-based learning, incremental planning, learning-based planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
