The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Dyna, an integrated architecture for learning, planning, and reacting
Abstract
Dyna is an AI architecture that integrates learning, planning, and reactive execution. Learning methods are used in Dyna both for compiling planning results and for updating a model of the effects of the agent's actions on the world. Planning is incremental and can use the probabilistic and ofttimes incorrect world models generated by learning processes. Execution is fully reactive in the sense that no planning intervenes between perception and action. Dyna relies on machine learning methods for learning from examples---these are among the basic building blocks making up the architecture---yet is not tied to any particular method. This paper briefly introduces Dyna and discusses its strengths and weaknesses with respect to other architectures.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
From Goals and Habits in the Brain (2013) · cites this paper as Sutton, 1991
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citationCraik, K. J. W. (1943) · Cited by 1,845
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 916
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 583
Learning and Sequential Decision Making
Barto, A. G., Sutton, R. S., & Watkins, C. (1989) · Cited by 349
Why the Law of Effect will not Go Away*
Dennett, D. C. (1975) · Journal for the Theory of Social Behaviour · doi:10.1111/j.1468-5914.1975.tb00350.x · Cited by 330
Learning to perceive and act by trial and error
Whitehead, S. D., & Ballard, D. H. (1991) · Machine Learning · doi:10.1007/bf00058926 · Cited by 300
Learning Sequential Decision Rules Using Simulation Models and Competition
Grefenstette, J. J., Ramsey, C. L., & Schultz, A. C. (1990) · Machine Learning · doi:10.1023/a:1022677607120 · Cited by 266
Werbos, P. J. (1987) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1987.289329 · Cited by 249
Holland, J. H. (1986) · Machine Learning · Cited by 121
Lookahead Planning and Latent Learning in a Classifier System
Riolo, R. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/3115.003.0043 · Cited by 87
Self-improving Reactive Agents: Case Studies of Reinforcement Learning Frameworks
Lin, L.-J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/3115.003.0041 · Cited by 52
Execution architectures and compilation
Russell, S. (1989) · International Joint Conference on Artificial Intelligence · Cited by 27
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Research works printed but not linked: 10
- Dennett D. C., 1978, Brainstorms, by D. C. Dennett, 71--89
- Holland J. H., 1986, R. Michalski, J. Carbonell &
- Lin Ji., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, 297--305
- Riolo R., 1991, Proceedings of the International Conference on the Simulation of Adaptive Behavior, MIT Press.
- Russell S. J., 1989, Proceedings of IJCAI-89
- 10.1023/A:1022633531479 · DOI · Paper page
- 10.5555/101883.102055 · DOI
- Sutton R.S., 1981, An adaptive network that constructs and uses an internal model of its environment, Cognition and Brain Theory Quarterly, 4, 217
- Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department. Watkins C. J. C. H. (1989) Learning with Delayed Rewards. PhD thesis Cambridge University Psychology Department.
- 10.1023/A:1022619109594 · DOI · Paper page
Cited by
Sorted by citations
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
“The policy is, in a sense, a plan, but one that is completely conditioned by current input.”
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
“In fact, the authors observed that subjects acted in a manner consistent with a model-based system having trained by a model-free one during an earlier phase of learning, as in an online or offline form of the DYNA-Q algorithms mentioned above (Sutton, 1991).”
InfluentialNeural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8463189 · Cited by 898
Metadata
- Authors
- Sutton, R. S.
- Affiliations
- GTE (United States) (US)
- Corresponding author
- Sutton, Richard S.
- ORCID
- Sutton, Richard S.
- Venue
- ACM SIGART Bulletin
- Published
- 1991-07
- Volume
- 2
- Issue
- 4
- Pages
- 160-163
- ISSN
- 0163-5719 (ISSN-L)
- OpenAlex
- W1980035368
- Cited by
- 777
- References
- 18
- Citations by year
- 2026: 56 · 2025: 71 · 2024: 57 · 2023: 63 · 2022: 57 · 2021: 79 · 2020: 115 · 2019: 62 · 2018: 42 · 2017: 25 · 2016: 9 · 2015: 16 · 2014: 5 · 2013: 13 · 2012: 11
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGART Bulletin (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- learning from examples, planning, model-based learning, incremental planning, learning-based planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
